How to move from building sd card images, to be able to remotely deploy new versions?

Help
1

Thanks for this and this I’m now able to build a raspberry pi 3b (aarch64-linux) sd card image on my x86 laptop. This works well. I can move the sd card image onto an sd card, and then boot up the raspberry with that sd card. Here’s my current flake files:

flake.nix

{
  description = "NixOS Raspberry Pi configuration flake";
  outputs = { self, nixpkgs }: {
    nixosConfigurations.rpi = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [
        "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
        ({ ... }: {
          config = {
            # Time, keyboard language, etc
            time.timeZone = "Europe/Oslo";
            i18n.defaultLocale = "en_US.UTF-8";
            console = {
              font = "Lat2-Terminus16";
              keyMap = "no";
            };

            # User
            users.users.stian = {
              isNormalUser = true;
              extraGroups = [
                "wheel" # Enable ‘sudo’ for the user.
                "networkmanager"
              ];
              openssh.authorizedKeys.keys = [
                "ssh-ed25519 AAAAredacted"
              ];
              password = "redacted";
            };

            # Allow ssh in
            services.openssh.enable = true;

            networking = {
              hostName = "rasp2";
              networkmanager.enable = true;
            };

            # This makes the build be a .img instead of a .img.zst
            sdImage.compressImage = false;

            system = {
              stateVersion = "22.05";
            };
          };
        })
      ];
    };
  };
}

flake.lock

{
  "nodes": {
    "nixpkgs": {
      "locked": {
        "lastModified": 1659446231,
        "narHash": "sha256-hekabNdTdgR/iLsgce5TGWmfIDZ86qjPhxDg/8TlzhE=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "eabc38219184cc3e04a974fe31857d8e0eac098d",
        "type": "github"
      },
      "original": {
        "id": "nixpkgs",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 7
}

(Note that this ref is the latest 21.11 version of nixos. With 22.11 I’m unable to boot for some reason. See this other discourse post for details on that issue: Problems booting on Raspberry Pi 3b+)

What I want to do now though is to avoid having to create the sd image. I’d like to build the configuration for the raspberry pi on my laptop, and then deploy that straight to the raspberry pi using ssh. I know this is possible, but I’m struggling to find the next steps. I’m assuming that I have to something like this:

  • extract the configuration out of the above nixosConfigurations.rpi, and keep only the sd-image card stuff there
  • create a new nixosConfigurations.rpiConfiguration or something that has only the system, and not the sd-image stuff
  • at this point I hopefully have a flake that allows me to either build the sd image, or just the system configuration
  • learn how to use something like GitHub - serokell/deploy-rs: A simple multi-profile Nix-flake deploy tool. or similar to get the build result over to my raspberry pi

Any pointers on how to go through these steps? Or feedback on the steps themselves?

1 Like
2

This is the part you want to extract in an extra nix file. Not sure what other concrete questions you have.

1 Like
3

Thanks for responding!

I was able to extract that part, and now my configuration looks ike this:

{
  description = "NixOS Raspberry Pi configuration flake";
  outputs = { self, nixpkgs }: {
    nixosConfigurations = {
      rpi = nixpkgs.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [
          "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
          ./configuration.nix
        ];
      };
    };
  };
}

But my configuration.nix still contains this:

  # This makes the build be a .img instead of a .img.zst
  sdImage.compressImage = false;

My first step to create two different nixos configurations is this:

{
  description = "NixOS Raspberry Pi configuration flake";
  outputs = { self, nixpkgs }: {
    nixosConfigurations = {
      rpi = nixpkgs.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [
          "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
          ./configuration.nix
        ];
      };
      rpiImage = nixpkgs.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [
          "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
          ./configuration.nix
        ];
      };
    };
  };
}

But I don’t want to include any of the sd image-stuff in the first one, only in the second one. So I remove the sdImage.compressImage from configuration.nix and add it into the nixosConfigurations.rpiImage block. Now it looks like this:

{
  description = "NixOS Raspberry Pi configuration flake";
  outputs = { self, nixpkgs }: {
    nixosConfigurations = {
      rpi = nixpkgs.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [
          ./configuration.nix
        ];
      };
      rpiImage = nixpkgs.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [
          "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
          ./configuration.nix
          # Inline configuration here
          ({ ... }: {
            config = {
              # This makes the build be a .img instead of a .img.zst
              sdImage.compressImage = false;
            };
          })
        ];
      };
    };
  };
}

And building with nix build .#nixosConfigurations.rpiImage.config.system.build.sdImage seems to still work as before. However, how do I build nixosConfigurations.rpi? My goal is to build that and get something in my nix store that I can copy over to the running raspberry.

4

First, I’m not sure.
I looking into doing something similar. I’m building a custom image and write it on a thumbdrive. Next I boot a rpi4 with the thumbdrive.
I think of it as a live image and Next install nixos on an attached usb ssd.

5

That should be ignored but you can move it to the sd building pieces if you want.

.# nixosConfigurations.rpi.config.system.build.toplevel or nixos-rebuild build --target-host rpi but for that you would need to cross compile. See the --help for details.

2 Likes
6

Thanks!

➜ nix build .#nixosConfigurations.rpi.config.system.build.toplevel
warning: Git tree '/home/stian/devp/raspberrypi2' is dirty
error:
       Failed assertions:
       - The ‘fileSystems’ option does not specify your root file system.
       - You must set the option ‘boot.loader.grub.devices’ or 'boot.loader.grub.mirroredBoots' to make the system bootable.

How can I figure which options were set in the sd card image? Because I assume that I should use the same ones?

7

You don’t need two separate nixosConfigurations definitions, you can (and probably want to) keep the configuration from sd-image-aarch64.nix in the host config you’ll deploy. As you can see, the filesystems are defined there.

When you want to deploy you’ll build the toplevel output, and when you want to build the image you can instead build nixosConfigurations.<name>.config.system.build.sdImage, but they will share their configuration.

4 Likes
8

That makes perfect sense, thanks! I now know that I can do

  • nix build .#nixosConfigurations.rpi.config.system.build.sdImage to build the sd card image, and
  • nix build .#nixosConfigurations.rpi.config.system.build.toplevel to build (only) the system
9

My next step is to be able to copy the built system over to my raspberry pi. Googling has led me to try nix copy and nix-copy-closure, but I get into issues related to things not being signed (see details below). My built system looks like:

➜ ls -ahl . | grep result
lrwxrwxrwx    85 stian 31 Jan 21:05  result -> /nix/store/0jslanivx1pfwvj3970gs90k6fndhwnj-nixos-system-rasp2-21.11.20220802.eabc382

➜ ls -ahl result
Permissions Size User Date Modified Name
.r-xr-xr-x   14k root  1 Jan  1970  activate
lrwxrwxrwx    91 root  1 Jan  1970  append-initrd-secrets -> /nix/store/9jga3iqpaw1cbmbmyil1c9cm3ij5q7vw-append-initrd-secrets/bin/append-initrd-secrets
dr-xr-xr-x     - root  1 Jan  1970  bin
.r--r--r--     0 root  1 Jan  1970  configuration-name
.r-xr-xr-x  2.2k root  1 Jan  1970  dry-activate
lrwxrwxrwx    63 root  1 Jan  1970  dtbs -> /nix/store/850gzj56pzrqcv6154x5q3hm0fkivl47-linux-5.10.126/dtbs
lrwxrwxrwx    51 root  1 Jan  1970  etc -> /nix/store/60i73igpxch3gc4zvyq8l4pckw4yzjmi-etc/etc
.r--r--r--    75 root  1 Jan  1970  extra-dependencies
lrwxrwxrwx    65 root  1 Jan  1970  firmware -> /nix/store/w8b3lrwpbdwwhdzncv3hn5v1ynyg40f1-firmware/lib/firmware
.r-xr-xr-x  5.9k root  1 Jan  1970  init
.r--r--r--     9 root  1 Jan  1970  init-interface-version
lrwxrwxrwx    72 root  1 Jan  1970  initrd -> /nix/store/pnmlhjm2mfxsapdhzn99k99sz2vxi964-initrd-linux-5.10.126/initrd
lrwxrwxrwx    64 root  1 Jan  1970  kernel -> /nix/store/850gzj56pzrqcv6154x5q3hm0fkivl47-linux-5.10.126/Image
lrwxrwxrwx    58 root  1 Jan  1970  kernel-modules -> /nix/store/snmrci59rrym8xwlyl820yhhyznqpxrr-kernel-modules
.r--r--r--    71 root  1 Jan  1970  kernel-params
.r--r--r--    22 root  1 Jan  1970  nixos-version
dr-xr-xr-x     - root  1 Jan  1970  specialisation
lrwxrwxrwx    55 root  1 Jan  1970  sw -> /nix/store/0zhw6xqig0ycpbaalri8gi8p6mvfza6k-system-path
.r--r--r--    13 root  1 Jan  1970  system
lrwxrwxrwx    57 root  1 Jan  1970  systemd -> /nix/store/fgcrc8gxf14ddyzlxxjaw6dxdqm52pav-systemd-249.7

nix copy

I’m trying this:

➜ nix copy -vvv --to ssh://stian@192.168.68.120 /nix/store/0jslanivx1pfwvj3970gs90k6fndhwnj-nixos-system-rasp2-21.11.20220802.eabc382
...
copying 10 paths...
copying path '/nix/store/0zhw6xqig0ycpbaalri8gi8p6mvfza6k-system-path' to 'ssh://stian@192.168.68.120'...
[1/0/10 copied (0.0/4.8 MiB)] copying path '/nix/store/0zhw6xqig0ycpbaalri8gi8p6mvfza6k-system-path' to 'ssh://stian@192.168.68.120'error: cannot add path '/nix/store/0zhw6xqig0ycpbaalri8gi8p6mvfza6k-system-path' because it lacks a valid signature
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 1038144, received 53096 bytes, in 1.0 seconds
Bytes per second: sent 1034995.3, received 52935.0
debug1: Exit status 1
error: writing to file: Broken pipe

I’m not sure how to get past that. Googling led me to try adding --no-check-sigs, but that doesn’t seem to have any effect.

nix-copy-closure

I’m trying this:

➜ nix-copy-closure -vvv --to --use-substitutes stian@192.168.68.120 /nix/store/0jslanivx1pfwvj3970gs90k6fndhwnj-nixos-system-rasp2-21.11.20220802.eabc382
...
adding path '/nix/store/0zhw6xqig0ycpbaalri8gi8p6mvfza6k-system-path' to remote host 'stian@192.168.68.120'
copying path '/nix/store/0zhw6xqig0ycpbaalri8gi8p6mvfza6k-system-path' to 'ssh://stian@192.168.68.120'...
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
error: cannot add path '/nix/store/0zhw6xqig0ycpbaalri8gi8p6mvfza6k-system-path' because it lacks a valid signature
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 972896, received 53096 bytes, in 1.2 seconds
Bytes per second: sent 788891.9, received 43053.9
debug1: Exit status 1
reaping 8 worker threads
killing process 21798
error: writing to file: Broken pipe

I learned that I should be able to add this to configuration.nix to add myself as a trusted user:

  nix.settings.trusted-users = [ "root" "stian" ];

But that option came with 22.11, and I’m forced to stay on 21.11 because anything newer doesn’t boot. How can I add a trusted user, then?

10 
nix.extraOptions = ''
  trusted-users = [ "root" "stian" ]
'';

https://search.nixos.org/options?channel=unstable&show=nix.extraOptions&from=0&size=50&sort=relevance&type=packages&query=nix.extraOptions

1 Like