How best to deploy to Raspberry Pi

Assume the following setup:

  • A flake.nix which configures an instance of nixos.
  • The flake uses sops-nix for secrets management, where the secrets are stored in a separate, private, github repo.
  • I want to deploy this flake with nixos-anywhere (i.e. there is a disko config, etc).

This has so far worked for me very nicely and turned out to be a quite sustainable way to have reoccurring deployments.

My issue is that I need to repeat this for a Raspberry Pi 4B. Getting the image onto the SD card is a bit more of a headache than I would like, but that worked fine. The issues started with the disko config. This is somewhat more complicated than I would like and even discouraged by the maintainer of disko (see this thread here).

Before continuing into this directioin, I thought it might make more sense to just create the correct image, with everything I want so that I can just flash that from the start onto the Pi. Building the image worked fine after some tinkering, but my issue here is that I dont know how to sanely deploy the ssh key needed to decrypt the sops secrets.

So to summarize options and maybe ask questions:

  • nixos-anywhere + disko: Is this feasible, i.e. has anybody tried it and had success with it?
  • Build SD Image: Not sure how to get the secrets to decrypt the sops-file to the SD card correctly. Any ideas on how to approach this?

Does the community have any input on which method is preferable?

One thing to note is that the kernel used on Raspberry Pi’s seems not to support kexec as can be seen in this issue: kexec fails on Raspberry Pi OS: missing /proc/kcore · Issue #183 · nix-community/nixos-anywhere · GitHub and as I found to still be the case when I tried it ~3 months ago.

This means one must boot a nixos installer image, probably from USB which requires a little bit of setup on a Raspberry Pi as I recall, and then using Nixos-Anywhere through that.

Which is all a little awkward, which is why I elected to not bother reinstalling, and keep my existing install based on the prebuilt sdimage.

This is not to say that using Nixos-Anywhere and Disko is not possible, I am pretty sure it is possible, just that I looked into it and elected not to bother.

Regarding the build sdimage sops issue, I think I just used the prebuilt image, login as nixos, update sops config with the autogenerated sshkeys, and rebuild to my config. Pretty manual, but I last did this when my nix skills were up to nothing more interesting.

Your mileage may vary.

Didnt know that kexec was disabled on the Pi by default. Thanks for this…

I think I just used the prebuilt image, login as nixos, update sops config with the autogenerated sshkeys, and rebuild to my config.

To clarify, assume that Host A is a host that can add and change sops secrets.

  1. Download the correct NixOS image and flash it onto the Pi over the Pi flasher tool.
  2. Go through the standard setup procedure for NixOS and install it.
  3. Once installed, reboot the system, and get the age key from the /etc/ssh/ssh_host_ed25519_key.
  4. On Host A add this key to .sops.yaml and update your secrets.
  5. Create a ssh key and add it to Github so that the Pi has access to your flakes.
  6. Run
    sudo nixos-rebuild switch --flake github:username/nix-config-flake#rpi
    
    on the Pi.

Does that sum up your proposed solution correctly?

1 Like

Yep, that seems exactly right, and is how I recall doing it some months ago. Except my flake is public, and so no ssh key adding to any git provider was required, just use https.

Again, there are almost certainly more elegant solutions requiring less manual work, but that is what I used when I last did this.

Getting secrets into an sd card image is annoying. You can look into mcopy or virt-edit or similar

In Thymis for example we inject secrets onto a raspberry pi sd card image using mcopy

Well, I can at least confirm that this worked… Not super happy about the solution, but at least the system works, and I dont have any better ideas^^

Apologies, but I’m not familiar with the tools, and after quickly glancing at them, Im still not sure I understand how they are supposed to be used…

So, I recently came back to this issue to redeploy my PI, and found that one can actually use nixos-anywhere, one must just install from a bootable USB rather than trying to kexec into an image from SD card.

A quick run through is:

Firstly, create an SD image for the appropriate RaspberryPi using GitHub - nvmd/nixos-raspberrypi: Nix flake for a fully declarative NixOS on Raspberry Pi . Somewhat confusingly, this can actually be flashed to a USB stick and then booted. To do so, remove the SD card from the PI, insert the USB stick, and power on. Once booted to the USB stick one should see the root password and such on screen. Insert the SD card. Now one can treat the raspberry pi as one would any system for installation using nixos-anywhere.

Note that one must have a firmware partition with the required files for booting. An example of this using Disko can be seen at Using disko to generate raspberry pi sd card images · Issue #550 · nix-community/disko · GitHub . Much of this example is specific to trying to create an image suitable for flashing to an SD card and then self expanding, which is not necessary as we are using nixos-anywhere. For reference, here is how the firmware is populated+configured in the nixpkgs SD images: nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64.nix at ee889ba3c108edae7bb38a33a0fa33f318aa09e9 · NixOS/nixpkgs · GitHub . My own setup, derived largely from the issue comment above, using btrfs and impermanence is

{ disk, pkgs, ... }:
let
  configTxt = pkgs.writeText "config.txt" ''
    [pi4]
    kernel=u-boot-rpi4.bin
    enable_gic=1

    # Otherwise the resolution will be weird in most cases, compared to
    # what the pi3 firmware does by default.
    disable_overscan=1

    # Supported in newer board revisions
    arm_boost=1

    [all]
    # Boot in 64-bit mode.
    arm_64bit=1

    # U-Boot needs this to work, regardless of whether UART is actually used or not.
    # Look in arch/arm/mach-bcm283x/Kconfig in the U-Boot tree to see if this is still
    # a requirement in the future.
    enable_uart=1

    # Prevent the firmware from smashing the framebuffer setup done by the mainline kernel
    # when attempting to show low-voltage or overtemperature warnings.
    avoid_warnings=1
  '';
in
{
  disko.devices.disk."${disk}" = {
    type = "disk";
    device = "/dev/${disk}";
    content = {
      type = "gpt";
      partitions = {
        firmware = {
          size = "60M";
          priority = 1;
          type = "0700";
          content = {
            type = "filesystem";
            format = "vfat";
            mountpoint = "/firmware";
            postMountHook = toString (
              pkgs.writeScript "postMountHook.sh" ''
                (cd ${pkgs.raspberrypifw}/share/raspberrypi/boot && cp bootcode.bin fixup*.dat start*.elf *.dtb /mnt/firmware/)
                cp ${pkgs.ubootRaspberryPi4_64bit}/u-boot.bin /mnt/firmware/u-boot-rpi4.bin
                cp ${configTxt} /mnt/firmware/config.txt
              ''
            );
          };
        };
        ESP = {
          type = "EF00";
          size = "2G";
          content = {
            type = "filesystem";
            format = "vfat";
            mountpoint = "/boot";
            mountOptions = [ "umask=0077" ];
          };
        };
        root = {
          size = "100%";
          content = {
            type = "btrfs";
            extraArgs = [ "-f" ]; # Override existing partition
            subvolumes = {
              "root" = {
                mountOptions = [
                  "compress=zstd"
                  "noatime"
                ];
                mountpoint = "/";
              };
              "home" = {
                mountOptions = [
                  "compress=zstd"
                  "noatime"
                ];
                mountpoint = "/home";
              };
              "nix" = {
                mountOptions = [
                  "compress=zstd"
                  "noatime"
                ];
                mountpoint = "/nix";
              };
              "persistent" = {
                mountOptions = [
                  "compress=zstd"
                  "noatime"
                ];
                mountpoint = "/persistent";
              };
            };
            postCreateHook = ''
              MNTPOINT=$(mktemp -d)
              mount /dev/disk/by-partlabel/disk-${disk}-root "$MNTPOINT" -o subvol=/
              trap 'umount $MNTPOINT; rm -rf $MNTPOINT' EXIT
              btrfs subvolume snapshot -r $MNTPOINT/root $MNTPOINT/root-blank
            '';
          };
        };
      };
    };
  };
  fileSystems."/persistent".neededForBoot = true;
}

A separate approach is using systemd-repart to create the SD images, as can be seen here: GitHub - MatthewCroughan/raspberrypi-nixos-example: The simplest possible way to begin using and extending a NixOS Configuration with a Raspberry Pi

I did not look into this too much, as I prefer using nixos-anywhere. And it was not immediately clear to me how to add secrets when using repart, so I went ahead with nixos-anywhere.

2 Likes

I built this week a setup to create an image with disko that you can then flash onto the SD card.

What isn’t part of it yet is generating the host keys to then decrypt the agenix secrets. However this shouldn’t be too hard to add as I already generate and copy the SSH keys for SSH in initrd to be able to unlock the LUKS remotely. Adding the main host SSH keys should be basically the same.

The whole thing boots with UEFI and uses the normal NixOS kernel as that works for my needs just fine but I don’t use any special Raspberry Pi features atm.

Firstly, create an SD image for the appropriate RaspberryPi using GitHub - nvmd/nixos-raspberrypi: Nix flake for a fully declarative NixOS on Raspberry Pi . Somewhat confusingly, this can actually be flashed to a USB stick and then booted.

I am in a similar situation - I wish to achieve a fully-automated deployment of NixOS on my Raspberry Pi 4 (including disk partitioning/formatting) using NixOS-anywhere. Could you elaborate a bit more on this step?
I tried generating one of the installer images (after disabling image compression) with nix build /my/fork/of/nixos-raspberrypi#installerImages.rpi4. The image builds successfully, and I can flash it on a USB stick (I configured my Pi 4 to boot from usb beforehand). Attempting to boot the image would result in a black screen shortly after NixOS stage 1.

I want to thank every brave person who’s tried this…my own experience has been successful but unenjoyable.

It seems kinda weird that, even with fixed hardware, there isn’t a trivial way to just get a Pi to work with NixOS.

Odd. I simply never encountered a similar issue, so not really sure what might cause that.

Maybe try an image built from upstream master nixos-raspberrypi?