Flake to create a simple SD image for RPI4 (cross)

Help
1

I am new to nix/nixos and wanted to start simple by trying to create a RPI image from a nixos installation using flakes. I was able to create a bootable sd image, but it seems like changing the configuration, like the password, does not trigger a rebuild of the image. And the user configuration is not applied.

Here is what I got so far:

{
  description = "Base system for raspberry pi 4";
  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    nixos-generators = {
      url = "github:nix-community/nixos-generators";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    nixos-hardware.url = "github:NixOS/nixos-hardware/master";
  };

  outputs = { self, nixpkgs, nixos-generators, nixos-hardware, ... }: {
    nixosConfigurations.rpi4-sys = nixpkgs.lib.nixosSystem {
      modules = [
        {
          nixpkgs = {
            config.allowUnsupportedSystem = true;
            hostPlatform.system = "aarch64-linux";
            buildPlatform.system = "x86_64-linux";
          };
          fileSystems = {
            "/" = {
              device = "/dev/disk/by-label/NIXOS_SD";
              fsType = "ext4";
              options = [ "noatime" ];
            };
          };
          users.users.admin = {
            password = "admin";
            isNormalUser = true;
            extraGroups = [ "wheel" ];
          };
          system.stateVersion = "23.11";
        }
        nixos-hardware.nixosModules.raspberry-pi-4
      ];
    };

    packages.rpi4-sys = {
      sdcard = nixos-generators.nixosGenerate {
        system = "aarch64-linux";
        format = "sd-aarch64";
      };
    };
  };
}

I wanted it to be self-contained and simple as a start. Other scripts I found where either not complete, used some nix magic, I didn’t understand yet, I don’t want to delve into right now, or where rather complex in other ways.

I created the image with:

$ nix flake check
warning: Git tree '/proj' is dirty
trace: warning: system.stateVersion is not set, defaulting to 23.11. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.

$ nix build .#packages.rpi4-sys.sdcard
warning: Git tree '/proj' is dirty
trace: warning: system.stateVersion is not set, defaulting to 23.11. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.

Looking into the created image, the user admin was not created, but the /etc/fstab looks right. Also it complains about system.stateVersion not being set, but I set it there.

Is the structure not right? Is there some better information on this? Anything I could improve or any hints where to this stuff is better explained?

2

I found a configuration that seem to work for me, but I haven’t really understood why yet:

{
  description = "Base system for raspberry pi 4";
  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    nixos-generators = {
      url = "github:nix-community/nixos-generators";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    nixos-hardware.url = "github:NixOS/nixos-hardware/master";
  };

  outputs = { self, nixpkgs, nixos-generators, nixos-hardware, ... }: 
  {
    nixosModules = {
      system = {
        system.stateVersion = "23.11";
      };
      users = {
        users.users.admin = {
          password = "admin";
          isNormalUser = true;
          extraGroups = [ "wheel" ];
        };
      };
      cross = {
        nixpkgs.config.allowUnsupportedSystem = true;
        nixpkgs.hostPlatform.system = "aarch64-linux";
        nixpkgs.buildPlatform.system = "x86_64-linux";
      };
    };

    packages.aarch64-linux = {
      sdcard = nixos-generators.nixosGenerate {
        system = "aarch64-linux";
        format = "sd-aarch64";
        modules = [
          self.nixosModules.system
          self.nixosModules.users
          #self.nixosModules.cross # disabled, causes 'aarch64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-m64'' build errors
          #nixos-hardware.nixosModules.raspberry-pi-4 # disable, causes 'modprobe: FATAL: Module sun4i-drm not found in directory /nix/store/[...]-linux-6.1.21-1.20230405-modules/lib/modules/6.1.21' build errors
        ];
      };
    };
  };
}

So first what I learned: I used nix build and that does not take into account any configuration in nixosConfigurations. I found that out by reading Flakes - NixOS Wiki more closely. I also looked at what MyNixOS does, and it seems to use nixosModules to avoid duplicating code, if multiple outputs are used, I tried incorporating that idea, without requiring multiple files, because that makes it more difficult to follow IMO. And that seems to work now.

I disabled self.nixosModules.cross because it caused:

error: builder for '/nix/store/pss4gpq9jkbvj1k6d69gyq91h2ym2a8h-libhugetlbfs-aarch64-unknown-linux-gnu-2.23.drv' failed with exit code 2;
       last 10 log lines:
       > make: *** [Makefile:313: obj64/sys-elf_x86_64.o] Error 1
       > make: *** Waiting for unfinished jobs....
       > make: *** [Makefile:303: obj64/elflink.o] Error 1
       >     CC64 obj64/init.o
       > aarch64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-m64'
       > make: *** [Makefile:303: obj64/hugeutils.o] Error 1
       > aarch64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-m64'
       > make: *** [Makefile:303: obj64/version.o] Error 1
       > aarch64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-m64'
       > make: *** [Makefile:303: obj64/init.o] Error 1
       For full logs, run 'nix log /nix/store/pss4gpq9jkbvj1k6d69gyq91h2ym2a8h-libhugetlbfs-aarch64-unknown-linux-gnu-2.23.drv'.
error: 1 dependencies of derivation '/nix/store/iy7x7zyalcpibxzvffzsyx2f9pax1dsd-nvme-cli-aarch64-unknown-linux-gnu-2.6.drv' failed to build
error (ignored): error: cannot unlink '/tmp/nix-build-zip-aarch64-unknown-linux-gnu-3.0.drv-2': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-unzip-aarch64-unknown-linux-gnu-6.0.drv-1': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-libewf-aarch64-unknown-linux-gnu-20201230.drv-4/libewf-20201230': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-tcpdump-aarch64-unknown-linux-gnu-4.99.4.drv-2': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-efivar-aarch64-unknown-linux-gnu-38.drv-0/source/src': Directory not empty
error: 1 dependencies of derivation '/nix/store/6hnm09ymf3rznjziv8gi2dr2w88jbbrx-system-path.drv' failed to build
error (ignored): error: cannot unlink '/tmp/nix-build-xfsprogs-aarch64-unknown-linux-gnu-6.4.0.drv-1/xfsprogs-6.4.0': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-reiserfsprogs-aarch64-unknown-linux-gnu-3.6.27.drv-2/reiserfsprogs-3.6.27': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-linux-config-aarch64-unknown-linux-gnu-6.1.61.drv-1/linux-6.1.61/arch': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-sysstat-aarch64-unknown-linux-gnu-12.7.4.drv-3': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-rt5677-firmware.drv-3/source/build/linux/drivers': Directory not empty
error: 1 dependencies of derivation '/nix/store/mgg0byx6psphr055z21q8zq7vwyh3ixc-nixos-system-nixos-23.11.20231104.85f1ba3.drv' failed to build
error: 1 dependencies of derivation '/nix/store/0hg18h4cal82fhh3mzca00znlbgpcc47-ext4-fs.img.zst-aarch64-unknown-linux-gnu.drv' failed to build
error (ignored): error: cannot unlink '/tmp/nix-build-uboot-rpi_4_defconfig-aarch64-unknown-linux-gnu-2023.07.02.drv-2': Directory not empty
error (ignored): error: cannot unlink '/tmp/nix-build-uboot-rpi_3_defconfig-aarch64-unknown-linux-gnu-2023.07.02.drv-2': Directory not empty
error: 1 dependencies of derivation '/nix/store/52qv3wgrp6s53bddilgp6y8bh0g1vqhz-nixos-sd-image-23.11.20231104.85f1ba3-aarch64-linux.img-aarch64-unknown-linux-gnu.drv' failed to build

I got this from NixOS on ARM - NixOS Wiki where it states that this needs to be set for building images in flakes. It also sort of makes sense to me, because I am building on a x86_64 for aarch64. Not sure why it sets -m64 on a aarch64 compiler and where that happens.

With nixos-hardware.nixosModules.raspberry-pi-4 used, I get errors when building the kernel:

error: builder for '/nix/store/7fa15bggzq5fvl54h70pimr6zzpcr4q8-linux-6.1.21-1.20230405-modules-shrunk.drv' failed with exit code 1;
       last 10 log lines:
       >   copying dependency: /nix/store/l5z0h4z1g7lgsbai5bn8a1b4553r7jcp-linux-6.1.21-1.20230405-modules/lib/modules/6.1.21/kernel/drivers/message/fusion/mptscsih.ko.xz
       >   copying dependency: /nix/store/l5z0h4z1g7lgsbai5bn8a1b4553r7jcp-linux-6.1.21-1.20230405-modules/lib/modules/6.1.21/kernel/drivers/message/fusion/mptspi.ko.xz
       > root module: vmxnet3
       >   copying dependency: /nix/store/l5z0h4z1g7lgsbai5bn8a1b4553r7jcp-linux-6.1.21-1.20230405-modules/lib/modules/6.1.21/kernel/drivers/net/vmxnet3/vmxnet3.ko.xz
       > root module: vsock
       >   copying dependency: /nix/store/l5z0h4z1g7lgsbai5bn8a1b4553r7jcp-linux-6.1.21-1.20230405-modules/lib/modules/6.1.21/kernel/net/vmw_vsock/vsock.ko.xz
       > root module: simplefb
       >   builtin dependency: simplefb
       > root module: sun4i-drm
       > modprobe: FATAL: Module sun4i-drm not found in directory /nix/store/l5z0h4z1g7lgsbai5bn8a1b4553r7jcp-linux-6.1.21-1.20230405-modules/lib/modules/6.1.21
       For full logs, run 'nix log /nix/store/7fa15bggzq5fvl54h70pimr6zzpcr4q8-linux-6.1.21-1.20230405-modules-shrunk.drv'.
error: 1 dependencies of derivation '/nix/store/zx90np1pqymr0fyygwnm3zhydqkqja32-stage-1-init.sh.drv' failed to build
error: 1 dependencies of derivation '/nix/store/76yzlf0s9inmqz7dz9qh4b5c6a9m70ia-initrd-linux-6.1.21-1.20230405.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kkg332wd4cn5c0g5i6a2zl18j6y98i1w-nixos-system-nixos-23.11.20231104.85f1ba3.drv' failed to build
error: 1 dependencies of derivation '/nix/store/z7a752vzvg9b4l89hs40q20476ab5ngc-ext4-fs.img.zst.drv' failed to build
error: 1 dependencies of derivation '/nix/store/0bskv3ylj6nglisdj4cn8ymivi0ns82m-nixos-sd-image-23.11.20231104.85f1ba3-aarch64-linux.img.drv' failed to build

I guess it tries to install a module, which was not build, because raspberry pi 4 doesn’t need it. Why does this happen?

Update: I think I figured out. It seems like the sd-aarch64 format imports this config: https://github.com/NixOS/nixpkgs/blob/43a394b99a03d5ce665c8f7c051491ad2d7479c5/nixos/modules/installer/sd-card/sd-image-aarch64.nix, which contains a bunch of raspberry pi specfic stuff, that make it boot there. I would have expected that to be defined in nixos-hardware.nixosModules.raspberry-pi-4 instead, because the sd-aarch64 image format will not work on any other aarch64 system.

Anyway, removing the those two lines and the created image seems to boot. And now I just want to understand why… :wink:

Also why do I get the warning about system.stateVersion, and need to set it, I haven’t seen any other examples I peeked at doing that.

3

Here is another issue I noticed. For some reason ZFS is installed, boot.zfs.enabled is set to true, but setting it to false results in an error:

{
  description = "Base system for raspberry pi 4";
  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    nixos-generators = {
      url = "github:nix-community/nixos-generators";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = { self, nixpkgs, nixos-generators, ... }: 
  {
    nixosModules = {
      system = {
        boot.zfs.enabled = false;
        system.stateVersion = "23.11";
      };
      users = {
        users.users.admin = {
          password = "admin";
          isNormalUser = true;
          extraGroups = [ "wheel" ];
        };
      };
    };

    packages.aarch64-linux = {
      sdcard = nixos-generators.nixosGenerate {
        system = "aarch64-linux";
        format = "sd-aarch64";
        modules = [
          self.nixosModules.system
          self.nixosModules.users
        ];
      };
    };
  };
}

results in:

$ nix build .#packages.aarch64-linux.sdcard
warning: Git tree '/proj' is dirty
error: The option `boot.zfs.enabled' is read-only, but it's set multiple times. Definition values:
       - In `/nix/store/kcmipm57ph9bpzz8bs80iiijiwbyzwy3-source/nixos/modules/tasks/filesystems/zfs.nix': true
       - In `/nix/store/kcmipm57ph9bpzz8bs80iiijiwbyzwy3-source/flake.nix': false
(use '--show-trace' to show detailed location information)

I don’t know how (or where) to set boot.zfs.enabled without triggering this issue.

4

Maybe you need to set
boot.zfs.enabled = lib.mkDefault false;

5

boot.zfs.enabled is readonly because it doesn’t actually control anything; it’s a reflection of whether or not ZFS is in either boot.supportedFilesystems or boot.initrd.supportedFilesystems.

The installer profile includes ZFS. We have installers that don’t have ZFS, but the implementation is kind of silly. It’d nice to have a cleaner implementation for this stuff.

6

Like not having ZFS enabled in installers by default? :slight_smile: It was done like that for a long time and enabling it was actually pretty easy for users who needed it.

7

I think it’s good to have by default. Just should be easier to make one without it.

8

But that’s way harder currently since supportedFilesystems is a list “interface”, so you need to force it with another list that contains all except zfs (where you need to figure out what all means). Maybe we could expand it to a record like supportedFilesystems.zfs.enabled but that again is non-trivial backwards incompatible change affecting all NixOS users.

9

Right but what I just said is that it should be easier.

And you don’t have to do it as you described. I linked a module before that does it more easily.

Anyway this isn’t the thread to discuss how the ISO modules in nixpkgs should be changed.

10

If ZFS is something common for NixOS users, then I agree with just having the support for ZFS enabled, however it also installs and runs a daemon (zfs-zed). Which is only really necessary for systems that actually use ZFS. So therefore I would rather have zfs not enabled by default, if it requires an additional process, or have the process optional.

11

It’s only there by default within the installer. If you install a system without using ZFS, then ZFS won’t be in your system, even if it was in the installer.

12

Reading the module it suggests exactly the same thing so that is something to consider.

I agree, I might just file a PR some day.

13

Where in my flake do I explicitly use the installer? I just want to create a minimal sd-card image for rpi4.

14

Ah, my bad. The SD card modules are based on the installer modules, because it’s sort of meant to be a starting point. Flash the SD card with this “installation medium”, and then update the system to your actual preferred config. Or at least that was the original design.

15

nixos-generator has the sd-aarch64-installer format, which I don’t use.

16

Ah, I see. Ok the sd-aarch64 non-installer format still ends up including a fair amount of extra stuff, if you follow the imports from here. It’s very much not a minimal config, like it probably should be.

Eventually those imports lead you here, which is where the dependency on ZFS (and several other file systems) comes from.

1 Like
17

Thanks! I read the first link already (I have posted it in this thread before), and was very surprised how raspberry pi specific it is. I would have expected something much more general to the whole aarch64 platform. This stuff I would have expected in nixos-hardware.nixosModules.raspberry-pi-4.

The second link, I haven’t found before, but it does add a lot of unnecessary stuff. Is there a way to remove that include from my flake?

I guess I could just try implementing my own format…

18

That’s not really possible. ARM doesn’t have any kind of standard boot mechanism across boards, except for EBBR which is (mostly) only adopted on servers. See: Planning for a better NixOS on ARM (and other non-x86_64 systems)

You can probably add it to the disabledModules list (note, that section is titled “replace modules”, but you don’t actually have to provide a replacement).

1 Like
19

Well that is what the nixos-hardware is for, I thought. I was surprised that the sdcard format also had aarch64 in its name.

20

I think calling the format “sdcard” is probably a misnomer. There’s nothing special about SD cards; if it was platform agnostic, the format would work on any kind of block device. Realistically, it’s the rpi format. And yes, if you read the post I linked, we would very much like to remove the sd-image things from NixOS and keep these device-specific configurations in nixos-hardware.

1 Like