Why can't I switch aarch64 system on a Rapberry Pi 4?

I’m trying to apply a system configuration on an SD card I created myself from this configuration which boots fine(despite agenix decryption errors due to new host private key) but for some reason I cannot switch to the correct config.

When I try to build my flake config as root it fails:

root@leliel:/etc/nixos/ > nixos-rebuild boot --flake /etc/nixos#leliel --impure                                                
error:
       … while updating the lock file of flake 'git+file:///etc/nixos?ref=refs/heads/master&rev=55618b80277b8eaf012d542bad8472a1f5f55c9b'

       error: path '/nix/store/84ab7r04568279h0d0vll47a7h6bky42-source' is a symlink

But when I switch the same system as non-root it at least builds:

jakubgs@leliel:~/nixos/ > nixos-rebuild build --flake .\#leliel --impure
building the system configuration...
warning: Git tree '/home/jakubgs/nixos' is dirty
jakubgs@leliel:~/nixos/ > readlink result
/nix/store/3wqqp6hgj8kzmdynglpf0qvd8p1z9crl-nixos-system-leliel-24.11.20241205.4dc2fc4

But when I try to switch or boot as root:

jakubgs@leliel:~/nixos/ > sudo /nix/store/3wqqp6hgj8kzmdynglpf0qvd8p1z9crl-nixos-system-leliel-24.11.20241205.4dc2fc4/bin/switch-to-configuration switch 
activating the configuration...
[agenix] creating new generation in /run/agenix.d/3
[agenix] decrypting secrets...
decrypting '/nix/store/dkbvl1cnzax9l1l27hd4683xwlxzp672-pass-hash.age' to '/run/agenix.d/3/hosts/users/jakubgs/pass-hash'...
decrypting '/nix/store/227pcd2wirc246h4lmavs979idvzqbzm-server.key.age' to '/run/agenix.d/3/service/landing/server.key'...
decrypting '/nix/store/xbv4alprw01pspj1qrdl4r3aqvgxjbyh-wifi.age' to '/run/agenix.d/3/service/wifi'...
decrypting '/nix/store/mqrmp9cjbgznvw4qfcjwqlvs4yc2myr5-magi.age' to '/run/agenix.d/3/service/zerotier/magi'...
[agenix] symlinking new secrets to /run/agenix (generation 3)...
[agenix] removing old secrets (generation 2)...
[agenix] chowning...
setting up /etc...
reloading user units for jakubgs...
restarting sysinit-reactivation.target

I don’t see the system generation added to the profile folder:

jakubgs@leliel:~/nixos/ > sudo ls -l /nix/var/nix/profiles/                                                                                             
total 8
lrwxrwxrwx 1 root root   43 Dec 13  2024 default -> /nix/var/nix/profiles/per-user/root/profile
drwxr-xr-x 3 root root 4096 Jan  1  1970 per-user
lrwxrwxrwx 1 root root   13 Jan  1  1970 system -> system-1-link
lrwxrwxrwx 1 root root   86 Jan  1  1970 system-1-link -> /nix/store/akx2159g42w6swq0894rpwfz6w9ajl8l-nixos-system-leliel-24.11.20241205.4dc2fc4

Notice the different hash, and no system-2-link. I’m a bit confused as to how I’m supposed to make this work. This used to work fine on 24.05.

I wish I could just make the SD card image myself, but I can’t since I can’t know the host private key in advance which means I can’t re-encrypt agenix secrets in advance for the SD card image. I don’t get how to square that circle.

I can see the /boot/extlinux/extlinux.conf file is updated:

jakubgs@leliel:~/nixos/ > sudo grep -A4 'LABEL nixos-default' /boot/extlinux/extlinux.conf
LABEL nixos-default
  MENU LABEL NixOS - Default
  LINUX ../nixos/m98j6k2mqns771l9f3gbrsvs6h0sxq0g-linux-6.6.63-Image
  INITRD ../nixos/d7qjcj8q4kcch90wycdda93c667dqjch-initrd-linux-6.6.63-initrd
  APPEND init=/nix/store/3wqqp6hgj8kzmdynglpf0qvd8p1z9crl-nixos-system-leliel-24.11.20241205.4dc2fc4/init cma=64M console=tty0 loglevel=4

But when I reboot it still boots into the previous system config that has incorrect agenix secrets.

And in the boot logs I can only see the NixOS - Default option, no second option.

Before I used boot.loader.generic-extlinux-compatible config without issues, but since this PR was merged and is part of NixOS 24.11:

It is recommended to use boot.loader.generic-extlinux-compatible, which I did, but now I can’t properly boot into a new system config. I’m really confused.

I’m using GitHub - NixOS/nixos-hardware: A collection of NixOS modules covering hardware quirks. for my RPI4 without issue, maybe take a look at that (or use it if you don’t mind an extra dependency).

1 Like

You could generate a key in advance and copy it to the SD card after flashing and manually expanding the image.

That’s true, but I wish I could provide the private key via NixOS config rather than by manually modifying the image after it has been built, but it does seem simpler than booting the broken image and then fixing it afterwards.

Yeah, i will try that, but it seems to use the same config for booting as I do:

I tried by adding channels.hardware.nixosModules.raspberry-pi-4 to my imports but it still fails to boot the correct system.

My contents of /boot look like this:

 > ls -l /boot 
total 23468
-rwxr-xr-x 1 root root     512 Dec 14  2024 armstub8-gic.bin
-rwxr-xr-x 1 root root   32499 Dec 14  2024 bcm2710-rpi-2-b.dtb
-rwxr-xr-x 1 root root   34691 Dec 14  2024 bcm2710-rpi-3-b.dtb
-rwxr-xr-x 1 root root   35326 Dec 14  2024 bcm2710-rpi-3-b-plus.dtb
-rwxr-xr-x 1 root root   32278 Dec 14  2024 bcm2710-rpi-cm3.dtb
-rwxr-xr-x 1 root root   33668 Dec 14  2024 bcm2710-rpi-zero-2.dtb
-rwxr-xr-x 1 root root   33668 Dec 14  2024 bcm2710-rpi-zero-2-w.dtb
-rwxr-xr-x 1 root root   56116 Dec 14  2024 bcm2711-rpi-400.dtb
-rwxr-xr-x 1 root root   56112 Dec 14  2024 bcm2711-rpi-4-b.dtb
-rwxr-xr-x 1 root root   56596 Dec 14  2024 bcm2711-rpi-cm4.dtb
-rwxr-xr-x 1 root root   53399 Dec 14  2024 bcm2711-rpi-cm4s.dtb
-rwxr-xr-x 1 root root   52476 Dec 14  2024 bootcode.bin
-rwxr-xr-x 1 root root     946 Dec 14  2024 config.txt
drwxr-xr-x 2 root root    4096 Jul  6 06:26 extlinux
-rwxr-xr-x 1 root root    3230 Dec 14  2024 fixup4cd.dat
-rwxr-xr-x 1 root root    5456 Dec 14  2024 fixup4.dat
-rwxr-xr-x 1 root root    8450 Dec 14  2024 fixup4db.dat
-rwxr-xr-x 1 root root    8454 Dec 14  2024 fixup4x.dat
-rwxr-xr-x 1 root root    3230 Dec 14  2024 fixup_cd.dat
-rwxr-xr-x 1 root root    7325 Dec 14  2024 fixup.dat
-rwxr-xr-x 1 root root   10295 Dec 14  2024 fixup_db.dat
-rwxr-xr-x 1 root root   10295 Dec 14  2024 fixup_x.dat
drwxr-xr-x 4 root root    4096 Dec 14  2024 nixos
-rwxr-xr-x 1 root root  811868 Dec 14  2024 start4cd.elf
-rwxr-xr-x 1 root root 3756808 Dec 14  2024 start4db.elf
-rwxr-xr-x 1 root root 2259328 Dec 14  2024 start4.elf
-rwxr-xr-x 1 root root 3006952 Dec 14  2024 start4x.elf
-rwxr-xr-x 1 root root  811868 Dec 14  2024 start_cd.elf
-rwxr-xr-x 1 root root 4828680 Dec 14  2024 start_db.elf
-rwxr-xr-x 1 root root 2983520 Dec 14  2024 start.elf
-rwxr-xr-x 1 root root 3730600 Dec 14  2024 start_x.elf
-rwxr-xr-x 1 root root  582216 Dec 14  2024 u-boot-rpi3.bin
-rwxr-xr-x 1 root root  650960 Dec 14  2024 u-boot-rpi4.bin

And my extlinux config like this:

 > grep -A3 LABEL /boot/extlinux/extlinux.conf
LABEL nixos-default
  MENU LABEL NixOS - Default
  LINUX ../nixos/yvs0kaw6k5i57255rbjmnk9w7x09m34f-linux-rpi-6.6.31-stable_20240529-Image
  INITRD ../nixos/2kfpl56f42yv56m60y0snlamff60rc89-initrd-linux-rpi-6.6.31-stable_20240529-initrd
  APPEND init=/nix/store/z5yb5mklbfhrrn8zbxmdd1zlgkr1fr46-nixos-system-leliel-24.11.20241205.4dc2fc4/init cma=64M console=tty0 loglevel=4
--
LABEL nixos-2-default
  MENU LABEL NixOS - Configuration 2-default (2024-12-14 12:04 - 24.11.20241205.4dc2fc4)
  LINUX ../nixos/yvs0kaw6k5i57255rbjmnk9w7x09m34f-linux-rpi-6.6.31-stable_20240529-Image
  INITRD ../nixos/2kfpl56f42yv56m60y0snlamff60rc89-initrd-linux-rpi-6.6.31-stable_20240529-initrd
  APPEND init=/nix/store/z5yb5mklbfhrrn8zbxmdd1zlgkr1fr46-nixos-system-leliel-24.11.20241205.4dc2fc4/init cma=64M console=tty0 loglevel=4
--
LABEL nixos-1-default
  MENU LABEL NixOS - Configuration 1-default (1970-01-01 01:00 - 24.11.20241205.4dc2fc4)
  LINUX ../nixos/m98j6k2mqns771l9f3gbrsvs6h0sxq0g-linux-6.6.63-Image
  INITRD ../nixos/zzyd6fxz6d5wgkikcgrxcx506bas298q-initrd-linux-6.6.63-initrd
  APPEND init=/nix/store/akx2159g42w6swq0894rpwfz6w9ajl8l-nixos-system-leliel-24.11.20241205.4dc2fc4/init console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0 cma=64M console=tty0 nohibernate loglevel=7

And yet the system that boots by default is the original one built with the SD card image, which fails to load any agenix secrets:

 > sudo find -L /var/run/agenix.d -type f | wc -l
0

Considering that the default entry is the same as nixos-2-default maybe a bootloader setting is funny (or the extlinux.conf doesn’t DEFAULT to nixos-default).
For reference my /boot looks like this

ls /boot
╭───┬────────────────┬──────┬─────────┬─────────────╮
│ # │      name      │ type │  size   │  modified   │
├───┼────────────────┼──────┼─────────┼─────────────┤
│ 0 │ /boot/nixos    │ dir  │ 4.0 KiB │ a month ago │
│ 1 │ /boot/extlinux │ dir  │ 4.0 KiB │ 2 hours ago │
╰───┴────────────────┴──────┴─────────┴─────────────╯
1 Like

I realized the vfat partition might not be used, and when I unmounted /boot I realized there’s another /boot there:

jakubgs@leliel ~
 > sudo umount /boot

jakubgs@leliel ~
 > ls -l /boot 
total 8
drwxr-xr-x 2 root root 4096 Dec 15  2024 extlinux
drwxr-xr-x 3 root root 4096 Dec 15  2024 nixos

And when re-ran sudo nixos-rebuild boot, then the /boot on my root partition was updated, and then the system booted into correct version:

 > readlink /run/booted-system
/nix/store/z5yb5mklbfhrrn8zbxmdd1zlgkr1fr46-nixos-system-leliel-24.11.20241205.4dc2fc4

Now the question is, do I even need the vfat partition with firmware? If the kernel and system is loaded directly from the root partition? If it is necessary does it ever have to be mounted or not?

If you flashed the new bootloader to the EEPROM, which is u-boot since some time (before they had a rpi specific way to doing things IIRC) then no you don’t as u-boot can read ext4 without issues.

EDIT: i still have the firmware on a separate FAT partition but don’t use it as /boot, not entirely sure if u-boot still needs it for the firmware blobs though.

1 Like

I don’t recall ever doing that, but it does sound plausible based on what you described.

I need to research this. Thanks for your suggestions, very helpful.

This is my info about EEPROM bootloader:

 > sudo vcgencmd bootloader_version
Apr 16 2020 18:11:26
version a5e1b95f320810c69441557c5f5f0a7f2460dfb8 (release)
timestamp 1587057086
update-time 0
capabilities 0x00000000

Googling for a5e1b95f320810c69441557c5f5f0a7f2460dfb8 gives me nothing. But it might be the u-boot one.

It’s definitely not the latest. For reference:

[nix-shell:~]$ doas vcgencmd bootloader_version
doas (zimward@shila) password:
2024/05/17 12:26:58
version 72caf66729df313801bcefe9b1ff7099c71bb5ce (release)
timestamp 1715945218
update-time 1717572516
capabilities 0x0000007f

your update-time suggests that it’s the factory flashed version, so updating it won’t hurt.

1 Like

Interesting, there’s apparently a UEFI firmware called edk2 for Raspberry Pi 4:

I use edk2 fork on my devices with Rockchip RK3588S, where it works great. Maybe it’s worth a try. Thanks again for the help!

The usuable files can be found here: GitHub - pftf/RPi4: Raspberry Pi 4 UEFI Firmware Images

1 Like