I’ve detailed some of my adventures getting a NixOS-Pi3 running on BTRFS-root with subvolumes:
My goal is to get it booting directly from a compressed subvolume (as opposed to booting from the BTRFS root, or from the FAT partition, etc.) – mostly just because I think it should be possible.
However, as mentioned I am still having trouble with a specific scenario:
- the Pi boots / works great from BTRFS root
- the Pi boots / works great from BTRFS in a subvolume
- If I add compression (zstd), the initial SD card boots (I imagine because it’s not using BTRFS compression at this point, since I manually create the subvolumes and copy contents in a script), but after
nixos-install it goes into a boot loop, with the error:
ERROR: arch-specific fdt fixup failed
- must RESET the board to recover.
FDT creation failed!
resetting ...
It looks like u-boot supports zstd decompression, and I have the following in my u-boot config:
CONFIG_CMD_BTRFS=y
CONFIG_ZSTD=y
Does anyone know why compression would prevent me from booting the second time (after essentially nixos-install --root / && reboot)? It seems to find and read the initrd just fine (I can post some pics of the boot screen if it would be helpful).
I tried the copyKernels setting thinking that may do the trick, but no change in behavior.
Thanks for any help or suggestions. I’ve been tinkering with this for nearly a year now and have just run out of ideas.
A few screenshots of some example boot error screens (right before it reboots and cycles with the same error messages).
OCR of the text (apologize for any OCR errors, I’ve edited it briefly to fix up a few things, but thought it might help future searchers):
Scanning mmc 0:2...
Found /@boot/extlinux/extlinux.conf
Retrieving file: /@boot/extlinux/extlinux.conf
2938 bytes read in 27 ms (105.5 KiB/s)
1 NixOS Default
2 NixOS Configuration 4 (2022-02-24 21:34 22.05pre356135.7f9b6c2babf)
3 NixOS Configuration 3 (2022-02-25 03:59 22.05pre356435.7/9b6eZbabf)
4: NixOS Configuration 2 (2022-02-24 21:34 22.05pre356435.7/9b6c2babr)
5: NixOS
Enter choice: 1: NixOS - Default
Retrieving file: /@boot/extlinux/..nixos/zhbharzhzqga617uc37uk0g6q1bx891u-initrd-linux-5.10.101-initrd
20588420 bytes read in 1910 ms (10.3 MiB/s)
Retrieving file: /@boot/extlinux/..nixos/6c1m6j0k1gmjrmikOyyizariiry?Zlqr-linux-5.10.101-Image
50629120 bytes read in 6863 ms (7 MiB/s)
append: init-mix/store/pikl8ju4077iaix36n8mifzu6l6rcajz-nixos-system-nixpi-22.05p re356435-7/9b6eZbabf/init console=tty1 console=ttyAMAO console=ttyS0, 1115200 root= LABEL=-NIXOS_SD rootfstype=btrfs rootflags=subvol=@ loglevel=4
Retrieving file: /@boot/extlinux/../nixos/6c16j0k1gmjrmikOyyizari1ry72lqr-linux-5.10.101-dtbs/broadcon/bcn2837-rpi-3-b.dtb
14310 bytes read in 66 ms (210.9 KiB/s)
Moving Image from 0x80000 to 0x200000, end=32f0000
## Flattened Device Tree blob at 03700000
Booting using the fdt blob at 0x3700000
Using Device Tree in place at 0000000003700000 end 00000000037067e5
fdt_find_or_add_subnode: memory: FDT_ERR_BADSTRUCTURE
ERROR: arch-specific fdt fixup failed
- must RESET the board to recover.
FDT creation failed!
resetting ...
I don’t think nixos actually updates your u-boot. So unless you’re booting freshly built SD images every time, I don’t think adding CONFIG_ZSTD=y to the u-boot package will do anything. Try replacing the u-boot binary manually or making a new SD image
Thanks for your thoughts and response.
I am booting a freshly built SD image to start with, and I’m not replacing the firmware partition (or its u-boot-rpi3.bin file) at any point, so I assumed it would continue with the firmware built with BTRFS and ZSTD support.
It does seem to have references to BTRFS and ZSTD that don’t exist in, for example, the rpi4 u-boot file that also gets included:
$ grep -ac BTRFS /tmp/inspect1/u-boot-rpi3.bin
62
$ grep -ac ZSTD /tmp/inspect1/u-boot-rpi3.bin
2
$ grep -ac BTRFS /tmp/inspect1/u-boot-rpi4.bin
0
$ grep -ac ZSTD /tmp/inspect1/u-boot-rpi4.bin
0
As a test, I just ran the below and rebooted, and it failed to reboot – which I assume is because I should have overridden extraConfig somehow.
# nix build --file '<nixpkgs/nixos>' pkgs.ubootRaspberryPi3_64bit
# cp result/u-boot.bin /firmware/u-boot-rpi3.bin
# reboot
Indeed, when I comment out my overlay:
$ grep -ac ZSTD /tmp/inspect1/u-boot-rpi3.bin
0
$ grep -ac BTRFS /tmp/inspect1/u-boot-rpi3.bin
0
So it seems like the overlay is working, at least for the initial SD image. I don’t think anything gets overwritten there by default after a nixos-install, but I’ll check that out next.
Never figured this out. Most recent iteration just disables COW on the boot partition: GitHub - n8henrie/nixos-btrfs-pi at 0b33a340eea39aa8642ef9f6223aeebbf3d16316
