I’ve been struggling for a few weeks to get NixOS 21.05 running on a Raspberry Pi 3 on a BTRFS root.
I’ve detailed my process at r/NixOS but thought I would also ask here.
I’ve converted other Pis to BTRFS, which usually entails making a BTRFS partition with the necessary structure, then modifying the boot partition with something like root=UUID=ABCD rootflags=subvol=@root.
So far, I’ve been able to copy the root partition from the NixOS sd_image to a USB key, convert it to BTRFS, and boot to it (using SD card for boot partition, USB as root), but when I try to clone the USB image back to the SD card’s root partition, I get boot-time errors about “unknown filesystem”.
Does anyone have any ideas how I could modify the boot partition to recognize BTRFS? Or perhaps if that’s not the problem at all?
I was able to nixos-rebuild switch, which seemed really encouraging, but after a reboot I found that the u-boot-rpi3.bin is identical to the one from the original SD image (md5sum matches), so apparently that didn’t do anything. (EDIT: bootcode.bin is also unchanged.)
For my next attempt, I put this into btrfs_sd_image.nix and ran the command that is commented out at the top, with the plan to mount the resulting image and see if the u-boot-rpi3.bin has been modified or not.
# nix-build '<nixpkgs/nixos>' -I nixos-config=./btrfs_sd_image.nix -A config.system.build.sdImage
{ pkgs, ... }:
let uboot = pkgs.uboot.override { extraConfig = "CONFIG_FS_BTRFS=y"; };
in {
imports = [
<nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64.nix>
];
}
Today tried adding boot.initrd.availableKernelModules = ["btrfs"]; to the btrfs_sd_image.nix and again found that the resulting FIRMWARE partition is identical to the default SD image – it’s having no effect at all.
Does anyone know why these changes aren’t affecting the sd image’s boot partition? What am I missing here?
@c00w – if you see this by any chance, is there any chance you could point me in the right direction as to how I’d use your library to create a BTRFS-based sd image? Thanks for your work!
Heh - I actually want one of these - I had one of these working before the boot config was moved to the main partition in sd-image.nix, and that’s why I made the PR (which depressingly still hasn’t gotten merged).
My thought is probably to add a couple options to the sd-image file. The first one is to replace the ext4 fs image with a btrfs root fs image (using that PRs code). The second one is to either merge the firmware and boot partition (and have them be ext4), or retain the separate partitions and add a third one that is the btrfs code.
While I have zero issues with trying to teach the uboot code to understand BTRFS (and it’s cool if you can get that working), I’ve found that debugging raspberry pi images can be extremely painful, so I tend to try and not do anything complicated when making changes.
Let me see if I can get this working - I’ll try and put up a rough draft sd image.
After a bit of looking and tweaking, I think I’ve figured out that this strategy uses extlinux-conf-builder.sh to populate boot files onto the FAT / firmware partition, which is kind of unfortunate since it looks like u-boot should be able to read the BTRFS root partition by itself (as posted above):
I’m still trying to figure out booting from a compressed @boot subvolume. I’ve made the necessary u-boot tweaks to boot directly from the subvolume, which I’m psyched about, but compressing either @boot or @nix seems to break things. Boots the first time, but after a nixos-install --root / goes into a boot loop.
I think the current iteration is pretty satisfactory. It has options to put the boot files on FIRMWARE, which allows booting from a data=DUP BTRFS root (maybe handy for SD cards? I image should also work if you had a BTRFS RAID setup), or to boot directly from a @boot subvolume if you don’t plan keep data=single.
Never figured out this issue with zstd compression on @boot screwing up u-boot.