Raspberry Pi NixOS on BTRFS Root?

Help
#1

Hi all,

I’ve been struggling for a few weeks to get NixOS 21.05 running on a Raspberry Pi 3 on a BTRFS root.

I’ve detailed my process at r/NixOS but thought I would also ask here.

I’ve converted other Pis to BTRFS, which usually entails making a BTRFS partition with the necessary structure, then modifying the boot partition with something like root=UUID=ABCD rootflags=subvol=@root.

So far, I’ve been able to copy the root partition from the NixOS sd_image to a USB key, convert it to BTRFS, and boot to it (using SD card for boot partition, USB as root), but when I try to clone the USB image back to the SD card’s root partition, I get boot-time errors about “unknown filesystem”.

Does anyone have any ideas how I could modify the boot partition to recognize BTRFS? Or perhaps if that’s not the problem at all?

Many thanks in advance!

Raspberry Pi 3B+ ethernet stopped working
#2

I wonder if I can somehow customize the uboot with the extraConfig parameter and add CONFIG_FS_BTRFS=y?

#3

Maybe you need something like this: uboot nix derivation for Rock64 with CONFIG_BAUDRATE=115200 · GitHub.

#4

Well, after a lot of googling, I put the following into my configuration.nix:

nixpkgs.overlays = [ (self: super: {
  uboot = super.uboot.override { extraConfig = "CONFIG_FS_BTRFS=y"; };
}) ];

I was able to nixos-rebuild switch, which seemed really encouraging, but after a reboot I found that the u-boot-rpi3.bin is identical to the one from the original SD image (md5sum matches), so apparently that didn’t do anything. (EDIT: bootcode.bin is also unchanged.)

For my next attempt, I put this into btrfs_sd_image.nix and ran the command that is commented out at the top, with the plan to mount the resulting image and see if the u-boot-rpi3.bin has been modified or not.

# nix-build '<nixpkgs/nixos>' -I nixos-config=./btrfs_sd_image.nix -A config.system.build.sdImage
{ pkgs, ... }:
let uboot = pkgs.uboot.override { extraConfig = "CONFIG_FS_BTRFS=y"; };
in {
    imports = [
        <nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64.nix>
    ];
}
#5

Yeah, that didn’t change anything. The entire FIRMWARE partition is identical to the original. What am I doing wrong here?

#6

Today tried adding boot.initrd.availableKernelModules = ["btrfs"]; to the btrfs_sd_image.nix and again found that the resulting FIRMWARE partition is identical to the default SD image – it’s having no effect at all.

Does anyone know why these changes aren’t affecting the sd image’s boot partition? What am I missing here?

#7

Also tried adding fileSystems = lib.mkForce {} with a few BTRFS subvolumes – nothing, it still ends up with a single ext4 root partition. ¯\(ツ)/¯

#8

I’ve been watching this thread since I had been considering an RPI4. Maybe share the entire hardware.nix and configuration.nix?

#9

When it comes to trying to build the SD image, there’s not much that I’ve left out (other than the specific BTRFS subvolumes I like):

# nix-build '<nixpkgs/nixos>' -I nixos-config=./btrfs_sd_image.nix -A config.system.build.sdImage
{ lib, pkgs, ... }:
{
	imports = [
		<nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64.nix>
	];
	boot.initrd.availableKernelModules = [
		"btrfs"
	];

# fileSystems = lib.mkForce {
# 	"/boot" = {
# 		device = "/dev/disk/by-label/FIRMWARE";
# 		fsType = "vfat";
# 	};
# 	"/" = {
# 		device = "/dev/disk/by-label/NIXOS_SD";
# 		fsType = "ext4";
# 	};
# };

	fileSystems = lib.mkForce {
		"/" = {
			device = "/dev/disk/by-label/NIXOS_SD";
			fsType = "btrfs";
			options = [ "noatime" "ssd_spread" "discard=async" "compress-force=zstd" "autodefrag" "subvol=@" ];
		};
		"/boot" = {
			device = "/dev/disk/by-label/NIXOS_SD";
			fsType = "btrfs";
			options = [ "noatime" "ssd_spread" "discard=async" "compress-force=zstd" "autodefrag" "subvol=@boot" ];
		};
		"/var" = {
			device = "/dev/disk/by-label/NIXOS_SD";
			fsType = "btrfs";
			options = [ "noatime" "ssd_spread" "discard=async" "compress-force=zstd" "autodefrag" "subvol=@var" ];
		};
		"/home" = {
			device = "/dev/disk/by-label/NIXOS_SD";
			fsType = "btrfs";
			options = [ "noatime" "ssd_spread" "discard=async" "compress-force=zstd" "autodefrag" "subvol=@home" ];
		};
		"/nix" = {
			device = "/dev/disk/by-label/NIXOS_SD";
			fsType = "btrfs";
			options = [ "noatime" "ssd_spread" "discard=async" "compress-force=zstd" "autodefrag" "subvol=@nix" ];
		};
		"/swap" = {
			device = "/dev/disk/by-label/NIXOS_SD";
			fsType = "btrfs";
			options = [ "noatime" "ssd_spread" "discard=async" "compress-force=zstd" "autodefrag" "subvol=@swap" ];
		};
		"/.snapshots" = {
			device = "/dev/disk/by-label/NIXOS_SD";
			fsType = "btrfs";
			options = [ "noatime" "ssd_spread" "discard=async" "compress-force=zstd" "subvol=@snapshots" ];
		};

	};

	swapDevices = [ { device = "/swap/swapfile"; size = 1024; } ];
}
#10

Newest attempt with GitHub - Robertof/nixos-docker-sd-image-builder: Build custom SD images of NixOS for your Raspberry Pi (or any other supported AArch64 device) in 5-20 minutes. – also no luck.

#11

Still working on this.

Some relevant links:

F2FS example:

Another F2FS example:

BTRFS example by @c00w:

@c00w – if you see this by any chance, is there any chance you could point me in the right direction as to how I’d use your library to create a BTRFS-based sd image? Thanks for your work!

#12

Heh - I actually want one of these - I had one of these working before the boot config was moved to the main partition in sd-image.nix, and that’s why I made the PR (which depressingly still hasn’t gotten merged).

My thought is probably to add a couple options to the sd-image file. The first one is to replace the ext4 fs image with a btrfs root fs image (using that PRs code). The second one is to either merge the firmware and boot partition (and have them be ext4), or retain the separate partitions and add a third one that is the btrfs code.

While I have zero issues with trying to teach the uboot code to understand BTRFS (and it’s cool if you can get that working), I’ve found that debugging raspberry pi images can be extremely painful, so I tend to try and not do anything complicated when making changes.

Let me see if I can get this working - I’ll try and put up a rough draft sd image.

#13

Got it working - Code is at ~c00w/nixpkgs: / - sourcehut git and ~c00w/useful-nixos-aarch64: / - sourcehut git

sd image for testing is at https://download.daedrum.net/file/useful-nixos-aarch64/pi4bbtrfs.img
root password is pi4bbtrfs
I booted this locally and it came up cleanly.

The nixpkg repo it’s using is a stack of commits as I fiddled with things, but I can probably clean it up for submission to nixpkgs.

1 Like
#14

Thank you so much! I’ve been looking forward to some time to sit down and tinker with this.

I’m trying to get it working with GitHub - Robertof/nixos-docker-sd-image-builder: Build custom SD images of NixOS for your Raspberry Pi (or any other supported AArch64 device) in 5-20 minutes. – building now, we’ll see!

Thanks again.

#15

After a bit of looking and tweaking, I think I’ve figured out that this strategy uses extlinux-conf-builder.sh to populate boot files onto the FAT / firmware partition, which is kind of unfortunate since it looks like u-boot should be able to read the BTRFS root partition by itself (as posted above):

Can't boot into new configuration after nix-rebuild, raspberry pi3 B
#16

I’ve continued tweaking and working on this and have made some pretty good progress: https://github.com/n8henrie/nixos-btrfs-pi

I’m still trying to figure out booting from a compressed @boot subvolume. I’ve made the necessary u-boot tweaks to boot directly from the subvolume, which I’m psyched about, but compressing either @boot or @nix seems to break things. Boots the first time, but after a nixos-install --root / goes into a boot loop.

BTRFS Pi won't boot from compressed subvolume
#17

For context, here’s the error I get on bootup, just after it looks like it’s going to work and Retrieving file: /@boot/extlinux/../nixos/.../linux:

ERROR: arch-specific fdt fixup failed
 - must RESET the board to recover.

FDT creation failed!
resetting ...

Immediately after I get a blue screen and a reboot.

BTRFS Pi won't boot from compressed subvolume
#18

I’ve continued to work on this project (with some additional info on my biggest hangup at BTRFS Pi won't boot from compressed subvolume - #5 by n8henrie).

I’ve rewritten the project and encapsulated almost everything into a flake so if any reader are interested they should be able to build the project themselves: https://github.com/n8henrie/nixos-btrfs-pi

I’ve also configured a GitHub Action to automatically build and upload the resulting image as an artifact (which will expire after a few weeks IIRC): Workaround for https://github.com/actions/upload-artifact/issues/92 · n8henrie/nixos-btrfs-pi@3c1e629 · GitHub

1 Like
#19

Still continuing to tinker, but at a slower rate.

I think the current iteration is pretty satisfactory. It has options to put the boot files on FIRMWARE, which allows booting from a data=DUP BTRFS root (maybe handy for SD cards? I image should also work if you had a BTRFS RAID setup), or to boot directly from a @boot subvolume if you don’t plan keep data=single.

Never figured out this issue with zstd compression on @boot screwing up u-boot.

Current iteration: GitHub - n8henrie/nixos-btrfs-pi at 0b33a340eea39aa8642ef9f6223aeebbf3d16316

Hosted by Flying Circus.