RPi4 can't USB boot from BTRFS with rpi kernel

Hi all,

I’ve detailed several of my adventures getting a Pi3 to boot NixOS from a compressed BTRFS subvolume, and now I’m working on doing the same for a Pi4.

By default, using the default NixOS kernel (not the RPi kernel), it hangs at looking for /dev/disk/by-label/NIXOS. I figured out a few missing kernel modules that allow the boot process to succeed.

Unfortunately, using that process, some of the Pi-specific stuff like missing, for example /dev/gpiomem doesn’t exist, which I use for home automation purposes. I presume using the Pi4 kernel should fix this.

Unfortunately, when I go back to using the pi4 kernel, I go back to being unable to boot from my USB drive (fails at the same point as above looking for /dev/disk/by-label/NIXOS, even with the kernel modules from above included).

I even tried importing all-hardware.nix to see if there were other kernel modules that could be needed (requiring this workaround), but with the pi4 kernel it still fails to find /dev/disk/by-label/NIXOS and therefore fails to boot. It seems to be finding and reading things from the @boot subvolume, so at least that is working.

So I’m in a bit of a predicament. I guess I’d rather use the stock kernel if anyone knows how I can get /dev/gpiomem to appear on a Pi4. But if that’s not feasible (or requires rebuilding the kernel every upgrade which would be a bit of a pain), I’m wondering if anyone has any ideas why the Pi kernel is having trouble booting from a compressed BTRFS subvolume, since I’m able to do so from Raspbian.

Perhaps I need to use the raspi bootloader and specify the initrd in config.txt? Currently using a slightly customized u-boot to get the BTRFS and ZSTD support.

May have been a problem with my filesystem, unclear. rpi4 on compressed BTRFS root won't boot · Issue #640 · NixOS/nixos-hardware · GitHub