An update, and call for help:
- I have a Raspberry Pi 4 (
rpifour1) booting u-boot and then mainline kernel.
- I have another RPi4 (
rpifour2) that directly netboots mainline (via atftpd) the first one.
- I can toggle the power to
rpifour2 via a script and the home-assistant api so I can iterate easily
-
rpifour1 monitors the serial output from rpifour2 so I can see the kernel boot log without HDMI, can copy/paste/etc.
When the second one boots, the kernel and initrd are loaded, the kernel starts to boot, and then it fails to mount root and I get this error:
# snipped
[ 2.634172] Key type .fscrypt registered
[ 2.638205] Key type fscrypt-provisioning registered
[ 2.648414] fe201000.serial: ttyAMA0 at MMIO 0xfe201000 (irq = 24, base_baud = 0) is a PL011 rev2
[ 2.657692] serial serial0: tty port ttyAMA0 registered
[ 2.679032] raspberrypi-firmware soc:firmware: Attached to firmware from 2020-11-30T22:12:08
[ 2.760065] dwc2 fe980000.usb: supply vusb_d not found, using dummy regulator
[ 2.767544] dwc2 fe980000.usb: supply vusb_a not found, using dummy regulator
[ 2.875860] dwc2 fe980000.usb: EPs: 8, dedicated fifos, 4080 entries in SPRAM
[ 2.898010] sdhci-iproc fe300000.sdhci: allocated mmc-pwrseq
[ 2.957512] mmc0: SDHCI controller on fe300000.sdhci [fe300000.sdhci] using PIO
[ 2.973255] ALSA device list:
[ 2.976417] No soundcards found.
# snipped noisy mmc controller fail
[ 3.096075] VFS: Cannot open root device "nfs" or unknown-block(0,255): error -6
[ 3.103706] Please append a correct "root=" boot option; here are the available partitions:
# snipped list of memory partitions
[ 3.212496] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,255)
[ 3.221086] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.10.15 #1-NixOS
[ 3.227723] Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
[ 3.233652] Call trace:
[ 3.236154] dump_backtrace+0x0/0x1d8
[ 3.239881] show_stack+0x20/0x68
[ 3.243255] dump_stack+0xd0/0x12c
[ 3.246714] panic+0x164/0x364
[ 3.249823] mount_block_root+0x2b0/0x344
[ 3.253900] mount_root+0x78/0x88
[ 3.257270] prepare_namespace+0x138/0x178
[ 3.261435] kernel_init_freeable+0x270/0x2c4
[ 3.265867] kernel_init+0x1c/0x128
[ 3.269416] ret_from_fork+0x10/0x34
[ 3.273055] SMP: stopping secondary CPUs
[ 3.277054] Kernel Offset: 0x4f96b9000000 from 0xffff800010000000
[ 3.283248] PHYS_OFFSET: 0xffffaa4fc0000000
[ 3.287500] CPU features: 0x0240022,61806000
[ 3.291840] Memory Limit: none
[ 3.294954] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,255) ]---
I have found old mailing list threads that hint that maybe we need to go out of our way to provide an nfsmount:
At the same time though, I’m very suspicious that I don’t see anything about genet in the kernel init log, despite my forcing it into initrd via boot.initrd.kernelModules. On the rpifour1 booting from USB-NVME, it clearly shows genet in the log and then eth0 appearing.
Relevant config for rpifour2:
{
imports = [
"${modulesPath}/installer/netboot/netboot.nix"
];
config = {
boot.kernelPackages = pkgs.linuxPackages_latest;
boot.initrd.supportedFilesystems = lib.mkForce [ "vfat" "nfs" ];
boot.initrd.kernelModules = [
"nfs" "genet" "broadcom"
"xhci_pci" "libphy" "bcm_phy_lib"
];
boot.kernelModules = config.boot.initrd.kernelModules;
networking.hostName = "rpifour2";
networking.useDHCP = true;
boot.initrd.network.enable = true;
};
}
The resulting cmdline.txt that is provided to the pi during netboot winds up looking like this:
earlycon=uart8250,mmio32,0xfe215040 ip=dhcp root=/dev/nfs nfsroot=192.168.1.2:/rpifour2,vers=4.1,proto=tcp ro rootwait elevator=deadline init=/nix/store/nqnz45qm8p9649w5hcg8pzma0vm223bg-nixos-system-rpifour2-21.03pre-git-ab8ca6d6/init isolcpus=3 nfsrootdebug
(notice the enabling of boot.initrd.network.enable and the forcing of genet in initrd)
I feel like someone out there has a nixos config with nfsroot somewhere. 
I can pick up the pieces if you throw it up in a gist! I think it would be really cool for this scenario to work out of box for NixOS. If anyone is interested in working on this, let me know. (same username on colemickens@freenode or @colemickens:matrix.org).
(Maybe this is also something that could/would be addressed in the course of converting more of our init to using systemd?)