RPI4b - Nixos USB Boot broken - unstable, new kernel

I’ve been playing around with raspberry pi 4 usb images - I can always get the sd card images to work, however with usb images I’m constantly getting timeouts while trying to find the sd via

/dev/disk/by-label/NIXOS_SD

A specfic repro is as follows

Download a specific unstable version (but AFAICT, none of the unstable versions work).

curl -O https://hydra.nixos.org/build/162795703/download/1/nixos-sd-image-22.05pre342139.ff202a635d2-aarch64-linux.img.zst

Expand this and write it to your usb drive (I’ve tried two, both are PNY CS900 with random usb adapters off of amazon, I’ve ordered a new drive that hasn’t arrived yet to confirm this isn’t something PNY CS900 related).

unzstd nixos-sd-image-22.05pre342139.ff202a635d2-aarch64-linux.img.zst
dd if=./nixos-sd-image-22.05pre342139.ff202a635d2-aarch64-linux.img of=/dev/sda status=progress

Boot the computer

Get the following error message

Timed out waiting for device /dev/disk/by-label/NIXOS_SD, trying to mount anyway
mounting /dev/disk/by-label/NIXOS_SD on /...
mount: mounting /dev/disk/by-lable/NIXOS_SD on /mnt-root failed: No such file or directory

I’ve attached a screenshot of the boot log, but stage1 clearly boots, and then the initrd can’t seem to mount the image. I’ve searched around and found a couple bugs saying that maybe lvm is slow, but nothing conclusive.

Note that I’ve validated the raspberry pi firmware versions - in this case it’s from 2021-Apr-29 (1619712685), so it should be current. The VL805_fw is also update to date - version 000138a1

I’ve tried with a couple different usb adapters (but the same ssds - PNY CS900s). I’ve also tried having a rpi4b emulate a usb drive, and the same error happens, so I’m pretty sure it’s unrelated to my SSDs and adapters. Raspbian does boot for reference (and is how I confirmed the rpi firmware version).

I haven’t yet gotten this wired up so I can just start bisecting, but when I plugged in a couple of other versions (i.e. the last couple stables), I had to go pretty far back to get a version which booted successfully. I’ll work on cleaning up this post so that I can tell you exactly what versions are broken.

If anyone doesn’t quite know what’s broken, but has thoughts on extra debugging steps - please let me know - I’m happy to either do them, or convert this into a job board posting :).

Edit - And of course once I start walking through the version matrix I get it to boot - Looks like the new kernel is broken, but the lts kernel works.

I’d obviously still like to get the new kernel to work, but I think I have a workaround.