Thanks for such a thorough and thoughtful response!
Interestingly, I was following the advice from the NixOS Wiki, which at the time of installation seemed like an unambiguous endorsement of following the OpenZFS instructions:
It has since been updated to be far less charitable about the OpenZFS guide:
Warning: This guide is not endorsed by NixOS and some features like immutable root do not have upstream support and could break on updates. If an issue arises while following this guide, please consult the guides support channels.
Based on Stage 1 starting, I assume it’s at least finding the kernel and initramfs okay. The advantages I was hoping to leverage include ZFS snapshots, ZFS send / receive for backing up as much as possible, and compression (this is a raspberry pi – please see the dozens of posts about “boot partition out of space” issues before deciding this is completely silly)…
$ zfs get compressratio bpool
NAME PROPERTY VALUE SOURCE
bpool compressratio 1.27x -
Personally I don’t consider these to be “little” benefit, although I suppose one might be assuming that NixOS provides me the ability to regenerate the boot partition with little trouble – unfortunately I’ve not found this to be the case with Pi4 on UEFI, which refuses to boot from the standard aarch64 ISO, so I’m left booting from the sd-image not in EFI mode, which means that grub can’t install the EFI bootloader stuff unless I change my configuration to
efiInstallAsRemovable = true;, and then I change it back and reinstall once things are booting… NixOS on an RPi with ZFS root + UEFI has been fine when it works, but when it doesn’t it’s been a handful, so I was hoping having a ZFS boot partition would help make sure I could recover with minimal headache.
Clearly my hopes were in vain
For full context, my other motivating factor was that I have other NixOS on ZFS root devices that are booting from ZFS mirrored boot partitions; my understanding is that grub is the only option for booting from a ZFS mirror. Sticking to grub for all my devices seemed like it would minimize what I’d need to learn (as opposed to learning intricacies and failure modes of multiple bootloaders).
Maybe I’ll give a shot to systemd-boot on this machine and see how it goes. I’m particularly interested in the potential of [WIP] nixos/systemd-boot: boot counting and automatic fallback by danielfullmer · Pull Request #84204 · NixOS/nixpkgs · GitHub