But when I reboot, zfs fails to import the pool, so /nix/store isn’t mounted:
loading module zfs...
loading module spl...
loading module dm_mod...
running udev
Starting systemd-udevd version 255.2
starting device mapper and LVM
importing root ZFS pool "pool"...................................
cannot import 'pool': no such pool available
[...]
mounting pool/nixos/nix on /nix...
mount: mounting pool/nixos/nix on /mnt-root/nix
failed: No such file or directory
The mounting of the directory works if done manually after boot:
john@sun:// > sudo mount -t zfs pool/nixos/nix /nix
john@sun:// > cd /nix
john@sun:/nix/ > ls -al
total 52319
drwxrwxr-t 5 root nixbld 5 Mar 29 13:41 .
drwxr-xr-x 22 root root 4096 Mar 29 13:15 ..
drwxr-xr-x 2 root root 130763 Mar 29 08:51 .links
drwxrwxr-t 1172 root nixbld 7737 Mar 29 17:16 store
drwxr-xr-x 4 root root 4 Aug 25 2023 var
The nix directory passes a nix-store --verify command:
john@sun:// > sudo nix-store --verify --check-contents
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...
john@sun:// > echo $?
0
I’d really appreciate thoughts on what might be going wrong?
Hi,
So i dont have a direct solution for you, but possible some input that might help for further checks.
The actual error that you might want to dig deeper is
cannot import 'pool': no such pool available
What you also have figured out.
Do you end in a emergency shell? I remember always ending up in a systemd emergency shell, when not being able to import my rootfs zfs pool (and i assume it should also when not having a /nix available)
If that is the case, you could investigate from there whats the problem of pool loading. So you could then just try to import it manually and check what its complaining about.
What confuses me is, that you are saying that you can manually mount if after boot. I would assume that it should not be bootable (as there is no nix store). Does possibly the old nix store still sit on /nix and zfs/kernel is refusing to “overmount” that /nix ?
Two things that i recognize, that could be worth checking:
If the pool was not exported correctly after creation, it will possibly not be loadable by boot, as you have disable force Import, so you might want to double check if that could be an issue, in case you did create the pool from live cd or so.
You are loading the pool by-label. Does the kernel actually load the disk there and does it have a reasonable label that you can mount?
A common recommendation for nixos and zfs to use legacy mountpoints as double hint (not sure if you are using that, but as you have not specified i thought pointing it out might make sense).
If the pool is made up of multiple devices, then this isn’t likely to work, since only one device gets to claim the symlink with the pool’s name in that directory. Leave it default (which is /dev/disk/by-id) or set it to something that will have a node/symlink for every device in it.
Otherwise, it’s likely a driver issue. Whatever kernel module is used to access the disk the pool is on needs to be in boot.initrd.kernelModules.
Real quick, to address some of @Shawn8901’s comments.
Note that the systemd shell will only happen for the rootfs or /nix fs if you use boot.initrd.systemd.enable, which is not currently the default. Otherwise you just get an error prompt from the scripted initrd on the console, which only offers a shell if you have boot.shell_on_fail in the kernel params.
Reading the link OP shared, yes, they copied /nix to a new file system, and then mounted it overtop /nix. The expectation here is that upon rebooting, the new file system will be mounted at /nix, and the old file system will just have a bunch of dead files being hidden under it.
That was it! Thank you so much It would be good if a better error could be shown when the initrd kernel module that’s necessary isn’t loaded but cest la vie!
Oops, I meant to say boot.initrd.availableKernelModules; but that doesn’t matter much.
The problem is how is NixOS supposed to know that you forgot a kernel module? It’s just as possible that your disk isn’t plugged in and initrd does have the kernel module for it.