Moving /nix to ZFS

Help
1

Hi there, I’ve been trying to migrate my Nixos flake configuration so that the /nix directory is mounted from a ZFS filesystem.

I created some zfs mountpoints:

john@sun:~/ > sudo zfs list | grep nixos 
pool/nixos                                                                 16.4G  9.04T   134K  /pool/nixos
pool/nixos/nix                                                             16.4G  9.04T  16.4G  legacy

Setup the config as such:

  boot = {
    loader = {
      systemd-boot.enable = true;
      efi.canTouchEfiVariables = true;
    };

    # Pin the kernel version to the latest version that ZFS supports.
    kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;

    # Import the ZFS pool
    zfs.extraPools = [ "pool" ];

    supportedFilesystems = lib.mkForce [
      # Needed for https://github.com/NixOS/nixpkgs/issues/58959
      "btrfs" "reiserfs" "vfat" "f2fs" "xfs" "ntfs" "cifs" "zfs"
    ];

    initrd = {
      kernelModules = [ ];
      availableKernelModules = [
        "ahci" "xhci_pci" "ehci_pci" "usb_storage" "usbhid" "sd_mod"
      ];
      supportedFilesystems = [ "zfs" ];
    };

    kernelModules = [
      "kvm-intel" "jc42" "tpm_rng" "ipmi_devintf" "ipmi_si" "zfs"
    ];
    extraModulePackages = [ ];
    supportedFilesystems = [ "zfs" ];
    zfs.forceImportRoot = false;
    zfs.devNodes = "/dev/disk/by-label";
  };

  fileSystems."/nix" = 
     { device = "pool/nixos/nix";
        neededForBoot = true;
       fsType = "zfs";
      };
  };

  services.zfs = {
    autoScrub.enable = true;
    trim.enable = true;
  };

I used the notes in Storage optimization - NixOS Wiki to copy the store into ZFS.

But when I reboot, zfs fails to import the pool, so /nix/store isn’t mounted:

loading module zfs...
loading module spl...
loading module dm_mod...
running udev
Starting systemd-udevd version 255.2
starting device mapper and LVM
importing root ZFS pool "pool"...................................
cannot import 'pool': no such pool available
[...]
mounting pool/nixos/nix on /nix...
mount: mounting pool/nixos/nix on /mnt-root/nix
failed: No such file or directory

The mounting of the directory works if done manually after boot:

john@sun:// > sudo mount -t zfs pool/nixos/nix /nix      
john@sun:// > cd /nix
john@sun:/nix/ > ls -al
total 52319
drwxrwxr-t    5 root nixbld      5 Mar 29 13:41 .
drwxr-xr-x   22 root root     4096 Mar 29 13:15 ..
drwxr-xr-x    2 root root   130763 Mar 29 08:51 .links
drwxrwxr-t 1172 root nixbld   7737 Mar 29 17:16 store
drwxr-xr-x    4 root root        4 Aug 25  2023 var

The nix directory passes a nix-store --verify command:

john@sun:// > sudo nix-store --verify --check-contents
reading the Nix store...
checking path existence...
checking link hashes...
checking store hashes...
john@sun:// > echo $?                      
0

I’d really appreciate thoughts on what might be going wrong?

2

Hi,
So i dont have a direct solution for you, but possible some input that might help for further checks.

The actual error that you might want to dig deeper is

cannot import 'pool': no such pool available

What you also have figured out. :blush:
Do you end in a emergency shell? I remember always ending up in a systemd emergency shell, when not being able to import my rootfs zfs pool (and i assume it should also when not having a /nix available)
If that is the case, you could investigate from there whats the problem of pool loading. So you could then just try to import it manually and check what its complaining about.

What confuses me is, that you are saying that you can manually mount if after boot. I would assume that it should not be bootable (as there is no nix store). Does possibly the old nix store still sit on /nix and zfs/kernel is refusing to “overmount” that /nix ?

Two things that i recognize, that could be worth checking:

If the pool was not exported correctly after creation, it will possibly not be loadable by boot, as you have disable force Import, so you might want to double check if that could be an issue, in case you did create the pool from live cd or so.

You are loading the pool by-label. Does the kernel actually load the disk there and does it have a reasonable label that you can mount?

A common recommendation for nixos and zfs to use legacy mountpoints as double hint (not sure if you are using that, but as you have not specified i thought pointing it out might make sense).

3

If the pool is made up of multiple devices, then this isn’t likely to work, since only one device gets to claim the symlink with the pool’s name in that directory. Leave it default (which is /dev/disk/by-id) or set it to something that will have a node/symlink for every device in it.

Otherwise, it’s likely a driver issue. Whatever kernel module is used to access the disk the pool is on needs to be in boot.initrd.kernelModules.

Real quick, to address some of @Shawn8901’s comments.

Note that the systemd shell will only happen for the rootfs or /nix fs if you use boot.initrd.systemd.enable, which is not currently the default. Otherwise you just get an error prompt from the scripted initrd on the console, which only offers a shell if you have boot.shell_on_fail in the kernel params.

Reading the link OP shared, yes, they copied /nix to a new file system, and then mounted it overtop /nix. The expectation here is that upon rebooting, the new file system will be mounted at /nix, and the old file system will just have a bunch of dead files being hidden under it.

1 Like
4

That was it! Thank you so much :slight_smile: It would be good if a better error could be shown when the initrd kernel module that’s necessary isn’t loaded but cest la vie!

That’s exactly it.

5

Oops, I meant to say boot.initrd.availableKernelModules; but that doesn’t matter much.

The problem is how is NixOS supposed to know that you forgot a kernel module? It’s just as possible that your disk isn’t plugged in and initrd does have the kernel module for it.

1 Like