My ZFS volumes fail to mount at boot

Hey,

I have a server with ZFS filesystem for both boot and my data. Now, the boot from ZFS works just fine, but the boot process falls on its face when it tries to mount the other ZFS volumes.

Error:

         Starting Mount ZFS filesystems...
[FAILED] FAILED to mount /mnt/data.
See 'systemctl status mnt-data.mount' for details.
[DEPEND] Dependency failed for Local FIle Systems
...
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or "exit"
to boot into default mode.
Give root password for maintenance
(or press Control-D to continue): 

When use Control-D the system starts up fine. When I check the systemd mount I find:

root@melchior> systemctl status mnt-data.mount                                                                                                          ~
â—Ź mnt-data.mount - /mnt/data
   Loaded: loaded (/etc/fstab; generated)
   Active: active (mounted) since Sun 2020-01-12 13:41:22 CET; 5min ago
    Where: /mnt/data
     What: DATA/data
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
       IP: 0B in, 0B out
    Tasks: 0 (limit: 4915)
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/mnt-data.mount

It’s mounted just fine:

 root@melchior> df -h /mnt/data                                                                                                                    ~
Filesystem      Size  Used Avail Use% Mounted on
DATA/data       799G  709G   90G  89% /mnt/data

But the logs show a different story:

root@melchior> journalctl -f -u mnt-data.mount                                                                                                          ~
-- Logs begin at Sat 2020-01-11 17:47:51 CET. --
Jan 12 13:40:37 melchior systemd[1]: Unmounting /mnt/data...
Jan 12 13:40:37 melchior systemd[1]: mnt-data.mount: Succeeded.
Jan 12 13:40:37 melchior systemd[1]: Unmounted /mnt/data.
-- Reboot --
Jan 12 13:41:21 melchior systemd[1]: Mounting /mnt/data...
Jan 12 13:41:22 melchior mount[2961]: filesystem 'DATA/data' cannot be mounted using 'mount'.
Jan 12 13:41:22 melchior mount[2961]: Use 'zfs set mountpoint=legacy' or 'zfs mount DATA/data'.
Jan 12 13:41:22 melchior mount[2961]: See zfs(8) for more information.
Jan 12 13:41:22 melchior systemd[1]: mnt-data.mount: Mount process exited, code=exited, status=1/FAILURE
Jan 12 13:41:22 melchior systemd[1]: mnt-data.mount: Failed with result 'exit-code'.
Jan 12 13:41:22 melchior systemd[1]: Failed to mount /mnt/data.

And hare are mount related settings:

root@melchior> zfs get -r type,mountpoint,canmount,mounted DATA/data                                                                                    ~
NAME       PROPERTY    VALUE       SOURCE
DATA/data  type        filesystem  -
DATA/data  mountpoint  /mnt/data   local
DATA/data  canmount    on          default
DATA/data  mounted     yes         -

Here is how it’s configured in my hardware-configuration.nix:

  fileSystems."/mnt/data" =
    { device = "DATA/data";
      fsType = "zfs";
      neededForBoot = false;
    };

I tried setting neededForBoot to false just in case but it does nothing, since it should be false anyway.
I loaded other ZFS settings in my configuration.nix:

  boot.supportedFilesystems = [ "zfs" ];
  boot.loader.grub.copyKernels = true;

My boot.loader.grub.devices contains only two devices from my root pool, not the DATA one.
And here is my full configuration:

I really don’t understand why the volume fails to mount.

It looks like both zfs and the system are trying to mount it.

Have you tried either setting the mount in zfs to legacy or removing the filesystems expression for it?

Yes, of course it boots just fine if I remove the the fileSystems entries from hardware-configuration.nix, but how can that be the solution if they will be put back again if I just run nixos-generate-config. I had this working before, but for some reason it stopped working without a clear reason.

I have the pool that I boot from set to legacy, but I don’t think that’s a correct setting for my other pools, since i’m not mounting them, I’m mounting the volumes:

 melchior > sudo zfs get -r mountpoint                                                                                                            ~/nixos
NAME            PROPERTY    VALUE           SOURCE
DATA            mountpoint  none            local
DATA/data       mountpoint  /mnt/data       local
DATA/git        mountpoint  /mnt/git        local
DATA/music      mountpoint  /mnt/music      local
DATA/nextcloud  mountpoint  /mnt/nextcloud  local
MEDIA           mountpoint  none            local
MEDIA/torrent   mountpoint  /mnt/torrent    local
SYSTEM          mountpoint  legacy          local

You don’t need to set the whole pool to legacy, just the dataset.

1 Like

But I don’t have a data set at all on that pool. I use the whole thing.

Sorry, I may have misunderstood what you were saying.

I wasn’t saying you should change your SYSTEM pool, just that you could set DATA/data to legacy without setting the whole zpool to legacy.

Hopefully someone with more nixos experience will come along to help but I assume this is the reason that the nixos zfs tutorials recommend using legacy mountpoints.

Oh wait, you’re right, all volumes need to be set to mountpoint=legacy:

 melchior > sudo zfs get -r mountpoint                                                                                                                  ~
NAME            PROPERTY    VALUE       SOURCE
DATA            mountpoint  none        local
DATA/data       mountpoint  legacy      local
DATA/git        mountpoint  legacy      local
DATA/music      mountpoint  legacy      local
DATA/nextcloud  mountpoint  legacy      local
MEDIA           mountpoint  none        local
MEDIA/torrent   mountpoint  legacy      local
SYSTEM          mountpoint  legacy      local

With this the boot process works fine and all volumes are mounted.

Maybe the warning here should be updated:
https://nixos.wiki/wiki/NixOS_on_ZFS
It states:

Warning: Add all mounts to your configuration as legacy mounts as described in this article instead of zfs’s own mount mechanism. Otherwise mounts might be not mounted in the correct order during boot!

Maybe it should say:

Warning: Add all mounts to your configuration as legacy mounts as described in this article instead of zfs’s own mount mechanism. Otherwise they will fail to mount at all.

Thanks for pointing me in the right direction @dalto.

2 Likes