NixOS, systemd, ZFS and NFSv4 bind mounts

Help
1

As seen in several other places (here and here for example), there’s a bit of an ordering, timing or even race condition like problem with trying to expose bind mounts for NFS (specifically NFSv4) when the underlying mount is coming from ZFS.

I’ve tried numerous incantations attempting to make use of the systemd based functionality of x-systemd.requires as a mount option without any luck, trying to depend on various things like zfs-mount.service or even my specific zfs-import-data.service for the pool in question. I also tried various path based approaches like x-systemd.requires=/data, all of which resulted in empty bind mount points under /srv/nfs.

Now, I am seeing at least the ordering of all of these things seemingly happening correctly:

Dec 31 00:24:27 arrakis systemd[1]: Finished Wait for udev To Complete Device Initialization.
Dec 31 00:24:27 arrakis systemd[1]: Starting Import ZFS pool "data"...
Dec 31 00:24:32 arrakis zfs-import-data-start[1591]: importing ZFS pool "data"...Successfully imported data
Dec 31 00:24:32 arrakis systemd[1]: Finished Import ZFS pool "data".
Dec 31 00:24:32 arrakis systemd[1]: Reached target ZFS pool import target.
Dec 31 00:24:32 arrakis systemd[1]: Mounting /srv/nfs/stuff...
Dec 31 00:24:32 arrakis systemd[1]: Mounting /srv/nfs/stuff2...
Dec 31 00:24:32 arrakis systemd[1]: Starting Mount ZFS filesystems...
Dec 31 00:24:32 arrakis systemd[1]: Mounted /srv/nfs/stuff.
Dec 31 00:24:32 arrakis systemd[1]: Mounted /srv/nfs/stuff2.
Dec 31 00:24:33 arrakis systemd[1]: Finished Mount ZFS filesystems.
Dec 31 00:24:33 arrakis systemd[1]: Reached target Local File Systems.

Empty directories are still the result, which is why I think this could be a timing/race type situation possibly. And I didn’t see any obvious way to inject any delay via the auto generated mount units via systemd-fstab-generator. Although, as it turns out, too much additional delay isn’t apparently necessary.

So for now, the working solution I’ve landed on is to define the file system bind mounts like this:

fileSystems."/srv/nfs/stuff" = {
  device = "/data/stuff";
  fsType = "none";
  options = [
    "bind"
    "noauto"
  ];
};

and then use my own systemd service to manually do the mounts:

"nfs-bind-mount" = {
  after = [ "zfs-import-data.service" ];
  description = "Bind NFS exports to ZFS paths";
  script = ''
    ${pkgs.util-linux}/bin/mount /srv/nfs/stuff
    ${pkgs.util-linux}/bin/mount /srv/nfs/stuff2
  '';
  wantedBy = [ "local-fs.target" ];
};

All of which gives me the desired result of having those bind mounts properly populated after boot finishes without me needing to manually mount them myself since no variation I attempted to make systemd handle this directly was successful.

I’m posting this partly to see if anyone else ever has gotten this working correctly through what seems like it should be the desired mechanism (systemd), but mostly just to document the problem, at least as it presents trying to use the systemd supplied mechanism. Maybe there is yet another better way to define this that doesn’t involve creating my own manual service?

Either way, maybe it will help someone else at least in the meantime.

1 Like
2

The tricky thing is the ZFS mountpoint. If you have fileSystems definitions for your ZFS datasets in your hardware-configuration.nix module or something, then you want those datasets to have mountpoint=legacy, in which case your bind mounts can use x-systemd.requires-mounts-for=/data/stuff (which is a tad different than x-systemd.requires=/data/stuff). But if your datasets have mountpoints like mountpoint=/data/stuff, then you don’t want those datasets in your Nix modules’ fileSystems, and your bind mount would use x-systemd.requires=zfs-mount.service. And what’s with the noauto? That just means they won’t be mounted automatically, which obviously does no good.

So, point being, one of these should have done the trick, depending on if you use legacy mountpoints or not:

# for legacy mountpoint
fileSystems."/data/stuff" = {
  device = "data/stuff";
  fsType = "zfs";
};
fileSystems."/srv/nfs/stuff" = {
  device = "/data/stuff";
  fsType = "none";
  options = [
    "bind"
    "x-systemd.requires-mounts-for=/data/stuff"
  ];
};

# For mountpoint=/data/stuff
fileSystems."/srv/nfs/stuff" = {
  device = "/data/stuff";
  fsType = "none";
  options = [
    "bind"
    "x-systemd.requires=zfs-mount.service"
  ];
};
3

For the data pool specifically, I’m using zfs.extraPools = [ "data" ]; which presumably is what is building the zfs-import-data.service. This was basically the fist thing I tried, using "x-systemd.requires=zfs-mount.service" unsuccessfully. It seemed like the most straightforward version of all the various things I’ve tried.

The noauto and everything I posted above is my current workaround solution. So I wasn’t trying to even bother with having them mounted out of fstab by systemd directly but rather define them for the service to be able to use the short form mount command to mount them later in the boot process. Which, happens later enough, that it works.

Based on my reading of x-systemd.requires at systemd.mount, it’s not clear to me how that and x-systemd.requires-mounts-for differ as the former also takes a path argument. Having said that, I didn’t try that and that appears to be the legacy option anyway.

Although it might be worth trying the legacy route as I’m already doing that indirectly via disko for my root file system and home pool.