What is the way to mount zfs filesystems?

Help
1

I have a ZFS pool “raid”, and a few other filesystems created within it, also the Docker data folder. Right now, the problem is, not all filesystems are mounted at boot.

First of all, Docker creates a bunch of legacy mount filesystems, which seems breaking systemd.services.zfs-mount, which only mounts non-legacy mount filesystems by alphabet order until the first legacy mount filesystem. All non-legacy mount ones after are just ignored.

So I disabled it, then no non-legacy mount filesystems were mounted at boot. I have to zfs mount -a after booting and reload a few services.

Then I set all filesystems to legacy mount, and added a bunch of fileSystems entries in my NixOS configuration. This time, it just mounted first 4 entries (by the order of the entries in NixOS configuration). And a bind mount depends on one zfs filesystems was directly mounted. Systemd mount unit says “mounted”, “successfully” about those not-mounted ones.

Maybe something unmounted those before I login, but I have no clues.

What should I do to get it working flewlessly?

2

wait wut. That’s not what it’s supposed to do. It should actually mount all non-legacy datasets. I wrote a quick NixOS test to check this, and it seems to be fine to me.

That order is not preserved; the nixos config is just creating an attrset, which does not remember the order in which they were written in code.

Can you explain what you mean by this? I’m not understanding.

Can you share your nixos config, as well as logs from when zfs-mount.service wasn’t working? It sounds like something is going horribly wrong here.

3

@ElvishJerricco

This is the not-so-working configuration.

And a bind mount depends on one zfs filesystems was directly mounted. Systemd mount unit says “mounted”, “successfully” about those not-mounted ones.

Sorry for the confusing. The first sentence was a mis-statement. I meant all not-mounted systemd mount units status were mounted and successfully turned into dead. But the result was filesystems actually not mounted. So I thought something unmounted those.

And this is the zfs list:

NAME                                                                                USED  AVAIL  REFER  MOUNTPOINT
raid                                                                               2.16T  1.92T   456K  /mnt/data
raid/backup                                                                        1.03T  1.92T  1.03T  legacy
raid/conf                                                                          13.3M  1.92T  13.3M  legacy
raid/db                                                                            47.2M  1.92T  47.2M  legacy
raid/docker                                                                         388G  1.92T  2.54G  legacy
# 1358 lines of filesystems within raid/docker like this one
raid/docker/0042c0e4a9fe44fd6bb2857c78fbc7fc9859551ee45e4f090184618fcb296a47       87.1M  1.92T   833M  legacy
raid/downloads                                                                      201G  1.92T   201G  legacy
raid/home                                                                           190G  1.92T   190G  legacy
raid/upload                                                                         374G  1.92T   374G  legacy

I updated all mountpoints except those 1358 lines (those are created by Docker, per another post, it only uses legacy). And removed those filesystems entries. After reboot, those were mounted. And I cannot zfs mount -a because after a while, something would unmount the ones outside those four. And I checked the attributes between a mounted one (db) and not-mounted one (backup), nothing except size/date/guid is different.

$ zfs mount
raid                            /mnt/data
raid/db                         /mnt/data/db
raid/conf                       /mnt/data/conf
raid/docker                     /mnt/data/docker

“zfs-mount” does not give log. Following is the log around zfs-mount.

Mar 08 23:10:04 nixos zfs-import-raid-start[1126]: importing ZFS pool "raid"...Successfully imported raid
Mar 08 23:10:04 nixos systemd[1]: Finished Import ZFS pool "raid".
░░ Subject: A start job for unit zfs-import-raid.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit zfs-import-raid.service has finished successfully.
░░
░░ The job identifier is 70.
Mar 08 23:10:04 nixos systemd[1]: Reached target ZFS pool import target.
░░ Subject: A start job for unit zfs-import.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit zfs-import.target has finished successfully.
░░
░░ The job identifier is 69.
Mar 08 23:10:04 nixos systemd[1]: Starting Mount ZFS filesystems...
░░ Subject: A start job for unit zfs-mount.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit zfs-mount.service has begun execution.
░░
░░ The job identifier is 67.
Mar 08 23:10:12 nixos systemd[1]: Finished Mount ZFS filesystems.
░░ Subject: A start job for unit zfs-mount.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit zfs-mount.service has finished successfully.
░░
░░ The job identifier is 67.
Mar 08 23:10:12 nixos systemd[1]: Reached target Local File Systems.
░░ Subject: A start job for unit local-fs.target has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit local-fs.target has finished successfully.
░░
░░ The job identifier is 21.

Also, just retried the legacy way, set all filesystems to legacy. The only mounted filesystems I got after boot, interestingly, were the same as non-legacy way (raid, raid/db, raid/conf, raid/docker), plus raid/home, which, unexpectedly, mounted to the bind mountpoint, but not original mountpoint.

And from console log, the missing mounts were “mounted”.

Reached target ZFS po                                                  [  OK  ] 
Mounted /mnt/data.                                                              Mounting /mnt/data/backup...                                                    Mounting /mnt/data/conf...     09 CI Express Gigabit Server Adapter).          Mounting /mnt/data/conf...                                                      Mounting /mnt/data/docker...                                                    Mounting /mnt/data/downloads...                                                 Mounting /mnt/data/home...                                                      Mounting /mnt/data/upload...                                           [  OK  ] 
Mounted /mnt/data/backup.                                              [  OK  ] 
Mounted /mnt/data/db.                              ation.              [  OK  ] 
Mounted /mnt/data/db.
5

Another interesting thing is that, since some were not mounted. I mount -a manually after boot. Then raid/home was mounted three times (total).

# mounted during boot
raid/home on /home type zfs (rw,relatime,xattr,noacl,casesensitive)
# original mount point
raid/home on /mnt/data/home type zfs (rw,relatime,xattr,noacl,casesensitive)
# bind mount point
raid/home on /home type zfs (rw,relatime,xattr,noacl,casesensitive)
6

So I did some test. I disabled mounting zfs via filesystems or zfs-mount. They are not mounted after reboot.

Then I mounted them manually as root. After a while, they were still there.

Then I noticed Docker was not running. So I systemctl restart docker. It took a bit long than usual. Right after it ended, some of the ZFS volumes were unmounted.

So it definitely is some “smart” move of systemd.

7

It’s docker doing that, not systemd.

9

It is yet hard to say. I did some tests and got a few threads in Docker community and other places seems indicating this is a Docker issue.

But there is a thread saying that this happens when “systemd reloading” appears in log.

And in my system (ZFS filesystems using non-legacy mountpoints), I stopped Docker and mounted all filesystems before sleeping. And today, they are all gone. Recalling the problem before, there was Docker running, and there were a few filesystems still mounted. And this time, there was no Docker (I stopped), and there was no filesystems mounted. ALL GONE.

10

I built an Archlinux VM with 4 disks in a ZPool. No such issue even I replaced all Docker binaries with 24.0.5.

Before digging into Systemd, let me upgrade my Docker in NixOS to 26 and see.

11

Updated to unstable channel and using Docker 25, so far so good. Let me wait another night and see.

12

Just to sum up what I got so far on this issue. No direct bug report or solution yet.

  1. The issue is gone with same hardware, same NixOS configuration but in unstable channel with Docker 25 (Channel 23.11 uses Docker 24). And Docker 24 in unstable still causes the issue.

  2. The issue cannot be reproduced in Virtualbox, with simple NixOS 23.11 configuration, plus ZFS and Docker. No ZFS properties difference was spotted.

  3. One case I cannot explain: When Docker was stopped, over night, all ZFS FS-es were unmounted, with Docker still stopped the next morning. Assuming something brought up Docker via its Systemd socket, two differences I noticed. One, Docker did not keep running. Two, instead of unmount some of the FS-es when starting, in this case, all were unmounted.