How to get ZFS working with Docker?

There is https://nixos.wiki/wiki/NixOS_on_ZFS, and there is https://nixos.wiki/wiki/Docker. But when the two get together, I got a bunch of mounts in fstab for Docker graph, for example:

raid/docker/5d6b50a1ac277f868d39351c0930a879dd6224c43aca0f9fe982280c5b9c6912 /mnt/data/docker/zfs/graph/5d6b50a1ac277f868d39351c0930a879dd6224c43aca0f9fe982280c5b9c6912 zfs defaults 0 0

Is manually removing those from hardware-configuration.nix the only solution?

1 Like

So far the only solution I have found is to stop docker and manually unmount remaining stuff that shouldn’t be permanent before running nixos-configure, though in general, I haven’t run that command anymore after the first one or two weeks of setting up that computer.

Docker has a ZFS storage driver that automatically detects if your images are being stored on ZFS. If so, it uses ZFS datasets and clones to handle images. It’s actually kinda clever but VERY annoying that it does so by default. There’s a way to disable it but I forget how.

Ye, regenerating configuration is rare, but a tricky part.

The generated mounts would block rebuild if something in Docker changed.

I did not find things like options in Docker doc. And I think I have to say, I need that feature. For example, using overlayfs failed me. And Podman does that as well.

I wonder if I could use non-legacy mount. It worked in other Linux distro. But I got errors (which block booting) when migrating to NixOS, which disappeared after changing to legacy mount.

I specifically set my storage driver to zfs. Presumably if you set it to another valid storage driver you might get the desired behavior? Ex/ https://github.com/colemickens/nixcfg/blob/c7e2737a1c9cb1fa24d05b56d216892466d03910/mixins/docker.nix#L7

Also, BTW: docker system prune -af is aggressive (maybe too much so depending on how you use containers, read the docs first), but will get rid of those things, so they don’t just pile up.

(Also, run the prune before changing drivers!!)

Oh, I did not mean to “remove” them. Just those Docker-handled-mounts may break nixos-rebuild switch when NixOS wants to handle them.

I wonder if there is a way to tell NixOS to leave those alone.

@Magicloud, in what way has overlay failed you?

I’m asking because I was thinking that it might be better to change the docker nixos module to always default to overlay2 instead of letting docker choose the driver matching the underlying fs.

I’m on btrfs and I needed to waste a lot of time to figure out that docker’s btrfs storage driver is buggy and there are no attempts to fix it.

Apart from the buggy docker driver, btrfs doesn’t cope well with intensive snapshotting, which will magically eat away your available free space without using it. Also there isn’t really a performance benefit either as far as I can tell.

I never tried running docker with the zfs storage driver. So I’m asking, what benefit does it really bring?

Overlayfs has size restriction, does not fit some of my usage. And particularly, there was some mysterious bug that all containers sharing the same base overlay. I did not dig much.

The reason I used ZFS, mostly importantly, my RAID card does not support level 6. And ZFS raid rebuilding is way more fast than normal RAID card. Snapshot and other advanced FS features also help on me saving my precious data.

Never tried Docker on BTRFS.

Weird, Discourse does not mark my last comment as replying to you.

Sorry I implemented that behavior, when I added zfs support to docker back than. You can pass --storage-driver to dockerd to change that in virtualisation.docker.extraOptions.

overlayfs does not work on zfs. You can format a zvol with ext4 and mount it to /var/lib/docker so.

1 Like

Both nixos and docker use legacy mountpoints, so I don’t see a way to make nixos-generate ignore it. Maybe docker could add some metadata to those datasets so we can detect it in nixos-generate itself. In the past docker used private mounts for so those mounts would be hidden but than they remove it for different reason again.

1 Like

docker could set user properties: https://manpages.debian.org/testing/zfsutils-linux/zfs.8.en.html#User_Properties This feature should be easy to add.
After this line https://github.com/moby/moby/blob/b837751e40e8a58d295d8261d0dc84a452cabe7c/daemon/graphdriver/zfs/zfs.go#L305 we need a zfs set.

1 Like

Oh, that is kind of harsh since I also have other data on the volume…