Hi, I am trying NixOS and one of the things I like is the clutter-free booting with just an EFI partition plus a non-crippled ZFS pool. However I am struggling to get it completely “clean”.
The wiki (https://nixos.wiki/wiki/ZFS) seems to imply that one can drop the legacy mountpoints by adding
boot.zfs.extraPools. When I do this, and remove
hardware-configuration.nix (except for
/boot of course) then (i) either the system does not boot (in case I remove e.g.
/nix), or (ii) an assertion fails during
nixos-rebuild (“The fileSystems option does not specify your root file system”).
I am sure the ZFS datasets can be made to load during boot just fine. E.g. according to other answers to https://discourse.nixos.org/t/zfs-systemd-boot/29956 one could import all ZFS datasets including root using
However, how do I get past this failing assertion?
I am just a newbie regarding both Nixos and zfs, but I try to give you a couple of ideas (I hope the experts will give you better help).
First, I would try to get a working setup with legacy mounts. If it works, then at least we have a working starting point.
Second, for what I see, all the examples I was able to find uses the legacy mountpoints, included the official guide to zfs on open-zfs site. So this make me wander whether new mount points are actually supported, or if there are reasons that prevent people to use them.
About the wiki, have you tested the parameters reported in this section?
It seems like it should just work.
Here I link the repo in which I am sharing the exploration I am doing regarding Nixos, so you can have a look to the zfs part, maybe it could be useful for you:
Please note that it is a work in progress. Also, I was able to mount by using the
boot.zfs.forceImportRoot = true; option, without it I was not able find a set of working configurations. Maybe you just need it as well.
Btw, I am currently working on other topics, but as soon as possible i will turn back to work on zfs, so in case you are able to find a solution, please share.
I’ve not tried this (still learning Nixos so not tried anything advanced) but here are some thoughts…
You may need boot.zfs.extraPools = [ “zpool_name” ]; instead of the legacy items (so this doesn’t really help your clean-up).
The pool needs to be created with canmount=on - the OpenZFS wiki has the boot and root with canmount=off, so if you followed this guide I don’t think it will work.
Thanks for your answer @FStefanni,
My current setup is not using legacy mountpoints and is booting correctly. The “issue” I am having is related to the “cleanness” of the setup: even though ZFS could be managing all mounts, that is not working. Instead, I still have to enumerate all ZFS mountpoints in
hardware-configuration.nix. So this kind of defeats the purpose.
I will try to reformulate the question this afternoon, as experimentation has lead me into a rabbit hole of
initrd details with
/mnt-root issues. In the meantime, if someone could point to a clear explanation of NixOS boot phases and all? I am plowing through the code now
Yeah did both those things. As mentioned to Fransesco I will reformulate and then also include my setup.
It’s unavoidable that you need the
fileSystems definitions that are necessary for stage 1 to boot. These are listed here: https://github.com/NixOS/nixpkgs/blob/0a2745684e8e97340d947d56b0ba778056c10395/nixos/lib/utils.nix#L13-L14
Namely, any datasets containing these:
These have to be mounted by stage 1. Datasets with non-legacy mountpoints are mounted in stage 2 by a service called
zfs-mount.service. You can keep the non-legacy mountpoints on these stage 1 file systems, but there’s not really any benefit to it because you do need the
fileSystems entries, and you have to add
zfsutil to their options if you do.
Ah thanks, a very clear answer although not what I was looking for
Are there any ways around this? Running
systemd also in stage 1 for example? If one wants to change this behaviour, where would one start?
PS @ElvishJerricco do you know of a comprehensive write-up on NixOS booting incl stages etc? Both the manual and the wiki are sparse on details.
How would systemd find the units, if you didn’t mount
/etc yet? Which again requires
/nix/store to be mounted for the cascade of links to follow.
So to be able to start that service, you need at least
/etc mounted the “legacy” way.
Just do yourself a favor, and keep the stage 1 entries, and do the rest in ZFS mounts. Your config will still be a lot cleaner than mine, where I have to mount every LV possible, as LVM does not have an equivalent of ZFS mounts.
Thanks, I’m convinced.
On second thought, I’m not
During stage 1
zfsutils are loaded, so mounting all datasets at their respective mount points should be no problem. The thing is, I have no idea of the specific role of
/mnt-root yet, but that should be solvable.
So what might be possible: during stage 1, mount all ZFS datasets, do a bind mount to
/mnt-root (again, I don’t yet understand what role this plays so maybe something else is better), then, at stage 2 hand everything over to
zfs-mount.service (which may come to the conclusion that all mounts are already established).
So while I have other fish to fry, I still believe a ZFS-native mount process during boot is possible. NixOS just doesn’t support it yet.
There is also disko
I’ve used it yet, I’m also looking into it.