Hey everyone. I’m considering trying out NixOS. I’m currently on Void Linux, mainly because it was the distro I found with the best ZFS support. I have one 1TB boot drive in a zroot pool and one 4TB storage drive in a zstorage pool. No RAID or anything like that for now.
I’m using ZFS-native encryption, zroot is encrypted with a passphrase, and zstorage uses a keyfile that is in zroot, so it decrypts automatically with a script when zroot is decrypted. I think that’s the only complicated thing in my setup.
Obviously I have to erase zroot to fresh install NixOS, but ideally I’d like to be able to not touch zstorage and just reimport it on the new install. If this is impossible it’s not a dealbreaker though.
I see that my installed ZFS version on Void is 2.2.3, while Nixpkgs seems to have version 2.2.2. Can I downgrade the pool somehow so it’s safe to import in a previous version?
Aside from installation questions, how is the experience using ZFS-only on NixOS? Has it ever broken? Is it worry-free? Does it update when it has to?.. If you have a system with it how long has it worked?
The install can be a bit more clunky than say Fedora if you want to use flakes for example (but not really any harder).
In terms of the system being worry free, well NixOS can always be fun - especially if not an expert. But from the using ZFS as the filesystem rock solid. I have several builds including respberry Pi with ZFS. I tend to use ZFS even with single disks as I then have a consistent command set to manage disks (zpool status etc).
This machine is from September 2023 and I’ve being using zfs root on general systems for just over 2 years.
In comparison to anything apt/dpkg based, you can no longer end up with a kernel that zfs doesn’t compile with.
Hö? How that? You just format your disk, generate the hardware config, do a few necessary adjustments when not using legacy mounts and you are mostly done.
I think that a zfs root install using flakes is more complex than it needs to be, or more long-winded depending how you approach it. Also the suggested config in the ZFS root documentation had a complicated example config (which is no longer there). For convenience, I’d really like the GUI to be able to handle ZFS for wider adoption and another killer feature of NixOS.
But to confirm, the process is not overly difficult.
Maybe slightly offtopic, but what are the benefits of ZFS vs btrfs on NixOS?
Seems like btrfs would eventually eat a little bit less RAM and might be a bit simpler?
There are probably more than this but off the top of my head:
ZFS is more resilient and less prone to corruption
ZFS has zvols with are basically virtual block devices
ZFS snapshots are part of the dataset and not separate subvolumes as in btrfs
ZFS is far more tunable
This is definitely true. Although, the amount of RAM ZFS uses is also tunable.
Btrfs is much simpler and can be used as any other filesystem without much knowledge. ZFS requires you to learn about it and configure it for optimal performance with your specific use case.
Ultimately, it depends what you features and functionality you care about. ZFS and btrfs both have use cases.
There’s a few things to be aware of, and most of this is unrelated to NixOS and applies to any distro with ZFS.
You cannot hibernate at all when ZFS is in use. It “works” but it still has known bugs (no, it wasn’t all fixed) that will kill any imported pools. NixOS flat out disables hibernation by default when using ZFS because of this.
Regardless of that, you still cannot store swap on ZFS. It “works” but it also still has known bugs (again, no, it wasn’t all fixed) though thankfully this will just crash your system rather than killing pools. So no swap on swapfile or on zvol. You need a separate partition if you use swap.
Native encryption is iffy. It has a laundry list of known bugs that have been unfixed for years now. Thankfully these bugs are pretty rarely encountered, and most of them only occur when using zfs send --raw, and they usually just crash the system rather than causing data loss. But there’s still a small chance that native encryption will just kill a dataset, or even a dataset that you replicate to even without --raw. Most people report no problems for long periods of time, but the bug list is long and largely unmitigated.
Finally, and this one is specific to NixOS, there’s the issue of mountpoints. This isn’t a problem, it’s just something to be aware of. The short version is that if you want to make things simple and don’t want to think about things, then you should use mountpoint=legacy and let nixos-generate-config put your file systems in hardware-configuration.nix like you would with any other file system. The long answer is that you can use non-legacy mountpoints for anything that you don’t list in hardware-configuration.nixexcept that the ones needed for boot (/ , /nix , and a couple others) do need to be in hardware-configuration.nix with the FS option zfsutil . But again, if you just use legacy, you don’t even have to think about it.
All that said, I use ZFS pretty much exclusively on all my systems, even the SD card on my rpi-based router. It’s been super reliable. I even use native encryption on a couple of them (knock on wood). But I also know what I’m doing to some degree.
I am a lot of Linux forums and I see people with corrupt btrfs filesystems regularly. To be clear, it is probably a very small percentage overall but it is definitely more prevalent than with any other common filesystem.
Don’t take my comment the wrong way. I said that ZFS is “more resilient”.
It is a relative statement. That doesn’t mean that btrfs is unreliable or unusable.
Native encryption is iffy. It has a laundry list of known bugs that have been unfixed for years now.
Oh snap. I didn’t know that. Then I can’t just reimport.
I guess this is unrelated to ZFS, but is there a way to have my setup with LUKS? With both drives encrypted, but I only have to decrypt one and the other is decrypted automatically. Like I said, I made my storage zpool be encrypted with a keyfile that was in my boot zpool.
Don’t take that the wrong way. ZFS encryption isn’t any different on Nixos than any other distro. If it is working fine for you now, it isn’t likely to be any different in NixOS.
You should be able to re-use the existing pool if you want to.
This should be put as a disclaimer to the Wiki, mirrors exactly everything I’ve learned over the time about encrypted zfs (and the other aspects as well). I’ve now switched back to LUKS + ZFS on some devices, LUKS is still the best block storage encryption system, unless you want to send encrypted pools to remote pools there is no massive benefit of using zfs native encryption (expect simplicity).
One last thing to add, ZFS on root with nixos-unstable is sometimes a bit sad concerning Kernel upgrades, which makes a difference if you are using hardware that still gets a lot of Kernel improvements.
You can use NixOS unstable for the time being which is already on ZFS 2.2.3. I’ve been exclusively using NixOS unstable on my personal hardware for several years now and it works quite well for a rolling-release cycle.
Generally, ZFS will ask for permission when upgrading the zpool to a newer version, so switching from 2.2.3 to 2.2.2 shouldn’t be an issue. I haven’t run zfs upgrade since I’ve created this pool and it reports no pending upgrades, so I would expect it being compatible with 2.2.2.
zfs upgrade
This system is currently running ZFS filesystem version 5.
All filesystems are formatted with the current version.
NixOS has first class support for ZFS but only for the “official” kernel. If you want to use other kernels (available in nixpkgs) like -xanmod, -rt, etc., you may end up with in an unbuildable state. This often happens when the “other” kernel receives an update that breaks ZFS and upstream ZFS (and ZFS packages) didn’t catch up with it yet. Fair to say though I’m on -unstable, ymmv if you’re on stable.
Other than that, the mountpoints are a bit tricky but so far I found that only a handful of mountpoints need to be statically declared (mountpoint=legacy and mounts set in hardware-config.nix). Specifically, in my case it’s these:
/
/var
/var/lib/docker
/nix
/nix/store
I’ve got about 10 other datasets mounted via the mountpoint prop (things like stuff in /home, some backups, etc.)
I’ve been using this setup for years now without issues. I think it might be possible to reduce the static list even further but I didn’t try (yet). Not even sure why I have /nix and /nix/store on separate datasets come to think of it…
As to encryption, I’m using native ZFS. No problems so far.
And btw, imho you don’t necessarily need to trash zroot pool, you can just create new datasets for NixOS usage.
Hey everyone, I’ve been experimenting with NixOS on a VM. I didn’t have any issues with the installation, but I’m having a really hard time trying to get a secondary storage drive to be decrypted and mounted automatically.
I’m using LUKS instead of ZFS native encryption which I guess is the issue.
I tried to do the same trick I did in my current install. I formatted the second drive with a keyfile instead of a keyphrase. I’ve been trying to get it to work with /etc/crypttab like this:
No luck though. Rebuilding this configuration crashed the entire system and put me in systemd’s emergency mode. journalctl -xe just says "timed out waiting for /dev/disk/by-uuid/<the uuid>, didn’t find any other useful info.
Trying to boot the configuration gets to stage 2, then hangs with two start jobs, one called Import ZFS pool "zstorage" and the other is just called /dev/disk/by-uuid/<the uuid>. After one times out, I’m in emergency mode again.