NixOS on encrypted ZFS for personal machine?

Help
1

For past 15 years, basically my entire linux life, I’ve been happily installing distros on ext4. It has yet to fail me.

I don’t know anything about filesystems beyond their names and rough feature set. My use of ext4 is limited to creating a partition, installing and OS to it and forgetting it exists.

But In have a new laptop coming in, and I was checking out ZFS as possible candidate for future filesystem on my personal machine.

There are aLOT of stuff on using NixOS on ZFS, but compared to ext4, it seems needlessly complicated and I’m not grokking it in one go.

This is my machine:

  1. Single 512G SSD
  2. 16G RAM
  3. Regular laptop with regular developer work

These are my criteria:

  1. Encryption (I don’t want to have unencrypted machine any more)
  2. That’s it. Ext4 is fine, but I’ve heard too many good things about ZFS over years.

These are my questions:

  1. Will ZFS even be a good fit/improvement for my use-case?
  2. How complicated is the setup? ( I found this and this to be simpler steps, but not sure if they’re still ideal ways to go)
  3. What kind of performance impact should I expect? I’ve read ZFS requires more RAM than ext, and considering this machine has non-upgradable memory I want to be future proof (5-8 years expected machine lifetime)
  4. What kind of maintenance overhead should I expect? In past 15 years I remember exactly 0 interaction/maintenance done on ext4 partitions beyond initial mkfs, but ZFS has extensive suit of tools that I’m not sure I’ll be up ti learn anytime soon)
1 Like
2

If you just want to use the system, stick to EXT4 on LUKS/LVM.

ZFS and BTRFS are definitely not “fire and forget”, especially ZFS will cause you a lot of pain if you need docker on the machine.

3 Likes
3

I think people tend to overstate the amount of complication in ZFS. The main benefits for something like a laptop would probably be:

  1. Checksumming. If data ever bitrots on the drive, ZFS will know and it’ll return IO errors instead of corrupted data.
  2. Snapshots. You can instantly take a CoW snapshot of your file system. This has saved my bacon more times than I can count. I have my systems setup to take snapshots automatically at least every hour, and being able to back and fix a mistake is surprisingly useful, even as someone who checks all my code into git.
  3. Transparent compression. This is hard to understate. My Nix store takes about half as much space as it otherwise would because ZFS will automatically compresses the entire file system without any real performance overhead.

There’s a bit of knowhow involved in first creating the FS, but after that I don’t really understand the claim that it’s not “fire and forget”. Once it’s set up you just… use it, like any other file system. I’ve never had issues with Docker (though I only use it on occasion).

Anyway, to answer the questions:

  1. ZFS will be a good fit if you like the things I mentioned above.
  2. As said, I don’t think it’s that complicated. It’s annoying that there’s so many opinions about how you should setup ZFS, but frankly the differences between them are unimportant for the most part, so just picking one and going with it will do you fine.
  3. ZFS’s performance is known to be weaker than other FSes like ext4 or xfs, but frankly even with NVMe, your bottleneck is much more likely to be the disk unless you have several of them in a RAID array of some kind. Oh and performance can sometimes be better because compressed data takes less time to read off disk. As for RAM usage, this is sort of a myth. ZFS does “use” a lot of RAM, but that’s because it has its own (much more sophisticated and effective) in-memory read cache that can grow to up to half of your system RAM. But it will evict that cache as your applications need RAM, so this is more of a turbo than a risk.
  4. Again, I don’t think it’s a maintenance burden most of the time. Unless you’re the kind of person who really wants to mess with the stuff ZFS lets you mess with, it’ll get out of your way after you set it up.
4 Likes
4

Thanks for detailed answer. I’m not really worried about snapshots (have my own hourly backup solution working well) but compression for Nix storeis intriguing.

Would you happen to have a convenient and simple step-by-step.guide I can follow with little/no changes necessary on my part? As mentioned above, I know as much filesystem as my cat does and that is unlikely to change. But if I get transparent encryption+compression without overhead, I’m prepared to put in one-time effort of a while

If you don’t have a simple guide, can I ask you to check any of the guides I linked in OP and tell me which would be better/relevant?

5

Both of those guides you linked are pretty straightforward and unopinionated; seem fine to me. The first one uses ZFS native encryption and the second one uses LUKS. If it’s all the same to you, I’d use LUKS. There are reasons to prefer ZFS native encryption, but there have been reports of buggy behavior with it, since it hasn’t really been given the attention it deserves since it was merged a few years ago. I personally use it and have never had issues with it, but the reasons I like it are unlikely to be relevant to you.

2 Likes
6

Thank you, I’ll try this one out.

7

Why? With the Docker ZFS driver I haven’t had any issue yet.,

8

Besides creating a container takes minutes to hours, as the ZFS integration is awfully slow?

9

ZFS with NixOS works fine (never tried docker, so can’t comment on that). I modified a shell script I found online - basically you create ZFS pool (you need a compatible boot pool or FAT partition for that). I use the standard ZFS encryption - the “downside” is that the message to enter your password can get lost in the boot messages (very minor issue). Although I haven’t got it working yet, you should be able to unlock the pool remotely using ssh.

When you have got the hang of ZFS it seems simpler (to me) than a standard filesystem…and is pretty much fire and forget in my opinion and the tools are more logical than most filesystem commands. I don’t do anything unless it tells me there is an issue, in which case recovery is much easier with a RAID setup (not that you have that).

In terms of memory usage, I have a raspberry pi (2Gb) with ZFS. Currently just running the “small” NixOS and the folding at home app, it is using about 1Gb, It will use free memory when available (my 16Gb laptop is using about 15Gb).

Suffice to say I use ZFS everywhere and have had no issues so far!

1 Like
10

There is an impression that zfs is “more work” — more to learn and know, more to do, more detail to keep track of and manage, etc.

There is some truth in this, particularly the “learning from scratch” part, but it’s fundamentally a little bit of misconception. ZFS gives you better tools to manage policy/preferences over organising your data and how it’s handled: things like compression, redundancy, backup, usage and quota allocations, etc.

In your ext4 example, you’ve made one big / and put everything in it, and this is what you’re used to. From that perspective, creating and managing datasets to subdivide your disk contents does seem like more work, because it seems like a thing you just don’t do. It turns out the main reason you don’t is because static partitioning and separate filesystems (which we used to do) is just too hard. The tools that ext4 (and similar) filesystems give you are too cumbersome; even with more flexible partitioning (like LVM) people eventually decided to give up on separate allocations.

But you still have to manage space: eventually the disk fills and you have to go trawling around (with tools like du) to work out what’s using it and what you can clean up. This “simplicity” is just deferred work, and the work is harder once you have to do it.

ZFS, on top of (and as part of) all the other things mentioned above, gives you really easy tools to manage your disk space, because making additional datasets is essentially trivial and supports the accounting (with commands like zfs list -o space) so you can see quickly where your space has gone according to the way you’ve chosen to organise your data.

All the rest of the benefits come from then applying policy settings on top of this: compression, quotas, snapshots, backup replication, etc. You get these benefits by applying policy to data how you’ve organised it (and by excluding some stuff from it). Especially if some of the data is cache that is application-managed and will clean when its filesystem starts getting full, doesn’t need to be backed up, etc.

If your intent is just to use zfs like ext4 and have everything in one big /, you still get some benefits (mostly integrity protection / detection), and it’s just as ‘simple’, but you’re not really making use of the opportunity.

1 Like