What is “the recommended” filesystem to install nixos on?
ZFS is great, however it lacks an installation guide. wiki presents a guide that forces an overcomplicated config on a beginner, and doesn’t help at all to understand what is actually happening. (I myself seem to have caught a version of that guide that was not yet overcomplicated, but i’m using legacy mountpoints and simple fileSystem.<mntpoin> = {} kind of simplest configuration)
I don’t think there’s a recommendation that differs from your usual filesystem recommendations for Linux systems. I think it roughly is:
I don’t think NixOS makes either of the filesystems particularly more or less interesting than normal. If anything, it makes ext4’s lack of snapshots less problematic, and zfs’ licensing issues a bit less problematic since there are a fair few zfs users among us.
I’ve started using btrfs for all new installs I’m doing. From my research, it seems to be the sweet spot right now, other than certain RAID setups which it apparently doesn’t handle very well yet.
If you don’t need legacy ext4 support, and don’t need specific advanced features that ZFS or XFS would bring, then I’d say go with btrfs.
One of the reasons I prefer btrfs over zfs for NixOS is because encrypted boot with zfs is tricky on NixOS, since the native encryption doesn’t work via LUKS and therefore falls flat in the face of the default init scripts (and afaict systemd can’t decrypt it at all yet, which will probably be a more significant problem in the future). My bad, @uep says it does work.
Block-based encryption is a really enticing feature though, maybe one day. Or btrfs gets block-based encryption one day, who knows?
Also the zfs instructions are indeed awkward, what with its pre-written system configuration that isn’t composable at all and lacks any explanations. As much as I understand their desire to minimize documentation maintenance overhead and complexity, the current state is awful.
Thank you for advices
What i look for in an FS is:
and a separate application:
zfs seem cool, but having it as dkms already caused a little bit of hassle for me (not on nixos)
never tried btrfs actually…
Just for clarity: both zfs native encryption and zfs-on-luks work fine, with both nixos scripted and systemd stage1 options. I’m typing this now on one of several systems using zfs native encryption and stage1 systemd.
I won’t dispute assertions about the beginner view of various installation guides; I was already well familiar with, and had strong opinions of my own about, zfs configurations before switching to nixos, so my view of the guides was very much more focused only on certain nixos-specific items.
And, yes, for me one of the very big advantages of zfs is the send/recv functionality for backup replication.
Much less so for os root config, but no less so than any other platform for your actual data.
Hrm, guess I’ll have to try that again.
Aside from the ability to restore a previous version of “home”, is there any way that btrfs snapshots do more, or are better, than NixOS generations?
I’m surprised nobody mentioned XFS.
In my opinion there’s no reason anyone should use EXT4 anymore. XFS has copy on write and it does dynamic inode allocation. It is rock solid. And the default for Red hat based server since version 7 almost 8 years ago. The default for mkfs.ext4 uses 5% of your disk space for inodes, which on today’s drives adds up to a pretty large amount of space wasted if you’re doing large files or on your /nix I’ve heard you could run out of inodes.
If you’re l looking for more, I use btrfs for single drives. If you have a physical server that needs the raid functionality of ZFS then you should look at that.
But on a single drive I think btrfs is the stand out.
On a slim server NixOS VM install on btrfs with compression=zstd /nix is 450mb. It’s all main line code, no jumping through hoops for ZFS install with arguably little to no gain.
if you store any useful data on your server, then yes. I use btrbk which is nicely integrated in nixos
it makes snapshots of subvolumes on a schedule keeping local backups. useful if you accidentally overwrite a file, or inadvertently try an option on minio that expires all your objects 
To be clear, while NixOS can restore previous generations of your configuration (i.e., everything in configuration.nix), it cannot restore data you add while the system is running, like any images, code repositories, gpg keys, any of that.
zfs/btrfs/xfs snapshots are literally copies of every file on the disk at the time, except they only take more space if you later edit or delete those files because everything is copy-on-write (actually, blocks, so the extra space is even more condensed, especially for small edits).
This is an incredibly cool feature, since it allows you to just go back in time as you please. It also makes backups really easy, since you can send the data to a second disk with the same filesystem and it will retain the deduplication, so sequential backups take much less space without losing any data as well.
These filesystems are really cool, if you’re not familiar with them go have a read on wikipedia or such for a basic overview.
It’s always scary to play around with a new FS on your workstation. You might misunderstand something and what you think is a backup is not a backup at all! or something gets corrupted in incremental ones.
I’m still confused for example, if CoW has a severe impact on ssd lifespan when working with VMs (since vm’s disk image changes should cause a lot of reallocations on btrfs, no?)
There is one more caveat with ZFS: hibernation. (see also Using ZFS in combination with suspend to disk may cause filesystem corruption · Issue #106093 · NixOS/nixpkgs · GitHub)
I had a hibernate/resume issue on a laptop that somehow completely destroyed the ZFS header so my only option was to wipe the disk and redeploy.
CoW typically only writes any new data to disk, instead of overwriting the current data. That way, if the we fail writing the new data to disk, the old data is still there and is valid.
The old data will be marked as stale/can be overwritten after the new write is written to disk, so it can be used later on.
i understand, but imagine you have one 50GB file (like qemu’s or virtualbox’s disk image file) that constantly has small changes in different places inside.
I don’t know details if the whole file is reallocated on each small change, or are files split into cow blocks (in btrfs for example). So strain on physical ssd seem to be bigger in any case, but how bigger? enough to worry?
This.
If your VM is backed by a qcow image, that file is internally split into COW blocks as well.
I don’t think any modern filesystem regularly shunts around entire files as a unit, even if SSD writes weren’t a consideration that’s a recipe for fragmentation. Nor does it map well to the hardware. That’d be one silly filesystem.
I’d conjecture you get fewer writes with general usage, since copying no longer means writing a full copy of the original file, until it is edited (and even then you have fewer writes until you’ve changed every block of the file).
I didn’t get this point. I have used BTRFS in the past, and the snapshots were always stored on the same disk, making it somewhat risky as a backup system. How do you send data to another disk?
The btrfs snapshots can be send to another location. This article is from Fedora Magazine, but explains briefly how it works. For further details see the btrfs manual.
It’s called “write amplification” and with ZFS it’s controlled by the recordsize setting. Any write at all to a large file will cause the entire recordsize-aligned block containing that write to be rewritten in its entirety. These writes are often aggregated, even when they’re synchronous via the ZIL, so it’s not too bad. But if you don’t tune recordsize appropriately for some workloads it can lead to performance problems.