Hi all, at the moment, when I can’t use last kernel because is incompatible with zfs I’m thinking if this fs is very helpful with Nixos or if others fs are more suitable to nixos. Currently I don’t understand if with snapshots of root I can be rollback or no
1 Like
ZFS is still quite useful IMO. Even though NixOS generations make snapshots of the nix store largely redundant, they’re still useful for things like /var and /home. But perhaps more importantly, ZFS adds transparent checksumming to ensure correctness of data. Personally I find this critical. ZFS also does transparent compression, which is especially effective on the nix store (my compressratio is typically over 1.5x). I also use ZFS’s various multi-device modes on various systems for various different reasons.
ZFS is great. Probably the biggest issues with it are that swap can’t be stored on ZFS and hibernation can’t be used at all without risking destroying the pool.
3 Likes
I’m using both ZFS and btrfs,
and for my personal preference - i like btrfs more: it’s in-tree, it sorta blends in much more naturally, it has most of the benefits of ZFS a regular user would want
2 Likes
btrfs is also pretty good as long as you avoid its raid modes. Even its raid1 is pretty lousy IMO.
1 Like
What feature of zfs do you personally care about? Difficult to answer without knowing that.
My major gripe with ZFS was atuin being unusably slow. I think this has been fixed in more recent versions. But, the nixos issue is still open.
Disclaimer: hardware failure is a somewhat niche usecase, but maybe worth consideration.
I had broken ram a few months ago (ram bits would get flipped on read according to memtest86+). ZFS couldn’t handle it; the pool metadata would quickly become corrupted and I couldn’t mount any of the devices in the pool. When I tried, the machine would hang.
With btrfs, the fs would mount and the system would function normally. I only noticed the issue because large nix derivations didn’t reproducibly build. Btrfs for better or worse was able to mask failing ram pretty well.
Don’t rely on this. There have been plenty of btrfs filesystems that were irrecoverably broken by faulty RAM.
No filesystem can actually protect you against RAM corruption in any meaningful way. This was pure luck or btrfs had a bug and didn’t discover an actually existing issue.
5 Likes
I have used both ZFS and btrfs extensively and switched back and forth a few times.
I want to echo the opinions expressed in this thread:
Btrfs is much better integrated and thereby nicer to use in most cases and offers most of the features you’d usually want. It’s also a lot more flexible which can be important for desktop use.
ZFS is a lot more suited to large server deployments with dozens of disks. Btrfs performs poorly for those; both in features and performance.
If you need to use a newer kernel than the latest LTS kernel, ZFS is not an option. It’s not compatible with the oldest non-LTS kernel most of the time, leaving the latest LTS kernel as the newest option.
1 Like
I’m using nixos on zfs because I’m use same partition and same pool on arch. I love use snapshot and zfsbootmenu as bootloader
This is tangential, but uh, running two differnet systems from zfs doesnt sound like a great idea. Does it actually work well? Id expect problems if the zfs version differ.
Definitively YES.
- snapshots as they should be form an operation point of view
- physical storage configuration as it should be from the same point of view
- volumes out of pooled storage easy to tweak with reserved space and maximum growth
- send/recv
Zfs is the most modern, or least obsolete storage system we have. Like NixOS is the most modern, mature, OS we have. Because after the '80s storage should have been like ZFS and OSes declarative like NixOS and we miss them for so long mainly due to bad IT evolution and widespread ignorance.
(I’ve done multiple nixos systems on the same zfs pool in this incredibly stupid way and I don’t recommend it but it is fun :P)
1 Like
With ZFS I only ever really notice performance issues during really heavy I/O workloads, like if I’m compiling a big Nix package, or when steam is unpacking a game after downloading it. Otherwise it’s like any other system.
ZFS has better behavior regarding disk full situations, i.e. it reserves space for deleting files.
Btrfs on the other hand cannot even properly calculate how much space is still free.
Also I like having autosnapshots enabled in zfs (every 15 minutes; life saver) and Btrfs seems to have performance problems with many snapshots. In fact when I used snapper in Btrfs, I once lost access to all my files on this system because of corruptions. If a file in zfs is corrupt it will just report the broken file and returns an IO error when reading it. Btrfs historically was impossible to mount when checksum errors occurs. These days it seems to have special mount flags that have to be set in this case. Still a deal breaker for me. I expect a filesystem to always let me read all files that are not broken instead of refusing service. Most of the time those files are not that important and most of the system is still usable.
1 Like
But are you using auto snapshot also with root dataset? Can you restore old root snapshot?
Currently I’m using zfs-2.2.6 on chimera and 2.3.0-rc on arch and nixos. These 3 distros are in the same pool and I don’t have problems (at the moment)
BTRFS and ZFS user here. They are very similar, but differences have already been mentioned. Even tho right now I’m running BTRFS on my main system - I recommend ZFS only because CLI is much nicer 
For kernel annoyances, one may do this:
boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
1 Like
Oh, just remembered. I remember some users were getting /nix/store corrupt due to ssd’s incorrectly reporting something on BTRFS… I have not encountered this issue on Samsung ssd’s
No, see zfs.latestCompatibleLinuxPackages is deprecated for more info.
Anyway, this is why I use good ol’ ext4 in 2024. The only upside of btrfs and zfs is teaching you the value of proper backup schemes, the hard way.
1 Like
I haven’t had one recently but I’m pretty sure btrfs has much improved in this regard.
It sure can. Here’s mine:
$ sudo btrfs filesystem usage /
Place your right index finger on the fingerprint reader
Overall:
Device size: 1.77TiB
Device allocated: 1.17TiB
Device unallocated: 611.92GiB
Device missing: 0.00B
Device slack: 0.00B
Used: 1.16TiB
Free (estimated): 616.59GiB (min: 310.63GiB)
Free (statfs, df): 616.59GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data,RAID0: Size:1.13TiB, Used:1.13TiB (99.60%)
/dev/dm-0 1.13TiB
Metadata,DUP: Size:20.00GiB, Used:19.14GiB (95.69%)
/dev/dm-0 40.00GiB
System,DUP: Size:32.00MiB, Used:144.00KiB (0.44%)
/dev/dm-0 64.00MiB
Unallocated:
/dev/dm-0 611.92GiB
So ~600GiB free and that’s also what df tells me:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/dm-0 1.8T 1.2T 617G 66% /
<snip>
I snapshot every 15 minutes and don’t have any performance issues. What you need to look out for is keeping too many snapshots around at any given time and I don’t see a use-case for keeping more than 30 snapshots at a time.
I have 19 snapshots of my user data subvol ranging back to 1 month ago and I observe no performance issues whatsoever.
It depends on what kind of corruption it is. If a file is corrupted, btrfs will just also simply return IO errors for that file but otherwise function normally.
If the metadata is corrupted, btrfs will (rightfully IMHO) fail loudly and refuse to mount or turn itself read-only. If it was just some dnode or inode that was corrupted, I could see the case being made that it should be able to ignore that but metadata houses critical data structures such as the extent tree and it cannot function if that isn’t integer.
With ZFS, if all copies of a piece of metadata are corrupted, …I don’t actually know what would happen because I don’t think I’ve ever seen any sort of recovery path mentioned in the ZFS docs.
I assume you’re just screwed?
2 Likes
The changelog entry for this is worded very incomprehensibly and I am still not sure what this now points to, if I actually need to set a Kernel version manually, or if I can just remove this and the default Kernel (when no other Kernel is selected anywhere) is the latest LTS Kernel.
Like, wat? I thought default linuxPackages is the latest LTS Kernel (at least according to the Wiki), but it’s not possible and I should use a LTS Kernel? Wat?