Good filesystem for the nix store

I saw there is this workaround to BTRFS not compacting /nix preallocateContents option: disable by default by martinetd · Pull Request #4094 · NixOS/nix · GitHub. The way to activate this workaround is with this option: nix.extraOptions = "preallocate-contents = false";.

But if I activate it I see this warning: warning: unknown setting 'preallocate-contents'

This option is not available in NixOS unstable?

if you do a lot of PR reviews, you will likely have a lot of similar files. So zfs with compression+dedup actually works pretty well:

$ zpool list nixstore
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
nixstore  1.81T   807G  1.02T        -         -    43%    43%  3.06x    ONLINE  -
$ zfs get compressratio nixstore/store
NAME            PROPERTY       VALUE  SOURCE
nixstore/store  compressratio  1.91x  -

close to 6x compress*dedup ratio

EDIT: I should also mention that dedup takes up a massive amount of RAM, so I wouldn’t do it without good justification

4 Likes

In Btrfs from time to time I force compression with this command:

sudo btrfs filesystem defragment -r -v -czstd /nix

And dedup:

sudo duperemove -drv /nix
1 Like

Wouldn’t Nix’s auto-optimise-store option make dedupe unnecessary?

let me check, it will take a while to run. But I think dedup runs on a per block instance, where as optimise just hardlinks files… And I’ve had cases where optimize reported negative savings before as well

2 Likes

True, but I wouldn’t expect block level dedup to occur very often unless the files the blocks are from are identical.

Yea, but when reviewing PR’s (including those targeting staging), it seems to be pretty common.

2 Likes

I catalogued my zfs usage here:

In short, despite autoOptimise on, I had a dedup of 1.35.

(I’ve turned off autoOptimise now)

I should also note that in my system of 8GB ram, nix-env --install irrespective of size of derivation are regular but nixos-rebuild takes my RAM usage from 30% of 98% and the system comes to a stand-still. I am not sure how to explain this …

FWIW (preallocate-contents option in nixUnstable), it should be available in nixos-unstable but not in 20.09 yet – someone would need to backport the update to stable for that.

Since the PR for auto-detection was brought up here I’ve updated it & it should be ready, let’s see how that goes, but it also won’t be available right away either. It’s not too complicated to update though, and nix seems well tested so I don’t think a PR updating nixUnstable would be refused :slight_smile:

1 Like

@asmadeus I’m using NixOS-unstable. I tried to nixUnstable but it’s not there yet.

Ah, my bad.
nixpkgs’s master branch’s f25bb567f608 ("nixUnstable: 3.0pre20200829_f156513 -> 3.0pre20201020_e0ca98c") updates nixUnstable to a sufficiently recent version, but it’s not in unstable yet. I somehow assumed nixos-unstable was directly equivalent to master but it’s not, sorry.

I’m not sure how the branches are updated, it looks linear to me. nixos-unstable-small has it but not nixos-unstable yet for some reason – then again the commit is only two days old so I assume it’ll get there sooner rather than later…

1 Like

the release channels have to finish all builds, and all “jobs” need to be successful for it to be updated.

Although nix{os,pkgs}-unstable follows master, it will always lag behind, sometimes for days.

Usually the long delays are because some aarch64 or darwin build is taking forever to complete, since those are architectures/platforms with less hardware dedicated to them.

EDIT: not to mention the aarch64 builder just has really slow single core performance

2 Likes

Hi @asmadeus! It seems that preallocate-contents is available now with nixUnstable, the version of nixUnstable is 3.0pre20201020_e0ca98c. I’m not seeing more that warning about preallocate-contents being a unknown setting:

❯ sudo nixos-rebuild switch                                                                                                                                                           
[sudo] senha para thiago: 
building Nix...
building the system configuration...
activating the configuration...
setting up /etc...
reloading user units for gdm...
reloading user units for thiago...
setting up tmpfiles
1 Like

Yep! It finally built a couple of days ago according to https://status.nixos.org :slight_smile:

Now just need preallocateContents option: disable by default by martinetd · Pull Request #4094 · NixOS/nix · GitHub reviewed & merged for automatic detection, but at least we can explicitly enable it for now

1 Like

The preallocate-contents option shouldn’t be necessary right? Since fallocate has been disabled by default, apparently.

1 Like

yes, if you use nixUnstable in any form in the unstable channel afaik (this hasn’t been backported to stable nix nor to stable channels)

1 Like

Maybe unrelated, but zfs is also a heavy RAM user, especially with dedup.

Only if you let it. It can pretty much use as little RAM as you ask it to; it just defaults to using a ton because the ARC is sophisticated enough to actually make great use of a large cache size, unlike the regular linux page cache. Dedup is of course the exception, but even then only for write workloads.

2 Likes

That’s true but when you make it too small you might turn a memory problem into a CPU problem because then the kernel is constantly busy evicting blocks from the ARC ([arc_prune]). The zpool on my laptop has 116 GB but setting zfs.zfs_arc_max to 512 MB already resulted in huge CPU load. Now my ARC is 1 GB which is 1/4 of my RAM but that actually works fine.

1 Like

Hm. Yea I guess I’ve never seriously considered < 1GB of RAM for ARC, which to me is pretty small. But if you have such requirements, you do make a valid point.

1 Like