Good filesystem for the nix store

8

I format my disks with xfs and pass “-m reflink=1” to mkfs.xfs to enable CoW for data. I tried many filesystems before including ext4, btrfs and f2fs, but they were all insufficient in one or another way.

  • ext4: doesn’t support reflinks
  • btrfs: sometimes destroys itself with no way to recover the partition
  • f2fs: needs a long time to perform fsck after kernel updates; eats files that are open when the computer has power loss

I automatically run the gc on all my systems so space does not become an issue so easily.

3 Likes
9

I’ve never noticed and thus never investigated. I suspect it might be a side-effect of Nix juggling with read-only remounts, unsharing, etc.

10

I observe that /nix/store btrfs compression property gets inherited for locally built store paths as opposed to substituted from binary cache ones.
As evidence, grub2 2.02 wasn’t able to boot NixOS after I added an out-of-tree (locally built) kernel module to boot.extraModulePackages because of unsupported zstd compression of init and initrd files. grub2 2.04 successfully reads zstd compressed files.

1 Like
11

I can confirm the same issues with f2fs, I think I have thread about it on discourse actually.

12

Thanks; let me open a ticket: btrfs compression ignored · Issue #3550 · NixOS/nix · GitHub

1 Like
13

I saw there is this workaround to BTRFS not compacting /nix preallocateContents option: disable by default by martinetd · Pull Request #4094 · NixOS/nix · GitHub. The way to activate this workaround is with this option: nix.extraOptions = "preallocate-contents = false";.

But if I activate it I see this warning: warning: unknown setting 'preallocate-contents'

This option is not available in NixOS unstable?

14

if you do a lot of PR reviews, you will likely have a lot of similar files. So zfs with compression+dedup actually works pretty well:

$ zpool list nixstore
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
nixstore  1.81T   807G  1.02T        -         -    43%    43%  3.06x    ONLINE  -
$ zfs get compressratio nixstore/store
NAME            PROPERTY       VALUE  SOURCE
nixstore/store  compressratio  1.91x  -

close to 6x compress*dedup ratio

EDIT: I should also mention that dedup takes up a massive amount of RAM, so I wouldn’t do it without good justification

4 Likes
15

In Btrfs from time to time I force compression with this command:

sudo btrfs filesystem defragment -r -v -czstd /nix

And dedup:

sudo duperemove -drv /nix
1 Like
16

Wouldn’t Nix’s auto-optimise-store option make dedupe unnecessary?

17

let me check, it will take a while to run. But I think dedup runs on a per block instance, where as optimise just hardlinks files… And I’ve had cases where optimize reported negative savings before as well

2 Likes
18

True, but I wouldn’t expect block level dedup to occur very often unless the files the blocks are from are identical.

19

Yea, but when reviewing PR’s (including those targeting staging), it seems to be pretty common.

2 Likes
20

I catalogued my zfs usage here:

In short, despite autoOptimise on, I had a dedup of 1.35.

(I’ve turned off autoOptimise now)

I should also note that in my system of 8GB ram, nix-env --install irrespective of size of derivation are regular but nixos-rebuild takes my RAM usage from 30% of 98% and the system comes to a stand-still. I am not sure how to explain this …

21

FWIW (preallocate-contents option in nixUnstable), it should be available in nixos-unstable but not in 20.09 yet – someone would need to backport the update to stable for that.

Since the PR for auto-detection was brought up here I’ve updated it & it should be ready, let’s see how that goes, but it also won’t be available right away either. It’s not too complicated to update though, and nix seems well tested so I don’t think a PR updating nixUnstable would be refused :slight_smile:

1 Like
22

@asmadeus I’m using NixOS-unstable. I tried to nixUnstable but it’s not there yet.

23

Ah, my bad.
nixpkgs’s master branch’s f25bb567f608 ("nixUnstable: 3.0pre20200829_f156513 -> 3.0pre20201020_e0ca98c") updates nixUnstable to a sufficiently recent version, but it’s not in unstable yet. I somehow assumed nixos-unstable was directly equivalent to master but it’s not, sorry.

I’m not sure how the branches are updated, it looks linear to me. nixos-unstable-small has it but not nixos-unstable yet for some reason – then again the commit is only two days old so I assume it’ll get there sooner rather than later…

1 Like
24

the release channels have to finish all builds, and all “jobs” need to be successful for it to be updated.

Although nix{os,pkgs}-unstable follows master, it will always lag behind, sometimes for days.

Usually the long delays are because some aarch64 or darwin build is taking forever to complete, since those are architectures/platforms with less hardware dedicated to them.

EDIT: not to mention the aarch64 builder just has really slow single core performance

2 Likes
25

Hi @asmadeus! It seems that preallocate-contents is available now with nixUnstable, the version of nixUnstable is 3.0pre20201020_e0ca98c. I’m not seeing more that warning about preallocate-contents being a unknown setting:

❯ sudo nixos-rebuild switch                                                                                                                                                           
[sudo] senha para thiago: 
building Nix...
building the system configuration...
activating the configuration...
setting up /etc...
reloading user units for gdm...
reloading user units for thiago...
setting up tmpfiles
1 Like
26

Yep! It finally built a couple of days ago according to https://status.nixos.org :slight_smile:

Now just need preallocateContents option: disable by default by martinetd · Pull Request #4094 · NixOS/nix · GitHub reviewed & merged for automatic detection, but at least we can explicitly enable it for now

1 Like
27

The preallocate-contents option shouldn’t be necessary right? Since fallocate has been disabled by default, apparently.

1 Like