[nix perf] nix store symlink vs hardlink

Development
1

Are the any good resources to read to understand the decision to use in /nix/store symlinks instead of hardlinks?

pro:

  • easy debugging (how many % prod systems are used develop/debug nixos?)

negative:

  • system resources bloat:
    • storage on disk (/nix/store), number of inodes
    • ram resources, metadata cache
    • added latency for every single resource os call

The nixos developer vs production situation would be solvable via having a dev and prod mode for /nix/store. Specially with resource constrained (embedded) systems and with the ram & flash price crisis could closing the performance gap between native linux and nixos an option?

Even as nix maintainer, I never hat the need to debug / follow a symlink.

Here some stats from my system, normal nixos client:

go run paepcke.de/fsdd/cmd/fsdd@latest /nix/store
FSDD [start] [/nix/store] [hash:MAPHASH]
FSDD [_done] [time: 5m32.985897549s]
FSDD [stats] [files:4372236] [inode(s): 6531941] [sym.valid: 5670766] [sym.invalid: 14656] [data blocks: 227.1 Gbytes]
FSDD [_info] [deduplication savings] [inode(s): 3525717] [data blocks: 131.5 Gbytes]
FSDD [_info] [possible new deduplication savings] [inode(s): 5670781] [symlink(s): 5670766] [data blocks: 175.1 kbytes]

findings:

  • around 5.6 mio (saveable) inodes via symlinking
  • 14656 lingering broken symlinks (and no nix native store tool to identify or fix store)
2

I can only guess but:

  • one cannot hardlink directories, and many store symlinks are actually directories (notably with buildEnv)
  • iirc hardlinks are a problem with reproducible builds in some archive formats (cpio I think)
  • 5.6 M to spare is really tiny. Nix is really heavy on disk for other reasons and if you are watching megabytes of inefficiencies you should probably not use nix.
2 Likes
3

Well, you can enable automatic dedup using hard links in the nix store: NixOS Search

But always using hard links in place of symlinks not flexible enough.

2 Likes
4

That’s what nix-collect-garbage is for, or nix-store --repair if it really is store corruption.

Lingering broken symlinks that aren’t cleaned up by those commands are part of packages and exist for other reasons, presumably packaging mistakes - you wouldn’t want to modify package contents to make them inconsistent with upstream, that’s the exact opposite of reproducibility.

2 Likes
5

Hardlinks are not themselves filesystem objects. It is by definition impossible to tell a hardlink apart from its “original” file.

Which means that switching over to “using hardlinks” would have to entirely change how Nix store paths are handled including binary caches. Nix would have to guess what hardlinks are being produced upon derivation build completion, and serialize them somehow. And you would need a way to grab the actual contents of the hardlinked files so they could be populated in /nix/store as the paths are unpacked.

Having dev/prod modes for using symlinks and hardlinks is not acceptable. While hardlinks are indistinguishable from hardlink destinations (you can tell that a file has more than the expected number of links, but not which one is the “source” and which is the “destination”), symlinks and hardlinks (i.e. on the surface, usually regular files) differ in behavior and have caused problems in the past e.g. https://codereview.qt-project.org/c/qt/qtbase/+/597856

3 Likes
6

Do you have more concrete data on this? This is genuinely the first time I’ve heard of a complaint of symlinks causing enough of a gap that would warrant differentating between “native linux” and “nixos”.

I know there are some cases especially with path lists (e.g. KDE applications are slow in Plasma · Issue #363068 · NixOS/nixpkgs · GitHub) where “the nix way” slows down program launch, but these are not caused by symlinks, and in fact genuinely could be solved by adding more symlinks.

2 Likes
7

The word “native” has truly been butchered into utter meaninglessness.

2 Likes