My recent deep dives into libgit2 have been pretty fruitful when it comes to finding new and unexpected bugs peculiarities of the library and the way nix holds it. So here goes:
Nix stores unpacked tarballs fetched via builtins.fetchTarball and tarball-style flake inputs: so all github:/gitlab: and nixpkgs channel tarballs (like https://channels.nixos.org/nixos-unstable/nixexprs.tar.zst) in a bare git repository in $XDG_CACHE_HOME/nix/tarball-cache-v2. Unpacking each tarball now yields at most 10 thin – meaning they don’t contain their whole git object reference closure – pack files containing blobs and trees not yet present in the cache at the time of unpacking. libgit2 doesn’t work great with large numbers of pack files because it caches file descriptors for pack/index files without an upper bound by default, so it can lead to file descriptor exhaustion (something that has been worked around in nix by bumping FD ulimit, but that’s just a symptom of the problem).
In particular, this means that you can greatly benefit from periodically repacking it and providing a multi-pack-file index for more locality for file lookups. Repacking can also apply delta compression to similar files and reduce storage requirements - something that’s not really common to encounter in the same pack file since those come from the same tarball, but is commonplace for similar nixpkgs revisions.
Up until the time that nix implements the repacking itself, you can do the repacking with a systemd user service + timer like below (amend to your tastes and frequency) if you are using home-manager.
systemd.user.timers."repack-nix-tarball-cache" = {
Install = {
WantedBy = [ "timers.target" ];
};
Timer = {
OnCalendar = "daily";
Unit = "repack-nix-tarball-cache.service";
};
};
systemd.user.services."repack-nix-tarball-cache" = {
Unit.Description = "Repack nix's tarball cache";
Service = {
Type = "oneshot";
ExecStart = pkgs.writeShellScript "repack-nix-tarball-cache" ''
set -eu
cd ${config.xdg.cacheHome}/nix/tarball-cache-v2
${pkgs.git}/bin/git multi-pack-index write
${pkgs.git}/bin/git multi-pack-index repack --batch-size 1024m
${pkgs.git}/bin/git multi-pack-index expire
'';
};
};
This service should be safe to run concurrently with any tarball-cache operations (i.e. nix commands).
Another great way to make tarball unpacking snappier is to disable delta compression on unpacking, which is the most expensive part of fetching nixpkgs now and doesn’t provide much deduplication at all for the reasons outlined above. As of libgit2 1.9.7 config value used for this is very unintuitive (and is probably just a bug, see pack: respect pack.window and core.bigFileThreshold by xokdvium · Pull Request #7367 · libgit2/libgit2 · GitHub)
git -C $XDG_CACHE_HOME/nix/tarball-cache-v2 config pack.deltaCacheSize 1
Ideally all these improvements would be merged to nix without requiring such local workarounds, but I figured I might as well share these tips here while my patches to libgit2 and nix are cooking. You can follow the progress on those in Draft: libgit2 and tarball unpacking performance improvements by xokdvium · Pull Request #16427 · NixOS/nix · GitHub and prerequisite libgit2 fixes.
Note that all this doesn’t apply to Lix which continues to unpack tarballs to the store as older CppNix versions used to do.