Consider building derivation A on remote builder, maybe as part of your CI, and A’s result will mention, and thus depend, on B, which is already present in your substituter’s store path. Clearly, nix will have to download B on the builder before building A.
But often, that is silly. If A is a nixos configuration, and B is the apache binary, then very likely the builder can assemble the nixos configuration without having the content of
/nix/store/B – it just needs to know the name of that path, and that’s already known from the derivations. And even if the builds needs some files of the the dependency, it rarely needs all of them.
So it seems silly to have to copy all of
B to the builder first. I see that kind of unnecessary and time-consuming transfers a lot in the logs of, say, builds on garnix.io.
I am wondering about two variants to improve this:
/nix/store. Most network file systems (NFS, Glusterfs, etc.) are quite good at only loading store contents on demand.
So in the case of a central nix store server and multiple builders around it, the builders could mount the full
/nix/storeof the central store, read-only, have an overlay FS on top of it for local modifications. This way, the complete store seems to be always available without extra work, and the build can start right away. If the build needs file from the dependencies, they are loaded individually and on demand (and then, if the file system is good, cached sensibly).
Of course care needs to be taken that the underlying store doesn’t change under the overlay fs. Rough idea: User btrfs or zfs on the server and only export immutable snapshots to the builders.
A custom lazy-substitutor-fs, overlaying
This could be a FUSE-implemented filesystem, with a bit of integration with
nixwould otherwise realize a nix store from a substitutor, it would just tell the filesystem that this patch is known to exist, and the filesystem would pretend it’s there, and only when some process actually accesses something within, it would download the NAR as usally, unpack into the underlying real
/nix/store, and everything proceeds normally from this point on.
This wouldn’t support partial store paths, but the advantage is that it would work with ordinary nix caches. I am sure your nix Github Action building your nixos configuration downloads more from cache.nixos.org than it has to
Did anyone try something along these lines already?