I recently played around with @jkarni’s https://garnix.io/ service, and was surprised to find that it shares a single nix store and cache among all users. Given the effort that @domenkozar’s https://www.cachix.org/ and @rickynils’ https://nixbuild.net/ put to keep their users’s caches separate, this got me thinking.
When are separate caches needed
I see two main reasons for keeping caches separate:
-
The uploader doesn’t trust all downloaders.
For example, you use nix to provision your servers, and if the nix store and/or cache were public, and and someone learns the nix path (e.g. from a error message), they can download the code and look for secrets or vulnerabilities.
-
The downloader doesn’t trust all uploaders.
By configuring a nix cache as a substituter on my machine, I am essentially fully trusting those who can upload to that cache. If they were malicious, they could upload a malicious store path and maybe for my next system upgrade, I’ll fetch their bad package.
Many caches are bad
On the other hand, many caches have downsides. My nix.conf currently says
substituters = https://cache.nixos.org/ https://nixcache.reflex-frp.org https://cache.iog.io https://digitallyinduced.cachix.org https://ghc-nix.cachix.org https://ic-hs-test.cachix.org https://kaleidogen.cachix.org https://static-haskell-nix.cachix.org https://tttool.cachix.org https://cache.nixos.org/
and that slows down everything, as nix queries all these caches, only to notice that the devShell I am about to enter is on none of them¹.
All of these nix caches are open source projects where there reason 1 (secrecy) obviously doesn’t apply. If we can also somehow address reason 2, then maybe such projects could use a single cache.
A single public common cache?
The benefits of such a “public common nix cache” would be:
- There could be many more projects (outside nixpkgs) where a user gets a cache hit when they run
nix run github:someones/repo - That there might be occasional synergies between projects (e.g. when they both happen to use the same non-nixpkgs-built GHC toolchain).
- Fewer caches to query for the user, so more responsive nix tools.
- Fewer random people with cache upload persmissions that one has to trust to not turn malicious, or lose the credentials to someone who is!
(Almost) all the things we love about cache.nixos.org, but extended to more projects!
… via a trusted build
So how could we address reason 2 (at least before Trustix takes off)?
By using one (or a small number of) trusted build provider, like nixbuild.net or garnix.nix. I could imagine nixbuild.net to provide the a single public cache (like garnix does), or team up with cachix to let them do the hosting. Only stores paths built by the trusted build provider are uploaded there; I can’t upload locally built paths; this ensures that the store paths there are as trustworthy as the build provider.
When building something on nixbuild.net² I can indicate that this is a non-secret build and reason number 1 does not apply. Then the build results can be pushed there. Also, nixbuild.net could always safely use that cache.
I could configure all my public projects to do that, and would no longer have to worry about providing caches, telling users to configure them, or sharing build artifacts between them.
A widely used trusted build infrastructure also has other benefits due to scale. For example, it’s more likely that a widely used build farm supports building on strange systems/architecture than a random small software project.
Storage cost and cache eviction
One challenge that would need solving is cost and cache eviction. A open-for-all cache can probably not grow forever, like cache.nixos.org does. But a simple LRU cache eviction like Cachix currently provides is also insufficient, as busy projects will displace less busy ones (a problem I am already having when sharing a Cachix cache between related projects).
It seems that the cache provider would have to allow users to register roots that ought to be alive, and then charge for the storage cost of stuff kept alive by that root. Maybe discounted if some paths are kept alive by multiple users. It seems @domenkozar is working on support for GC roots in Cachix, so maybe not so unrealistic.
An even bolder thought?
Thinking a bit more boldly: I wonder if there could even be a way to offer that feature under the cache.nixos.org cache, so that everyone benefits without extra work? A way for the public to submit builds to trustworthy build network that’s that feeds that cache and a way to register roots (with a way to pay for that, I guess)?
¹ I could probably mitigate that by using the flake feature to configure some of these caches only for the projects where they might be relevant. Still, it’s complexity I wish I would not have to think about. I’m also thinking of providing a cache multiplexer service that would in parallel query a set of caches, and then redirect the user to the right one – then they have to still configure all the keys, but only one cache endpoint.
² Typically via CI, but possibly also from a local machine using remote builders, which which is great! A CI-only solution is unnecessary restrictive, I’d say.