How does nix choose a substituter when there is more than one choice?

Help
1

I have set up an intra-company nix cache on an internal CI server.

It provides dependencies much faster than cache.nixos.org as it is local.

My local nix.conf specifies both the internal CI server as well as cache.nixos.org as substituters.

I’ve noticed that the internal CI server is picked for paths that exist on BOTH caches.
I wondered how does it pick a CI server so I tried changing the order of the CI servers in nix.conf but it kept picking the internal CI server.

Does Nix do some latency/speed check to choose a substituter?

Thanks!

2 Likes
2

After digging through the source, it appears that binary caches are ordered by a priority field, defined in the nix-cache-info file. https://cache.nixos.org/nix-cache-info lists that priority as 40. I’m guessing your internal cache must be defining a lower priority.

4 Likes
3

It is indeed as you said. The CI server cache’s priority is set to 30.
It must be something’s default value as I’ve never set it myself (I’m using nix-serve)

Thanks for your help!

4

Looks like nix-serve hardcodes a priority of 30.

1 Like
5

It would be nice to have an easily-deployable caching proxy for that kind of scenarios. A lot of companies could benefit from the lower latency and it would save some bandwidth on the public cache.

3 Likes
6

I can confirm that it would indeed be quite valuable in a company setting :slight_smile: For now we just make sure our cache has high-enough priority.

Note that if you do so, then your cache needs to be reachable from everywhere and at all time.

7

For the Chaos Communication Congress i hacked together a module which fulfulls this purpose. Essentially it is just a piece of nginx config, setting the internal hostname as proxy for deployment will cache upstream packages: https://git.shackspace.de/rz/stockholm/blob/master/krebs/3modules/cachecache.nix#L104-167

in addition to that it seems i have configured apt-cacher-ng as well, 4 years ago: Sign in · GitLab

Cheers

2 Likes
8

As mentioned on documentation: mention substituter precedence by asymmetric · Pull Request #3685 · NixOS/nix · GitHub and Allow overriding substituter priorities · Issue #3264 · NixOS/nix · GitHub at the end of 2019 a seemingly undocumented feature was added to substituter URIs allowing one to set the priority by appending a “?priority=…” parameter.

1 Like
9

I created a follow-up PR at doc: mention Priority for substituters by asymmetric · Pull Request #4796 · NixOS/nix · GitHub.

1 Like