Proposal: remove ability for substituters to set their priority

[asymmetric]

Currently, the nix tool decides the order in which substituters are queried by using a Priority value, which substituters can set themselves:

❯ curl -L cache.nixos.org/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 40

❯ curl -L nix-community.cachix.org/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 41

This strikes me as a big security concern, and is the reason I don’t use community caches as much as I’d like - they/Cachix could change their Priority and overtake cache.nixos.org.

Is there a good reason to keep this behaviour around?

[thufschmitt]

I don’t think this could go away, if only because of backwards compatibility, but you can override the priority locally using for example https://cache.nixos.org?priority=10

[blaggacao]

@thufschmitt

if only because of backwards compatibility

Maybe, trading against such a high good as “default security”, this is an opportunity to rebalance back-compat with some sort of deprecation policy for nix.

[edolstra]

I don’t see the security concern. If you don’t trust a binary cache you shouldn’t use it. Security cannot depend on binary cache priority because there is always a possibility that a higher-priority cache doesn’t have a substitute for some path or is unavailable.

[tomberek]

The missing capability (though more complex) seems to be that people may want to specify caches for one set of derivations and another list of caches for another set of derivations. One can imagine some scenarios like: “I want all of my services and suid programs to be sourced from cache B. But in interest for speed+bandwidth, user programs with less privilege can use A+B where A is less trusted.” Though it seems a far-fetched.

Barring some complex feature, the current setup seems to have the fewest edge case and problems. Trust in a binary cache is very much a binary proposition and most cases where one would really need to avoid a rogue-priority-stealing-cache can be solved by managing your own cache.

For discussion purposes, the worst-possible case seems to be:
Security of a cache is compromised and someone maliciously increases the priority AND replaces binary artifacts. With the new proposal, this would only impact artifacts NOT present in cache.nixos.org. Is it that the distinction and preservation of “core” artifacts the goal?

[blaggacao]

@asymmetric

Is there a good reason to keep this behaviour around?

Granted the conclusion about security relevant choices, I guess this OP’s question was somehow forgotten.

To me it also seems odd that caches can — at first glance * — provoke side effect amongst / against each other upon the user.

Was this meant as a somewhat server side archaic load balancing mechanism of some sort (taking literally chances at numbers)?

* Granted, if trustworthy, the things served under the same hash should be equivalent.

[xfix]

This is by design, you need to trust a cache to use it, and even if cache wouldn’t be able to set its own priority, it still would be able to substitute paths that aren’t in the main cache (for instance, systemd’s .service files).

@tomberek

The missing capability (though more complex) seems to be that people may want to specify caches for one set of derivations and another list of caches for another set of derivations. One can imagine some scenarios like: “I want all of my services and suid programs to be sourced from cache B. But in interest for speed+bandwidth, user programs with less privilege can use A+B where A is less trusted.” Though it seems a far-fetched.

I think content-addressed Nix may allow for different builds to use different caches, but currently it’s an experimental feature, and this part wasn’t implemented.

[nrdxp]

@asymmetric

Is there a good reason to keep this behaviour around?

In terms of the security concern, I believe the proper fix would be ca derivations along with Trustix when it’s ready.

Once that’s done, I would prefer if Nix actually raced to search caches in parallel with the fastest response being the winner, since if you have a lot of caches defined, sometimes querying the caches can take longer than building simple derivations, especially if one of them is slow to respond or experiencing issues.

[blaggacao]

@nrdxp

I would prefer if Nix actually raced to search caches in parallel with the fastest response being the winner, since if you have a lot of caches defined, sometimes querying the caches can take longer than building simple derivations, especially if one of them is slow to respond or experiencing issues.

I can absolutely relate with this observation. It suggests that we should get eventually rid of priority and favor parallelization.

I don’t quite get, why CA is a prerequisite, but that’s probably just me

I think this conclusion is worthy of being recorded into a proper upstream Issue.

[nrdxp]

CA is a requirement for Trustix. It addresses the underlying securty concern (mentioned by OP) since each cache would have to contain a trusted derivation anyway so a race to find the fatest response couldn’t change the result.

Of course you could do this without CA, I was trying to imagine a solution to the original question.