Summary of the blog post: Nix caches are a quite significant attack vector (leading to potential RCE and privilege escalation), and a common cache set up (CI-pushes-to-cache) makes this much worse, as a lot of people end up having cache push access.
Review all the
substituers,extra-substituters,trusted-substituters, andtrusted-public-keysin:
Also add in trusted-users, as that’s effectively root on a nix-managed system due to the same vectors mentioned in your article.
Can’t overstate how much this title resonates. I outright hated myself writing substitutor instructions for the old pre-nix-community cuda-maintainers binary cache, knowing I have root access to one builder, and Connor to the other two. Now we’re setting up infra for the GPU tests and, “temporarily”, we enable the Flox substitutor on the machine running Hydra, instead of in an ephemeral builder with its own isolated nix-store.
Derivations in ephemeral microvms are not a “nice to have”. Trustix is not a “nice to have”. Key rotations are not a “nice to have”. Signing content-addressed inputs is not a “nice to have”. Our present state is miserable and embarrassing, and fixing it is imperative.
Yeah, I kept wanting to mention key rotations and chain of trust in the blog post, because I agree that they’re also urgently needed, but it felt like it was a somewhat different topic.
I did not understand how the second approach makes it more secure. If someone gets your key they can also sign malicious payloads. Does the added security come from every signed package has also to be built instead of any package in a cache is implicitly trusted because the cache as a whole is trusted? If so I fail to see how the added step makes it less vulnerable when in any scenario you either compromise the key or get the cache to sign a malicious payload.
I might miss something very obvious so I am happy to be corrected.
It’s essentially about how many people/entities have access to the signing and upload keys. In the current scenario (GitHub Actions CI pushes to cache, for example), every developer of any project that pushes to a cache X has access to the key. And you add multiple caches, so then it’s the union of all the various developers of all the projects that push to any cache you use. That’s a lot of people. It’s not even possible to know who all those people are, because it’s not public information (in the GitHub case). In addition to this, the CI service itself has the key (i.e., if GitHub is compromised, they can get access to the key).
In the second scenario, only the CI service itself has the keys. All those other developers don’t. They can’t write another workflow that uses the key in some different way (e.g., cat-ing it to the logs). They can only submit derivations to be built, and then the CI service when done signs it. The interface to the CI service is specifically Nix then (e.g., submitting either derivations or expressions to be built).
Does that make sense?
Ok I think I get the nuance there now.
Added this to the Wiki Binary Cache - NixOS Wiki because I think it’s an important and valid point. Feel free to correct the wording etc.
Great idea - thanks!
As a nix-community member maintaining nix4nvchad.
I wonder how could I directly push cache to nix-community.cachix.org
According to what blog said.
You’re right, it was wrong of me to leave that implication in there. The organization membership does not imply write-access to all repos in an org, nor access to all secrets. It isn’t public information who has write access to a repo, and what secrets are accessible from that repo. Implying that a specific set of people do wasn’t great on my part.
I’ve removed that parenthetical statement from the blog post. Thanks again for pointing this out.
This was both a welcome and necessary post. Thank you!
Another idea: it should be possible to restrict a binary cache in which derivations can be fetched from them. For example, if I were to use a cache for Hyprland, I would only ever want to substitute Hyprland-related packages from there. When I try to build openssh, it shouldn’t even ask that cache.
Of course, how this would be done at a more granular level than per flake is not clear. But such a mechanism could reduce the impact of any particular cache being poisoned.
nix.conf - Nix 2.31.3 Reference Manual and S3 Binary Cache Store - Nix 2.31.3 Reference Manual indicate that you can set priorities of substituters, which should reduce the likelihood of such an attack. Assuming the official cache (or whatever you set with lower priority) has cached it.
TLDR: Is fundamentally hard.
On one hand, IA Nix Store is not sufficiently multi-tenant (CA Derivations were meant to partially address this). On the other, Nix builds are also too “anonymous”: you cannot even tell whether a particular derivation being realized came up as a dependency for Hyprland or for SSH.
This kind of compartmentalization requirement is maybe the line after which it finally makes sense to make a step past Nix and start inventing “containers” (well, microvms), i.e. to have multiple isolated Nix stores. All you gotta do is make it cheap (like, much cheaper than it is now) to compose them:
TLDR: Is fundamentally hard.
I agree, though maybe not impossible? One idea that comes to mind is that package authors could prefix their package names by a unique identifier of the cache, and nix would only substitute from caches with the right prefix (with presumably some exception for special caches like cache.nixos.org). “Lying” about your prefix is of course possible, but I can’t see any danger in that: you’re saying that a package you control should be fetched from a different cache, rather than a package that you don’t control should be fetched from a cache you do.
Sure, this would mean that any downstream dependency that happens to be built in that cache gets ignored, so there might be a bit more building, but that’s exactly what we want.
(Haven’t thought deeply about this, so there may be some pretty fundamental problems with the idea.)
Ah, I see, you’re basically considering a special case where we’re allowed to “taint” a derivation, giving it more “identity” in a sense. It’s as if you replaced allowSubstitutes :: bool with allowSubstituters :: List Uri or :: List ProjectName. Fwiw this maybe even makes sense for out-of-tree projects?
Exactly - and much better explained 
One reason I thought of the package name rather than some other attribute is that I suspect you could then implement it without even changing Nix (e.g. with a local proxy for it’s requests that 404s in the right cases) though maybe it’s better to do it in a non-hacky way.
edit: never mind, not needing to change Nix is no advantage, as we’d really really want nixpkgs packages to limit their substituters, so starting trying it out with upstream changes is somewhat useless.
Does anyone know whether any of nix teams (lix,snix,tvix,nix) is aware/working on this issue or has ever posted on this? Because this seems like something that warrants to be on some sort of long-term roadmap. Not necessarily the specific solutions described here, but the attack vector in general.
My paper is cited in the blog post, and I’m working on an implementation of it, which addresses these concerns by making it possible to change who you trust over time:
@Ericson2314 is working on some of the prerequisites in terms of scheduling and ca-derivations inside the Nix implementation. The idea is to contribute the finished signature format as as part of the effort to stabilize ca-derivations. This was announced as part of the supply chain security panel at NixCon. I’m also very much talking people from other implementations.
How this helps is by enabling users to effectively change who they trust over time. It’s not a straightforward fix to the issue, because recovering from from the effects of having code you no longer trust requires more locking down, but it puts us in a much better position.