TLDR: a backdoored binary downloaded by any Hydra build can be used to backdoor every Hydra build, Ken Thompson-style. To do this: submit a “referesh the bootstrap-files” LGTM-merge PR and use the storepath-hash of the exploit as the hash argument of the import <nix/fetchurl.nix> call. Hydra won’t bother trying to download from tarballs.nixos.org.
A friend of mine mentioned that one of their friends had submitted #332462 so I took a look at it (after it had been merged). Unlike the past bootstrap-files update PRs submitted by me and @trofi, this one didn’t have copypasta that the reviewer could use to reproduce the refresh process.
So I took a closer look.
It turned out that the URL in the fetchurl invocation’s url was 404! How strange, I thought to myself. How is this possible? Thinking about how that could happen is how I discovered this exploit. Fortunately, in the case of #332462, the outpath which built the artifacts hadn’t yet been garbage-collected (but it will be) and I was able to download it and verify the hashes. So this PR does not carry an exploit.
Here’s how it could have carried an exploit: there is no automated check that the hash field of import <nix/fetchurl.nix> matches nix hash path /nix/store/xxx-stdenv-bootstrap-tools-${stdenv.hostPlatform.config}.drv. If they don’t match, nobody will notice. In that situation, all we would know is that Hydra built the (potentially-exploited) busybox binary by realising some FOD whose outputHash is sha256-R6nAiaIOg.... But we wouldn’t know that the FOD was a fetch from http://tarballs.nixos.org/ – or even that it was a fetch at all. It could have been built by any other FOD in any version of nixpkgs at any point in the past.
In other words, any random github user who can sneak an exploit binary into any build for any package buried in the nether-regions of some sub-sub-sub-ecosystem like buildCrystalPackage or harePackages can then smuggle that exploited binary into the root of the bootstrap sequence and compromise every package.
The fact that nobody else noticed that the tarballs.nixos.org URL was 404 proves that nobody is checking these hashes.
More details
How did we get here?
The current situation seems to be a side effect of the fact that we've automated the process for updating the bootstrap-files.
So the bootstrap files are basically a big huge weak point in our security story.
Maybe we shouldn’t make it easy to poke that weak point. But even we should, doing so is not the kind of PR that should be “LGTM merged”.
Possible Mitigiations:
Require copypasta in bootstrap-files update PRs
@trofi and I always made sure to include copypasta in our bootstrap-files update PRs that the reviewer could execute for themselves to validate the data being copied into the expression.
I think we need to require these shell commands in any bootstrap-files updates, and the person merging the PR needs to confirm that they executed those commands and checked the hashes.
When merging a bootstrap-files update, you must confirm that you ran the copypasta commands and checked the result, before merging the PR.
Restrict who can merge bootstrap-files updates
In the past, only @lovesegfault could upload to `tarballs.nixos.org`, so he was the only one who could take the final step which would allow CI to pass. Because of this, he was basically always the person who merged these PRs, because they were unmergeable until he took action.
It looks like this upload process may have become automated somehow. That’s good
I don’t think github allows that granularity. We would need to enforce that every codeowner needs to approve PRs they are owner for which doesn’t work with our current workflow.
This would be a good context to think about replacing widespread write access with a bot that does most actual merging again.
See Automatic Merging Implementation · Issue #112 · NixOS/ofborg · GitHub and especially Feature Request: Automatic merging · Issue #104 · NixOS/ofborg · GitHub
Direct write access for most users could be replaced by some kind of whitelist of users that can instruct the bot to merge IF all “must-succeed” checks (such as “no bootstrap file changes” or “only allow bootstrap file changes when at least 3 <specific team> members approved”) succeeded.
The defense works by wrapping the bootstrapFiles fetcher in invalidateFetcherByDrvHash. This requires that invalidateFetcherByDrvHash is moved from pkgs to lib, because we don’t have a pkgs-set so early in the stdenv bootstrap.
Needs to go to staging. Would appreciate comments from @trofi. Also needs approval from the lib folks for moving invalidateFetcherByDrvHash from pkgs to lib (which seems reasonable, since invalidateFetcherByDrvHash is totally standalone code that doesn’t use anything from nixpkgs).
Yeah. We had also discovered this a while ago and I hat proposed to have a mitigation plan as part of a sovereign tech fund proposal, which didn’t made it into the final list.
Here is the idea: Have a CI job on master trying to download every new FOD from it’s url and if the hash mismatches, throw an alert for manual inspection. Bootstrap files are not very special I would say. Any critical dependency can be affected and cause damage. An attacker just must control any package and submit a malicious version that contains both the package to be updated and a different maliciously patched package.
Agreed; there are tons of packages that are at least nearly as critical as the bootstrap files, like the kernel. Any solution should not be exclusive to bootstrap files, and just checking every FOD in CI or something seems like the way to go to me. There is the concern of FODs that are too resource intensive for CI, but I think those should just be flagged as failures, just like if it was a hash mismatch. That way someone knows to manually check it.
Well, they are the only binaries that are exempt from NIXPKGS_ALLOW_NONSOURCE=0.
They’re also the only binaries upstream of stdenv.
Sure, you could run this same attack on the gcc source tarball, but the exploit would need to be source code. This doesn’t make the exploit any easier to discover (it didn’t for xz) but once discovered it makes it impossible to deny that an exploit happened. For the bootstrap-files we don’t have that guarantee.
In any case I’m certainly in favor of applying this defense more broadly, but the bootstrap-files are where we really need it most.
I wrote a script, and ran it on a few packages (hello and libreoffice), and it found no difference between the fetched derivation of FODs and their hash (outside of GitHub returning some temporary 403 errors, solved by retry). I’ll add a backoff retry logic, and share it for those who are interested to test other packages. It’s a python script directly calling nix commands.
I’ve been working on getting the existing packages updated and cleaned up a bit, and then will be taking the changes from the linked PR into my branch(with proper authorship maintained, of course), ensuring they work properly, and then will push all of that work through. Once that’s done, moving nixpkgs’ base from the existing bootstrap-tools files to these new bootstrapped derivations should be as easy as a derivation that creates the appropriate tarball, then substituted into the stdenv in place of the static tarball.
(note that while i say “easy as”, this is in reality still fairly involved and there’s a lot of work to do. However, this is something I expect could be done as soon as the 26.05 release(I am not making any promises, just guesses based on the speed of progress))
Note also that I am a college student doing this in her free time, and have other responsibilities in nixpkgs(25.11 release editor, package maintainence), and also do not want to work on a single project for months at a time, so I will be taking frequent breaks. I am trying to keep somewhat steady progress updates posted to the #stdenv:nixos.org matrix channel(Nixpkgs Stdenv) in the NixOS space, feel free to ping me there if you have any questions, comments, concerns, or would like to help!
Hopefully final edit: I 100% agree with the security concerns of trusting a bundle of static binaries, which is why I’m doing this! I do feel that this work has practical security benefits for nixpkgs users, and doing it gives us a more stable and trusted base to build off of. As a future cybersecurity major, doing stuff like this is why I got into the field, and it’s something I’m super passionate about, and even gave a very short talk at my university to a small group about why this work is important but also how difficult it can be.
@pyrox saw your PR, thanks advancing the progress on minimal bootstrap. Feel free to ping me if you need me to do nixpkgs-review on packages made with this bootstrap, I have a beefy build machine lying around.
I’ve been doing all this bootstrapping on my laptop!(It’s a modern laptop, but still, doesn’t need a build farm thankfully) Since it’s not the stdenv root at the moment, it’s fairly light and easy to build(you can build the whole thing in probably around 20-30 minutes by checking out the pr and running nix build -f . pkgsi686Linux.minimal-bootstrap.test!)
Edit 2: You will need to run nix build -f . make-minimal-bootstrap-sources first, then the above command. Weird bootstrapping thing that’s unavoidable. But those two commands build up the entirety of the package set from the PR!
Just the other day I was updating some pins in my configuration, but I accidentally swapped two hashes: Nix happily built my system with <nixpkgs> pointing to <nixos-mailserver> consequently breaking every nix command.