Bootstrap-files updates amplifiy exploit of *any* package into exploit of *every* package

Development Security
1

TLDR: a backdoored binary downloaded by any Hydra build can be used to backdoor every Hydra build, Ken Thompson-style. To do this: submit a “referesh the bootstrap-files” LGTM-merge PR and use the storepath-hash of the exploit as the hash argument of the import <nix/fetchurl.nix> call. Hydra won’t bother trying to download from tarballs.nixos.org.

A friend of mine mentioned that one of their friends had submitted #332462 so I took a look at it (after it had been merged). Unlike the past bootstrap-files update PRs submitted by me and @trofi, this one didn’t have copypasta that the reviewer could use to reproduce the refresh process.

So I took a closer look.

It turned out that the URL in the fetchurl invocation’s url was 404! How strange, I thought to myself. How is this possible? Thinking about how that could happen is how I discovered this exploit. Fortunately, in the case of #332462, the outpath which built the artifacts hadn’t yet been garbage-collected (but it will be) and I was able to download it and verify the hashes. So this PR does not carry an exploit.

Here’s how it could have carried an exploit: there is no automated check that the hash field of import <nix/fetchurl.nix> matches nix hash path /nix/store/xxx-stdenv-bootstrap-tools-${stdenv.hostPlatform.config}.drv. If they don’t match, nobody will notice. In that situation, all we would know is that Hydra built the (potentially-exploited) busybox binary by realising some FOD whose outputHash is sha256-R6nAiaIOg.... But we wouldn’t know that the FOD was a fetch from http://tarballs.nixos.org/ – or even that it was a fetch at all. It could have been built by any other FOD in any version of nixpkgs at any point in the past.

In other words, any random github user who can sneak an exploit binary into any build for any package buried in the nether-regions of some sub-sub-sub-ecosystem like buildCrystalPackage or harePackages can then smuggle that exploited binary into the root of the bootstrap sequence and compromise every package.

The fact that nobody else noticed that the tarballs.nixos.org URL was 404 proves that nobody is checking these hashes.

More details

How did we get here? The current situation seems to be a side effect of the fact that we've automated the process for updating the bootstrap-files.

I don’t know if frequent updates to the bootstrap-files was a good idea… we don’t have a full source bootstrap like Guix does, and never will since the nix interpreter isn’t written in nix.

So the bootstrap files are basically a big huge weak point in our security story.

Maybe we shouldn’t make it easy to poke that weak point. But even we should, doing so is not the kind of PR that should be “LGTM merged”.

Possible Mitigiations:

Require copypasta in bootstrap-files update PRs @trofi and I always made sure to include copypasta in our bootstrap-files update PRs that the reviewer could execute for themselves to validate the data being copied into the expression.

I think we need to require these shell commands in any bootstrap-files updates, and the person merging the PR needs to confirm that they executed those commands and checked the hashes.

When merging a bootstrap-files update, you must confirm that you ran the copypasta commands and checked the result, before merging the PR.

Restrict who can merge bootstrap-files updates In the past, only @lovesegfault could upload to `tarballs.nixos.org`, so he was the only one who could take the final step which would allow CI to pass. Because of this, he was basically always the person who merged these PRs, because they were unmergeable until he took action.

It looks like this upload process may have become automated somehow. That’s good

18 Likes
2

Great find. This is scary.

I suggest we set up a branch protection rule in combination with CODEOWNERS that only allows specific team to merge Bootstrap files.

If GitHub isn’t rich enough to express this constraints we might need to consider making merge bot mandatory and encode it on there

3 Likes
3

Would it be possible to verify bootstrap-files integrity using CI (i.e. ofBorg)?

A red CI would be a red flag (literally) to any committer.

4

I don’t think github allows that granularity. We would need to enforce that every codeowner needs to approve PRs they are owner for which doesn’t work with our current workflow.

5

There are issues for this already but ofborg development seems to be really dried out, so not sure if anything in that direction is going to happen.

Make GitHub suggestions for fixing sha256 fields · Issue #429 · NixOS/ofborg · GitHub Detect url/sha256 mismatch · Issue #647 · NixOS/ofborg · GitHub

6

This would be a good context to think about replacing widespread write access with a bot that does most actual merging again.
See Automatic Merging Implementation · Issue #112 · NixOS/ofborg · GitHub and especially Feature Request: Automatic merging · Issue #104 · NixOS/ofborg · GitHub
Direct write access for most users could be replaced by some kind of whitelist of users that can instruct the bot to merge IF all “must-succeed” checks (such as “no bootstrap file changes” or “only allow bootstrap file changes when at least 3 <specific team> members approved”) succeeded.

7

I believe these two commits prevent the attack:

Currently building with this.

The defense works by wrapping the bootstrapFiles fetcher in invalidateFetcherByDrvHash. This requires that invalidateFetcherByDrvHash is moved from pkgs to lib, because we don’t have a pkgs-set so early in the stdenv bootstrap.

Needs to go to staging. Would appreciate comments from @trofi. Also needs approval from the lib folks for moving invalidateFetcherByDrvHash from pkgs to lib (which seems reasonable, since invalidateFetcherByDrvHash is totally standalone code that doesn’t use anything from nixpkgs).

6 Likes
8

I didn’t know about invalidateFetcherByDrvHash before, very interesting! I guess the docs should probably move too if the function does.

9

Yeah. We had also discovered this a while ago and I hat proposed to have a mitigation plan as part of a sovereign tech fund proposal, which didn’t made it into the final list.
Here is the idea: Have a CI job on master trying to download every new FOD from it’s url and if the hash mismatches, throw an alert for manual inspection. Bootstrap files are not very special I would say. Any critical dependency can be affected and cause damage. An attacker just must control any package and submit a malicious version that contains both the package to be updated and a different maliciously patched package.

5 Likes
10

Agreed; there are tons of packages that are at least nearly as critical as the bootstrap files, like the kernel. Any solution should not be exclusive to bootstrap files, and just checking every FOD in CI or something seems like the way to go to me. There is the concern of FODs that are too resource intensive for CI, but I think those should just be flagged as failures, just like if it was a hash mismatch. That way someone knows to manually check it.

1 Like
11

Lib folk here, sounds good to me! I’ll also get auto-pinged as a code owner when a PR is opened

12

Well, they are the only binaries that are exempt from NIXPKGS_ALLOW_NONSOURCE=0.

They’re also the only binaries upstream of stdenv.

Sure, you could run this same attack on the gcc source tarball, but the exploit would need to be source code. This doesn’t make the exploit any easier to discover (it didn’t for xz) but once discovered it makes it impossible to deny that an exploit happened. For the bootstrap-files we don’t have that guarantee.

In any case I’m certainly in favor of applying this defense more broadly, but the bootstrap-files are where we really need it most.

13

I wrote a script, and ran it on a few packages (hello and libreoffice), and it found no difference between the fetched derivation of FODs and their hash (outside of GitHub returning some temporary 403 errors, solved by retry). I’ll add a backoff retry logic, and share it for those who are interested to test other packages. It’s a python script directly calling nix commands.