# Bootstrap-files updates amplifiy exploit of \*any\* package into exploit of \*every\* package

**URL:** https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534
**Category:** Security
**Created:** [August 13, 2024, 5:51am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534 "2024-08-13T05:51:57Z")
**Posts on this page:** 19
**Page:** 1

<div class="post-metadata">

### Author: ![amjoseph](https://discourse.nixos.org/user_avatar/discourse.nixos.org/amjoseph/32/28378_2.png) [@amjoseph](https://discourse.nixos.org/u/amjoseph)
#### Post date: [August 13, 2024, 5:51am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/1 "2024-08-13T05:51:57Z")

</div>

TLDR: a backdoored binary downloaded by **any** Hydra build can be used to backdoor **every** Hydra build, [Ken Thompson](https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf)-style. To do this: submit a “referesh the bootstrap-files” [LGTM-merge](https://github.com/NixOS/nixpkgs/pull/332462#pullrequestreview-2219014682) PR and use the storepath-hash of the exploit as the `hash` argument of the `import <nix/fetchurl.nix>` call. Hydra won’t bother trying to download from `tarballs.nixos.org`.

* * *

A friend of mine mentioned that one of their friends had submitted [#332462](https://github.com/NixOS/nixpkgs/pull/332462) so I took a look at it (after it had been merged). Unlike the past bootstrap-files update PRs submitted by [me](https://github.com/NixOS/nixpkgs/pull/183487) and [@trofi](https://github.com/NixOS/nixpkgs/pull/288866), this one didn’t have copypasta that the reviewer could use to reproduce the refresh process.

So I took a closer look.

It turned out that the URL in the `fetchurl` invocation’s [url](http://tarballs.nixos.org/stdenv/s390x-unknown-linux-gnu/8ba481d65eb21a4f9e6b1e812de3f83079eb8016/busybox) was 404! How strange, I thought to myself. How is this possible? Thinking about how that could happen is how I discovered this exploit. Fortunately, in the case of #332462, the outpath which built the artifacts hadn’t yet been garbage-collected (but it will be) and I was able to download it and verify the hashes. So this PR does not carry an exploit.

Here’s how it could have carried an exploit: there is no automated check that the `hash` field of `import <nix/fetchurl.nix>` matches `nix hash path /nix/store/xxx-stdenv-bootstrap-tools-${stdenv.hostPlatform.config}.drv`. If they don’t match, nobody will notice. In that situation, all we would know is that Hydra built the (potentially-exploited) `busybox` binary by realising _some_ FOD whose outputHash is `sha256-R6nAiaIOg...`. **But we wouldn’t know that the FOD was a fetch from `http://tarballs.nixos.org/`** – or even that it was a fetch at all. It could have been built by any other FOD in any version of nixpkgs at any point in the past.

In other words, any random github user who can sneak an exploit binary into **any** build for any package buried in the nether-regions of some sub-sub-sub-ecosystem like `buildCrystalPackage` or `harePackages` can then smuggle that exploited binary into the root of the bootstrap sequence and compromise _every_ package.

The fact that nobody else noticed that the `tarballs.nixos.org` URL was 404 proves that nobody is checking these hashes.

#### More details

> **How did we get here?**
>
> The current situation seems to be a side effect of the fact that we've automated the process for updating the bootstrap-files.
> 
> I don’t know if [frequent updates to the bootstrap-files](https://github.com/NixOS/nixpkgs/pull/284541) was a good idea… we don’t have a [full source bootstrap](https://guix.gnu.org/manual/en/manual/devel/en/html_node/Full_002dSource-Bootstrap.html) like Guix does, and never will since the nix interpreter isn’t written in nix.
> 
> So the bootstrap files are basically a big huge weak point in our security story.
> 
> Maybe we shouldn’t make it easy to poke that weak point. But even we should, doing so is not the kind of PR that should be “LGTM merged”.

Possible Mitigiations:

> **Require copypasta in bootstrap-files update PRs**
>
> @trofi and I always made sure to include copypasta in our bootstrap-files update PRs that the reviewer could execute for themselves to validate the data being copied into the expression.
> - [Example from one of my PRs](https://github.com/NixOS/nixpkgs/pull/183487)
> - [Example from one of @trofi’s PRs](https://github.com/NixOS/nixpkgs/pull/288866)
> 
> I think we need to require these shell commands in any bootstrap-files updates, and the person merging the PR needs to confirm that they executed those commands and checked the hashes.
> 
> When merging a bootstrap-files update, you must confirm that you ran the copypasta commands and checked the result, **before merging the PR**.

> **Restrict who can merge bootstrap-files updates**
>
> In the past, only @lovesegfault could upload to `tarballs.nixos.org`, so he was the only one who could take the final step which would allow CI to pass. Because of this, he was basically always the person who merged these PRs, because they were unmergeable until he took action.
> 
> It looks like this upload process may have become automated somehow. That’s good

---

<div class="post-metadata">

### Author: ![arianvp](https://discourse.nixos.org/user_avatar/discourse.nixos.org/arianvp/32/602_2.png) [@arianvp](https://discourse.nixos.org/u/arianvp)
#### Post date: [August 13, 2024, 6:27am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/2 "2024-08-13T06:27:35Z")

</div>

Great find. This is scary.

I suggest we set up a branch protection rule in combination with CODEOWNERS that only allows specific team to merge Bootstrap files.

If GitHub isn’t rich enough to express this constraints we might need to consider making merge bot mandatory and encode it on there

---

<div class="post-metadata">

### Author: ![Atemu](https://discourse.nixos.org/user_avatar/discourse.nixos.org/atemu/32/10383_2.png) [@Atemu](https://discourse.nixos.org/u/Atemu)
#### Post date: [August 13, 2024, 7:58am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/3 "2024-08-13T07:58:00Z")

</div>

Would it be possible to verify bootstrap-files integrity using CI (i.e. ofBorg)?

A red CI would be a red flag (literally) to any committer.

---

<div class="post-metadata">

### Author: ![Sandro](https://discourse.nixos.org/user_avatar/discourse.nixos.org/sandro/32/2865_2.png) [@Sandro](https://discourse.nixos.org/u/Sandro)
#### Post date: [August 13, 2024, 11:47am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/4 "2024-08-13T11:47:54Z")

</div>

> [@arianvp](#):
>
> I suggest we set up a branch protection rule in combination with CODEOWNERS that only allows specific team to merge Bootstrap files.

I don’t think github allows that granularity. We would need to enforce that every codeowner needs to approve PRs they are owner for which doesn’t work with our current workflow.

---

<div class="post-metadata">

### Author: ![Sandro](https://discourse.nixos.org/user_avatar/discourse.nixos.org/sandro/32/2865_2.png) [@Sandro](https://discourse.nixos.org/u/Sandro)
#### Post date: [August 13, 2024, 11:50am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/5 "2024-08-13T11:50:14Z")

</div>

> [@Atemu](#):
>
> Would it be possible to verify bootstrap-files integrity using CI (i.e. ofBorg)?

There are issues for this already but ofborg development seems to be really dried out, so not sure if anything in that direction is going to happen.

[Make GitHub suggestions for fixing sha256 fields · Issue #429 · NixOS/ofborg · GitHub](https://github.com/NixOS/ofborg/issues/429) [Detect url/sha256 mismatch · Issue #647 · NixOS/ofborg · GitHub](https://github.com/NixOS/ofborg/issues/647)

---

<div class="post-metadata">

### Author: ![hxtmdev](https://discourse.nixos.org/letter_avatar_proxy/v4/letter/h/aca169/32.png) [@hxtmdev](https://discourse.nixos.org/u/hxtmdev)
#### Post date: [August 13, 2024, 12:10pm UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/6 "2024-08-13T12:10:06Z")

</div>

This would be a good context to think about replacing widespread write access with a bot that does most actual merging again.  
See [Automatic Merging Implementation · Issue #112 · NixOS/ofborg · GitHub](https://github.com/NixOS/ofborg/issues/112) and especially [Feature Request: Automatic merging · Issue #104 · NixOS/ofborg · GitHub](https://github.com/NixOS/ofborg/issues/104)  
Direct write access for most users could be replaced by some kind of whitelist of users that can instruct the bot to merge IF all “must-succeed” checks (such as “no bootstrap file changes” or “only allow bootstrap file changes when at least 3 \<specific team\> members approved”) succeeded.

---

<div class="post-metadata">

### Author: ![amjoseph](https://discourse.nixos.org/user_avatar/discourse.nixos.org/amjoseph/32/28378_2.png) [@amjoseph](https://discourse.nixos.org/u/amjoseph)
#### Post date: [August 16, 2024, 7:21am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/7 "2024-08-16T07:21:02Z")

</div>

I believe [these two commits](https://git.sr.ht/~amjoseph/nixpkgs/log/pr/bootstrapFiles-attack-defense) prevent the attack:

- [testers.invalidateFetcherByDrvHash: move to lib.derivations](https://git.sr.ht/~amjoseph/nixpkgs/commit/07a907972aabfda14089d17adc6addd6903fd72a)
- [stdenv: bootstrapFiles: use invalidateFetcherByDrvHash on bootstrapTools](https://git.sr.ht/~amjoseph/nixpkgs/commit/40beef6268180dcb41f7c3455ef78a327b5fec90)

Currently building with this.

The defense works by wrapping the bootstrapFiles fetcher in `invalidateFetcherByDrvHash`. This requires that `invalidateFetcherByDrvHash` is moved from `pkgs` to `lib`, because we don’t have a `pkgs`-set so early in the `stdenv` bootstrap.

Needs to go to staging. Would appreciate comments from @trofi. Also needs approval from the `lib` folks for moving `invalidateFetcherByDrvHash` from `pkgs` to `lib` (which seems reasonable, since `invalidateFetcherByDrvHash` is totally standalone code that doesn’t use anything from nixpkgs).

---

<div class="post-metadata">

### Author: ![emily](https://discourse.nixos.org/user_avatar/discourse.nixos.org/emily/32/1628_2.png) [@emily](https://discourse.nixos.org/u/emily)
#### Post date: [August 16, 2024, 5:12pm UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/8 "2024-08-16T17:12:41Z")

</div>

I didn’t know about `invalidateFetcherByDrvHash` before, very interesting! I guess the docs should probably move too if the function does.

---

<div class="post-metadata">

### Author: ![Mic92](https://discourse.nixos.org/user_avatar/discourse.nixos.org/mic92/32/5265_2.png) [@Mic92](https://discourse.nixos.org/u/Mic92)
#### Post date: [August 16, 2024, 5:33pm UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/9 "2024-08-16T17:33:22Z")

</div>

Yeah. We had also discovered this a while ago and I hat proposed to have a mitigation plan as part of a sovereign tech fund proposal, which didn’t made it into the final list.  
Here is the idea: Have a CI job on master trying to download every new FOD from it’s url and if the hash mismatches, throw an alert for manual inspection. Bootstrap files are not very special I would say. Any critical dependency can be affected and cause damage. An attacker just must control any package and submit a malicious version that contains both the package to be updated and a different maliciously patched package.

---

<div class="post-metadata">

### Author: ![ElvishJerricco](https://discourse.nixos.org/user_avatar/discourse.nixos.org/elvishjerricco/32/984_2.png) [@ElvishJerricco](https://discourse.nixos.org/u/ElvishJerricco)
#### Post date: [August 16, 2024, 7:48pm UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/10 "2024-08-16T19:48:08Z")

</div>

Agreed; there are tons of packages that are _at least nearly_ as critical as the bootstrap files, like the kernel. Any solution should not be exclusive to bootstrap files, and just checking every FOD in CI or something seems like the way to go to me. There is the concern of FODs that are too resource intensive for CI, but I think those should just be flagged as failures, just like if it was a hash mismatch. That way someone knows to manually check it.

---

<div class="post-metadata">

### Author: ![Infinisil](https://discourse.nixos.org/user_avatar/discourse.nixos.org/infinisil/32/26523_2.png) [@Infinisil](https://discourse.nixos.org/u/Infinisil)
#### Post date: [August 19, 2024, 3:51pm UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/11 "2024-08-19T15:51:41Z")

</div>

Lib folk here, sounds good to me! I’ll also get auto-pinged as a code owner when a PR is opened

---

<div class="post-metadata">

### Author: ![amjoseph](https://discourse.nixos.org/user_avatar/discourse.nixos.org/amjoseph/32/28378_2.png) [@amjoseph](https://discourse.nixos.org/u/amjoseph)
#### Post date: [August 22, 2024, 6:18am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/12 "2024-08-22T06:18:00Z")

</div>

> [@Mic92](#):
>
> Bootstrap files are not very special I would say.

Well, they are the only binaries that are exempt from `NIXPKGS_ALLOW_NONSOURCE=0`.

They’re also the only binaries upstream of `stdenv`.

Sure, you could run this same attack on the `gcc` source tarball, but the exploit would need to be source code. This doesn’t make the exploit any easier to discover (it didn’t for xz) but _once discovered it makes it impossible to deny that an exploit happened._ For the bootstrap-files we don’t have that guarantee.

In any case I’m certainly in favor of applying this defense more broadly, but the bootstrap-files are where we really need it most.

---

<div class="post-metadata">

### Author: ![marius851000](https://discourse.nixos.org/user_avatar/discourse.nixos.org/marius851000/32/5005_2.png) [@marius851000](https://discourse.nixos.org/u/marius851000)
#### Post date: [August 30, 2024, 10:26am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/13 "2024-08-30T10:26:15Z")

</div>

I wrote a script, and ran it on a few packages (`hello` and `libreoffice`), and it found no difference between the fetched derivation of FODs and their hash (outside of GitHub returning some temporary 403 errors, solved by retry). I’ll add a backoff retry logic, and share it for those who are interested to test other packages. It’s a python script directly calling nix commands.

---

<div class="post-metadata">

### Author: ![boomshroom](https://discourse.nixos.org/user_avatar/discourse.nixos.org/boomshroom/32/4325_2.png) [@boomshroom](https://discourse.nixos.org/u/boomshroom)
#### Post date: [October 4, 2025, 5:17pm UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/14 "2025-10-04T17:17:27Z")

</div>

This is why the work on the [minimal-bootstrap](https://github.com/NixOS/nixpkgs/issues/123095) project is so important. [Here’s the most recent PR on the subject that I could find.](https://github.com/NixOS/nixpkgs/pull/260193)

---

<div class="post-metadata">

### Author: ![Lun](https://discourse.nixos.org/user_avatar/discourse.nixos.org/lun/32/5552_2.png) [@Lun](https://discourse.nixos.org/u/Lun)
#### Post date: [October 4, 2025, 10:33pm UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/15 "2025-10-04T22:33:13Z")

</div>

@pyrox has been working on it more recently

---

<div class="post-metadata">

### Author: ![pyrox](https://discourse.nixos.org/user_avatar/discourse.nixos.org/pyrox/32/29763_2.png) [@pyrox](https://discourse.nixos.org/u/pyrox)
#### Post date: [October 5, 2025, 12:37am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/16 "2025-10-05T00:37:49Z")

</div>

I’ve been working on getting the existing packages updated and cleaned up a bit, and then will be taking the changes from the linked PR into my branch(with proper authorship maintained, of course), ensuring they work properly, and then will push all of that work through. Once that’s done, moving nixpkgs’ base from the existing bootstrap-tools files to these new bootstrapped derivations should be as easy as a derivation that creates the appropriate tarball, then substituted into the stdenv in place of the static tarball.

(note that while i say “easy as”, this is in reality still fairly involved and there’s a lot of work to do. However, this is something I expect could be done as soon as the 26.05 release(I am not making any promises, just guesses based on the speed of progress))

Note also that I am a college student doing this in her free time, and have other responsibilities in nixpkgs(25.11 release editor, package maintainence), and also do not want to work on a single project for months at a time, so I will be taking frequent breaks. I am trying to keep somewhat steady progress updates posted to the `#stdenv:nixos.org` matrix channel(Nixpkgs Stdenv) in the NixOS space, feel free to ping me there if you have any questions, comments, concerns, or would like to help!

Hopefully final edit: I 100% agree with the security concerns of trusting a bundle of static binaries, which is why I’m doing this! I do feel that this work has practical security benefits for nixpkgs users, and doing it gives us a more stable and trusted base to build off of. As a future cybersecurity major, doing stuff like this is why I got into the field, and it’s something I’m super passionate about, and even gave a very short talk at my university to a small group about why this work is important but also how difficult it can be.

---

<div class="post-metadata">

### Author: ![siraben](https://discourse.nixos.org/letter_avatar_proxy/v4/letter/s/ad7895/32.png) [@siraben](https://discourse.nixos.org/u/siraben)
#### Post date: [October 5, 2025, 2:56am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/17 "2025-10-05T02:56:56Z")

</div>

@pyrox saw your PR, thanks advancing the progress on minimal bootstrap. Feel free to ping me if you need me to do `nixpkgs-review` on packages made with this bootstrap, I have a beefy build machine lying around.

---

<div class="post-metadata">

### Author: ![pyrox](https://discourse.nixos.org/user_avatar/discourse.nixos.org/pyrox/32/29763_2.png) [@pyrox](https://discourse.nixos.org/u/pyrox)
#### Post date: [October 5, 2025, 3:02am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/18 "2025-10-05T03:02:02Z")

</div>

I’ve been doing all this bootstrapping on my laptop!(It’s a modern laptop, but still, doesn’t need a build farm thankfully) Since it’s not the stdenv root at the moment, it’s fairly light and easy to build(you can build the whole thing in probably around 20-30 minutes by checking out the pr and running `nix build -f . pkgsi686Linux.minimal-bootstrap.test`!)

Edit 2: You will need to run `nix build -f . make-minimal-bootstrap-sources` first, then the above command. Weird bootstrapping thing that’s unavoidable. But those two commands build up the entirety of the package set from the PR!

Edit 3: For those interested, the PR: [minimal-bootstrap: lots of updates by pyrox0 · Pull Request #448701 · NixOS/nixpkgs · GitHub](https://github.com/NixOS/nixpkgs/pull/448701)

---

<div class="post-metadata">

### Author: ![rnhmjoj](https://discourse.nixos.org/user_avatar/discourse.nixos.org/rnhmjoj/32/1792_2.png) [@rnhmjoj](https://discourse.nixos.org/u/rnhmjoj)
#### Post date: [October 5, 2025, 7:31am UTC](https://discourse.nixos.org/t/bootstrap-files-updates-amplifiy-exploit-of-any-package-into-exploit-of-every-package/50534/19 "2025-10-05T07:31:38Z")

</div>

> [@amjoseph](#):
>
> use the storepath-hash of the exploit as the `hash` argument of the `import <nix/fetchurl.nix>` call. Hydra won’t bother trying to download from `tarballs.nixos.org`

Is there a way to prevent this?

Just the other day I was updating some pins in my configuration, but I accidentally swapped two hashes: Nix happily built my system with `<nixpkgs>` pointing to `<nixos-mailserver>` consequently breaking every nix command.
