Transforming global software distribution with Nixpkgs

Announcements
1

Hi everyone,

I’m excited to announce that the Sovereign Tech Agency decided to invest 230 000 EUR into improving global IT supply chain security, as well increasing sustainability and reducing tech debt around the Nix ecosystem! We had applied with a development program intended to address a number of known attack surfaces and bottlenecks by end of February 2026:

This presents a unique opportunity to have experts spend quality time on resolving difficult, long-standing issues, and reinforces the role of NixOS as a critical piece of digital infrastructure.

The Sovereign Tech Agency invests globally in open software components that underpin our digital infrastructure. Their investments scale across many sectors and benefit a broad range of users, directly enhancing productivity, sustainability, and capacity for innovation. Check out their other funded projects to see the broader impact they’re making in strengthening the open source ecosystem.

Background

Back in July 2024, a group of contributors came together to prepare a project application for the Sovereign Tech Fund, following our participation in the Contribute Back Challenge 2023.
After multiple revisions, we went through two iterations with the Sovereign Tech Agency to answer questions, adjust the proposal, and sign a contract.

I would like to thank everyone involved in making this possible, with special thanks to the NixOS infrastructure team (@hexa) and the NixOS Foundation board (@ron @edolstra) at that time.
Great many thanks to the Sovereign Tech Agency team for the trust put into the project team, and also to the giant Nix ecosystem contributor community on the shoulders of which this effort builds.

This announcement comes a bit late since we were first waiting for the contract to be official before talking about the outcome of the application in public, and then conflicting priorities on my end that delayed the write-up.

Progress so far

Some of us already met in March at Ocean Sprint 2025 to coordinate and prepare for the first milestones:

@multivac61 iterated on the FOD tracker design with @Mic92, and refined a plan for rolling it out to Nixpkgs CI checks. The necessary API surface is now implemented, and some scalability issues were ironed out. Next up will be supporting incremental and periodic consistency checks.

@das_J and @Conni2461 together with @hexa and @mic92 collected requirements for rewriting Hydra’s queue runner, which is currently a bottleneck that prevents utilising our full build server capacity. There is a (to be published) prototype that connects to the Hydra database and manages the entire communication between the scheduler and workers. Next up would be constructing sensible build queues, after which the prototype should be ready for testing and open for further improvements.

In other news, the initial investment from the Sovereign Tech Challenge in the Nix ecosystem is bearing fruit: The prototype Nixpkgs vulnerability tracker received private follow-up funding to get it ready for productive use.

Stay up to date

Subscribe to this thread to be notified of updates as milestones are reached!

95 Likes
2

Amazing news !!! Looking forward for the outcome.

3 Likes
4

I’ve never subscribed to a thread faster. Amazing news and a great set of projects. If anyone wants to blast out how to help on any of these efforts, especially if it’s user testing or building or self-hosting, I know I’m not the only one eager to help.

2 Likes
5

I am glad to see efforts about improving nix evaluator performance. Just curious, what is the multi-evaluator support?

Really looking forward to an improved build farm. I hope rebuilds of a flat package set that consists of thousands of small packages (each takes a few seconds to build) can go directly into master instead of staging one day.

1 Like
6

Paying taxes has never been so fun. I’m happy to see this come to fruition!

7 Likes
7

230,000 amounts to 0.00002% of Germany’s tax revenue. Which is awesome. I wish my government in the UK would do something obvious and good like this. Or maybe I should be paying German taxes instead. The nix repl failed to calculate this, so I had to nix-shell a better calculator.

❯ nix repl
Nix 2.28.3
Type :? for help.
nix-repl> (230000 / 947700000000) * 100
0

❯ nix-shell -p calc
this path will be fetched (0.85 MiB download, 5.28 MiB unpacked):
  /nix/store/33hds0liqby9s2mn2gy16yz230kgw9qf-calc-2.15.0.2
copying path '/nix/store/33hds0liqby9s2mn2gy16yz230kgw9qf-calc-2.15.0.2' from 'https://cache.nixos.org'...

❯ calc
C-style arbitrary precision calculator (version 2.15.0.2)
Calc is open software. For license details type:  help copyright
[Type "exit" to exit, or "help" for help.]

; (230000 / 947700000000) * 100
	~0.00002426928352854279
4 Likes
8

@matthewcroughan Your numbers are off by factor 100.

947.7 billion is not

9 477 000 000

(that is ~9 billion).

2 Likes
9

That only makes it more incredible. Thanks for catching, edited above.

1 Like
10

That makes it more credible in my opinion – a much smaller fraction of Germany’s tax money is spent on Nix.

11

Not only do I fail at math, but also at English. Though google defines “incredible” as “difficult to believe; extraordinary”. I’m shocked at how small a number this is. Compare it to the GDP which is 4.5 trillion, and wow again. Open source gets so little, and deserved more recognition long ago. I now wonder how much other projects or things get from tax revenue so we can add it all up.

2 Likes
12

On that note, I just found Sovereign Tech Agency - Wikipedia, which details some of the other funding that the agency has given. I calculated the total funding to be around 16669152 EUR, which comes to 0.18% of German tax revenue.

1 Like
13

This is wonderful news!

2 Likes
14

A bit lost in translation, probably to keep the proposal brief.
It refers to the possibility to have multiple “implementations” of an abstract evaluator interface, more so than multiple instances of the current implementation, which already mostly works¹.
Currently when code talks to the evaluation cache, it is quite different from other code that talks directly to the evaluator, yet it has to fall back on the uncached operations for some things.

This item is more of an architectural change and an intermediate step than a user-facing improvement.

¹ Multiple instances works in unit tests. Probably not useful in practice, and not tested in practice. Usefulness might improve after other changes which are not part of any current plans.

3 Likes
15

I’m not sure if this is what you refer to as “Assert reproducibility of Nixpkgs source retrieval”, but in case, this seems like a great place to mention (thanks @RaitoBezarius for pointing me here) this CVE I raised a year ago NVD - CVE-2024-36050 that I also discussed with the security team (I had no recent answer after I said I would create the CVE), and it is discussed in re-fetch source when url changes · Issue #969 · NixOS/nix · GitHub and Provide a binary cache for builds · Issue #68 · NixOS/ofborg · GitHub.

This describes an attack that is almost impossible to detect (for a PR reviewer) allowing a malicious PR to inject arbitrarily malicious code in basically any software, by simply changing the hash. The attack is very easy to run and can have devastating effects (trapdoor, viruses…). Yet, as far as I know, one year later, it is still effective, even if I proposed some proposals for potential fixes (the best fixes are not completely trivial as they require nix & caches to also maintain a list of verified couple hash/url, but as far as I see this is the only really robust solution to this issue).

@RaitoBezarius mentionned https://fod-oracle.org/, but this is a very hacky & expensive way to check nixpkgs using external tools, and would not apply outside of nixpkgs. A cleaner solution would require directly nix & caches to maintain (and distribute) a list of verified (hash, url) tupples, but this require some transformation on the nix APIs, but at least it is clean and should be reasonably efficient (i.e. avoid re-downloading all sources when fetching an untrusted derivation).

16

Hey @tobiasBora, I’m Óli,the maintainer of fod-oracle.

Thank you for your detailed feedback on the CVE-2024-36050 vulnerability and the broader security concerns around Nixpkgs source retrieval. I appreciate you taking the time to explain the attack vector and its potential impact.

You’re absolutely right that the current situation represents a significant security risk. The ability for malicious PRs to inject arbitrary code simply by changing hashes is indeed concerning, especially given how difficult it would be for reviewers to detect such attacks.

I’d love to learn more about your thoughts on how fod-oracle could be improved or what a more elegant solution might look like. While I understand your concerns about it being “hacky & expensive,” I believe it serves an important role in the current ecosystem as we work toward better solutions.

Today, given a nix expression, you can use fod-oracle to detect all direct and transitive Fixed-Output Derivations (FODs). As far as I understand, this capability hasn’t been easily achievable until now and forms a solid foundation for building other security tools and services for the community.

For example, in nixpkgs CI, fod-oracle will help with:

• Hash validation: When someone changes a hash, we can confirm that this is the correct hash in the right place by verifying the actual content matches the expected hash

• Infrastructure change detection: When somebody modifies the underlying infrastructure that creates FODs but keeps the hash the same, we can verify whether all those hashes actually remain the same or whether they would change under the new infrastructure

I completely agree that the ideal solution would involve Nix and binary caches maintaining verified (hash, URL) tuples as you suggested. This would provide a much more robust and systematic approach to preventing these attacks. However, until such fundamental changes are implemented in Nix itself, tools like fod-oracle can serve as important interim security measures.

What specific improvements would you suggest for fod-oracle in the meantime? And do you have thoughts on how we might advocate for or contribute to the more comprehensive Nix-level solutions you mentioned?

4 Likes
17

@multivac61 Thanks a lot for joining this discussion, and for your time implementing fod-oracle. I’m really happy to see that someone takes this seriously, and my comment was not at all a criticism against your work but just a clarification about its scope, in the sense that the only clean way to fix this issue is to fix nix itself as I can easily come up with more complex (but targetted) attack vectors that are just impossible to detect by just watching nixpkgs.

That being said, I’m really curious to understand how you implemented fod-oracle, as knowing this may help to see potential issues. Can you maybe explain a bit more how it works internally + how you implement it? To be honest I would expect your solution to be more complex to code than fixing nix itself ^^

I’m confused… do you mean that you hash the content of the nix store /nix/store/XX-foo and check that you find XX (if so this helps to prevent the recently fixed race condition attack on nix, but not my CVE), or do you re-download the url and check that this precise url gives XX? (If so that would indeed help to detect issues about my CVE, but would be a bit costly since you need to redownload all sources of nixpkgs… but no other way to fix this without fixing nix itself) Do you also manage to do it either for the full nixpkgs and for the diff between the current version and previous one? (In practice no-one wants to redownload all sources of nixpkgs for each PR ^^) If so really curious to know how you can implement this without changing/reimplementing nix!

Btw, note that checking for duplicate hashes in nixpkgs itself is NOT enough to insure security since one can for instance use a hash injected in a previous version of nixpkgs to be more discrete since caches are persistent, use a hash present in a corrupted cache like cachix, or maybe (never tried) use the hash of an output derivation instead of the hash of an existing source.

To advocate for change, one can maybe run an actual attack to show that the current system is broken (at the risk of being banned) :stuck_out_tongue:

Regarding how to fix nix, I don’t know enough about its internal to say “modify this function”, but at a high level I can think of a smooth and modular change that would let hydra implement a new API first, and let clients opt-in if they wish latter, with a first a warning if the cache does not implement the API and in the longer term an error. I explain in Provide a binary cache for builds · Issue #68 · NixOS/ofborg · GitHub how this may be implemented in term of functionality, in practice I would certainly implement it by doing a change in nix so that every time it computes a FOD, it stores in a database the tuple (hash—to save space—of derivation including url etc, hash of FOD) if it is a new FOD and check before this same database to see if the FOD was not already created before, here or in a cache (needs the new API for cache but this can be at the beginning only an “if the cache implements it use it otherwise print a warning”, and since hydra uses no cache it would still detects many (not all, only the derivations built by hydra!) attacks even without an API). Then in a second time we can implement an API for caches to allow querying this database in a remote cache to allow the verification described above also for derivation not built by hydra, but this is rather in the cache/remote-related code. Not sure if it is clear, feel free to ask if not! I can also provide MWE I gave to nix’s security team to test attacks if needed.

Thanks for your work
!

2 Likes