Hi everyone, a big update almost 4 months into the project:
- 3 out of 7 milestones currently in progress
- New Hydra queue runner in staging, getting ready for production
- Bashless activation setup being prototyped
- FOD tracker not possible as planned, but pivot may end up better than the original design
New Hydra queue runner
Context: Nixpkgs is claimed to be the largest open source software distribution in the world, and Hydra is the continuous delivery (CD) tool to coordinate the corresponding build farm. But for a while now it has run into a bottleneck: build machines running idle because the queue runner did not catch up with scheduling new builds. Not only is this a waste of available resources, it precludes hardware donations to improve turnaround times on updates and security fixes.
The goal of this milestone was to investigate whether Hydra is still the correct solution to the problem, and fixing performance and maintainability issues if needed or otherwise coming up with a different setup.
Early technical discussion confirmed that there is no reasonable alternative to Hydra at Nixpkgs scale. A possible contender would have been buildbot-nix, but there were similar reservations about performance and maintenance effort; and more importantly swapping out such a crucial component would impose a lot of work on the volunteer infrastructure team.
@conni2461 and @das_j published GitHub - helsinki-systems/hydra-queue-runner: Hydra queue runner rewrite end of June 2025. They have been continuously meeting with @hexa @lassulus @mic92 to discuss design and progress. The service is written in Rust, and scheduler and workers communicate over gRPC (instead of the infamous Nix daemon protocol) over a bi-directional channel used the send commands, status updates, logs, and build results, and also allows clients do not need a public IP.
Recently the infrastructure team recently deployed a staging Hydra available at https://staging-hydra.nixos.org, which finally enables testing changes before rolling them out. This is a big deal! The new queue runner is part of that staging deployment, and already features the following:
- A new machine view is integrated in Hydra, querying the queue runner via REST API; build logs are working
- Some database queries were optimised along the way
- Stopping the queue-runner gracefully aborts active steps, and builder crashes are handled gracefully
- Configurable retry limits (max, interval, backoff), decision based on steps results (e.g. on connection reset); various scheduling functions, e.g. based on PSI
More testing and small corrections are still needed, but production deployment is in sight. After that the team will turn their attention to ofborg, the Nixpkgs continuous integration (CI) service that runs builds and tests on pull requests, and its deployment configuration.
Bashless activation prototype in progress
@nikstur is prototyping a hardened boot setup that does not require any interpreted scripts. This is now conceptually understood, and the next step is to write an integration test that demonstrates a system that activates without Bash.
The deliverable is supposed to be a foundation for people to build their specific use cases on top. This is a continuation of and follows the same pattern as the STF 2023 Contribute Back Challenge with removing Perl from the boot process. That had allowed an interactive (largely) Perl-less system, and ended up seeing much more uptake than originally anticipated, with a new, complied implementation of switch-to-configuration by @jmbaur:
Specifically, the goal here is to build an appliance system, which is primarily relevant for high-security applications — both as a validation that the setup generally works and a showcase for how to do it.
There’s more to unpack on why this is important and where it fits into the big picture of NixOS. Most of the boot process is rather obscure knowledge that is barely documented. But since this post is really long already, we’ll leave that for another time. Until then, you can read up on the details of work in progress in the planning document and the development branch.
FOD tracker delayed, pivoting
As noted recently, one milestone currently can’t be delivered as intended:
This writeup, which goes into quite some detail, comes only now since we’ve been busy corresponding internally and with the Sovereign Tech Agency (STA) on how to proceed.
The original plan entailed building a tool that would evaluate the source specifications for all packages and store the pairs of input and output hashes in a database. Then on each pull request, it would diff the hash pairs against the master branch, check for the conditions in the cache poisoning attack, and rebuild changed fixed-output derivations (FODs) to verify their hash pair validity. Additionally the tool was envisioned to periodically re-download all sources asynchronously to check for liveness and consistency.
The design with a separate FOD tracking service was acknowledged during planning to be less efficient than the alternative approach of patching the Nix evaluator and Hydra job scheduler to log FODs when encountering them in expressions and CI builds respectively, such that the work would happen only once. This was discussed with @picnoir @julienmalka and a few others, who had investigated closing the cache poisoning attack vector and related issues. The decision for the external service was taken to de-risk the timeline, as that design was more well-understood, and already had an (unmaintained but working) prototype implementation.
The alternative approach would have required touching brittle but depended-upon code in Hydra — for which we have an entire milestone to make it more maintainable, scheduled later in the year, so requiring that as a dependency would have been impractical — and unfinished code in Nix, which has very limited maintainer capacity and code review throughput we couldn’t rely on in planning. And keeping a patched Nix just for CI seemed like an undesirable long-term maintenance cost.
The FOD tracker architecture was implemented in Go as a CLI command and REST service at https://fod-oracle.org (source) by the beginning of June 2025. The demo deployment tracks FOD hash pairs for various Nixpkgs releases as intended. It managed to drastically improve performance compared to the Python prototype by heavily leaning on asynchronous operations, and was also able to leverage recently added facilities in Nixpkgs to parallelise evaluation. Work on the web UI envisioned for the out-of-band bulk-rebuild mode was paused in favor of first enabling the pull-request-based deployment the team deemed the more important, contributor-facing feature. The tool appeared to fulfill functional requirements and seemed ready for beginning to prepare for deployment.
During June 2025 @wolfgangwalther from the NixOS CI team brought up concerns with performance in the production environment. The email correspondence went on for more than two weeks, those involved trying to find a satisfying solution. Despite exhaustive analysis of possible workarounds, the outcome is that the tool will be an order of magnitude too slow for per-PR checks.
A full Nixpkgs evaluation with fod-oracle currently takes 20 min on a 16-core machine with 64 GB RAM, and a full re-download of all sources requires on the order of 10 hours, respecting GitHub’s rate limits. While the latter would only be needed once in a while, the former is far too much for the GitHub Actions runners which have only 2 cores. Also a full evaluation is heavily IO-bound since it has to write .drv files to disk for further processing, and placing the Nix store on a tmpfs would raise memory requirements to about 128 GB. A full evaluation would always be required to compute the FODs that changed in a PR. And simply throwing more hardware at the problem is not something the NixOS Foundation can (or should have to) afford short- or long-term.
At the end of June we had laid out an alternative strategy, detailed below. But as a result, we had to give up on the original plan. I see the reasons for this breakdown in these mutually confounding factors:
- The overall system’s complexity would have required developers with comprehensive domain knowledge and merge authority. During the planning phase, those weren’t available for the sustained time periods needed, which led us to take a suboptimal architectural approach. We were aware of the general problem and added buffers, but still underestimated the resulting risks.
- Competing commitments by everyone involved reduced communication quality, with written correspondence causing further delays, despite volunteers’ extraordinary dedication to resolve issues as they arised.
Alternative plan, requested extension of timeline
The problem analysis yielded that the ultimate security boundary is the release channel (the final artefacts built by Hydra), as that affects consumers and manifests in the cache that is susceptible to poisoning. Since the turnaround time for a release is on the order of a day or more, re-downloading even all sources in parallel wouldn’t affect CI performance at that stage, but still provide the desired security properties. In practice we’d only need to download sources for FODs that changed relative to the last released commit, which is substantially less. A full download could be scheduled at a much larger interval.
It was also noted that some fraction of sources is simply broken. While in PR mode this would not immediately block all work as we’d only check changed FODs, it would need time to stabilise that for a full successful rebuild of all sources. This is why the FOD check must not immediately block releases, and getting there will inevitably have to rely on volunteer efforts spread over time. In order to avoid loose ends, we should add a check in the release script with a flag that can be turned on once all sources are stable.
@multivac61 and I tried to refine and estimate the changes agreed on with the other contributors. In that session, we could not convince ourselves that the new timeline we were about to propose to the STA would work out as intended, primarily due to other obligations already planned. Rearranging the code to get more clarity on the new setup made it evident that the changes would overall amount to a rewrite, in the sense of moving and re-testing a lot of code. We concluded that although the insights gained so far would lead to the desired results, it would take more time than would be available to deliver them even with an extended timeline. @multivac61 decided to stepped down from the project; we can keep using the work under a free license.
As a last attempt, I asked Nix and Hydra maintainers @Ericson2314 @roberth @Mic92 @Conni2461 @das_j for their assessment on whether the more efficient solution (as mentioned earlier by @tobiasBora) could be done in reasonable time: Logging FODs during evaluation directly in Nix to use in nix-eval-jobs, adding a table for input/output hash pairs in Hydra, scheduling FOD rebuilds with the new queue runner, and displaying the status in a new Hydra UI view. My idea was that since the main reasons for failure so far were lack of aggregate domain expertise and lack of access to the essential components — with Hydra improvements well underway and the small required Nix changes being in principle possible with good-enough coordination, there was a chance to get the optimal solution in terms of performance and architectural complexity. And the sentiment from maintainers was that it should work.
We met to explore detailed options and possible risks. It turned out that most of the work would be needed in Hydra and would touch parts that are being worked on already or will be in upcoming weeks (that is, code that can now be dealt with), as well as a much smaller change in Nix that is deemed uncontroversial. Specifically, we converged on the following:
- In the new queue runner, make sure to traverse the entire derivation closure, cutting off at existing store paths. Cutoff must be decided at each derivation, and the old pruning behavior would be conditional on changed FOD hash pairs.
- To access the hash pairs directly from the Nix database, stabilise just the necessary interfaces from the long-experimental content-addressed (CA) derivations. (The hash pairs are currently called realisations in Nix CA terminology.)
- To further reduce risks, for simplicity Hydra could keep its own table for realisations, and we would only switch to Nix-native realisations if they land soon enough.
- When encountering an FOD in the queue runner, perform the scheduling logic as usual, and additionally query the hash-pairs table in order to decide whether to create an extra build step which is the same, except:
- It’s a reproducibility check (similar to existing repro check functionality, needs investigation)
- Nothing depends on this task (dependants use the build step as is)
- The build step has an extra
requiredSystemFeatures += fod-check which does not exist in the original derivation. It is only added for scheduling purposes.
- Have the infrastructure team provide a build machine with
supportedFeatures = fod-check, mandatoryFeatures = fod-check, for each system that needs it (only x86_64-linux for now, as already discussed), such that jobs will naturally be dispatched correctly. The queue for those will have limited concurrency to avoid rate limits.
- Extend the queue runner’s worker protocol to account for FOD rebuild status.
- Add a new UI view to Hydra that shows rebuild status for a jobset (needs refinement).
Notably, while the CA feature as a whole is decidedly not stable yet, FOD checks would not require full roll-out. Realisations add little surface and are well-understood, and are arguably ready for stabilisation. This design would allow not changing anything about ingesting Nix expressions into the build queue, but instead do the additional work downstream when processing the build graph. It would also re-use existing mechanisms for job scheduling, minimising the number of new moving parts to deal with for maintainers and the infra team.
Important for this milestone is that the queue runner is the crucial component needed for efficient FOD checking, and it’s getting ready for adding features. The new situation and revised plan allow capturing synergies that were out of reach in the original proposal: The changes to Nix could be made by Nix maintainers who are involved in the broader development program, now soon available to contribute. This would allow us to make FOD verification much more efficient and easier to maintain long-term than originally anticipated. We requested an extension of the due date to end of November 2025, in order to make sure not to affect any of the other milestones.
Observations
A few personal notes, since these few past weeks were not easy for some of us. I’m very impressed by everyone’s dedication to do the right thing and do it well, against all odds. And I’m grateful to participate in the process. Thank you very much @multivac61 @Mic92 @wolfgangwalther @roberth @Ericson2314 @Conni2461 @das_j — and everyone else who helped out here and there — for your patience, endurance, and expertise. Thank you also @ryantrinkle @lassulus from the NixOS Foundation board for your indispensable moral support.
Reality is that we had to make many compromises on the way, so from the outside the result may rightfully look dissatisfying. I already noted this in various opportunities for feedback: the STA’s support enables us to grow organisational capacity, and is a chance to demonstrate we can get things done as a community working on a shared resource; this is not self-evident and to me feels like a great leap for NixOS. Everyone I work with is committed to getting this to production and has put in substantial effort so far, with a high degree of professionalism, building a spirit of “we can do this”.
We’re now waiting for formal feedback from the STA, that we have been in close touch with. I’ll post notes here as we go.
Coming soon
We’ve used the opportunity of various emergency meetings for warming up with @roberth and @Ericson2314 about their milestones to investigate Nix evaluator performance improvements and enable installations with an unprivileged Nix daemon. These are planned for the last quarter of 2025 and the first months of 2026. @raboof and @julienmalka will also start out in October 2025 with further improving the collection of build result hashes. The goal of that milestone is to automatically track and report on the degree of build reproducibility – which is already pretty good as you can read up in Is NixOS truly reproducible? to pass the time until the next update.
If you have any questions, feel free to ask them here or in direct messages!
