Three weeks ago, unbound 1.25.1 was released fixing 11 CVEs including a CVSS 10.0 (cache poisoning) and CVSS 9.8 (RCE during DNSSEC validation).
The nixpkgs security tracker flagged it within 24 hours.
The fix was two lines, bump the version and update the hash. ofborg CI passed all 36 checks across all platforms. It has been sitting in staging ever since.
As of today, June 9th 2026 — 20 days after release — users on both nixos-unstable and nixos-26.05 are still running the vulnerable 1.25.0. The fix is merged and sitting in staging-next-26.05
The rebuild count that forced it into staging was reported as 5k+ packages. The actual number of packages that link libunbound is a fraction of that — a significant part of the count is nixosTests.* VM infrastructure that is not shipped software and cannot be broken by a DNS resolver point release with no ABI changes.
The staging process exists to protect against cascading breakage from dependency tree changes. A security point release with no API/ABI changes, a passing test suite, and a fixed changelog is the lowest-risk change imaginable. The heuristic of “rebuild count > N → staging” breaks down entirely here and the tooling applies it anyway.
Concrete proposal:
Exclude nixosTests.* from rebuild counts — they are validation artifacts, not shipped software
Add a security fast-lane: CVSS ≥ 9.0 + no ABI changes + passing tests → eligible for direct master merge with NixOS Security Team sign-off
I believe it exists to prevent mass rebuilds from impacting infra, not to prevent breakage.
Pure ABI changes shouldn’t matter when everything gets rebuilt.
Or more precisely, infra wouldn’t be able to manage such amount of rebuilds (i.e. long times without binaries, similarly to now if you followed staging). And even the moment of merging to master would be unpleasant, as people, PRs etc. would be missing huge amounts of binaries. Mass rebuild security fixes get merged at daily basis nowadays.
I think it might be helpful to re-read vcunat’s reply a few times. As far as I can tell, nothing they said implies the existence of some mechanism that did not get applied for libunbound.
PR merging is distinct from building is distinct from channel updates, despite all these things being related. Keep in mind that nix is input addressed, thus any referrer (including indirect referrers) MUST be rebuilt; this is the whole rationale for staging to exist.
i understand that part, but my point is that a lag of 20 days for a cve of this severity is too long? i did misunderstand the process a bit, but why is current nixpkgs, even unstable is still on the vulnerable version?
Per CONTRIBUTING.md, ‘critical security fixes’ should be merged to staging-next, not staging.
I don’t know who makes the decision about whether a security fix is critical per this policy, or what the relevant considerations are. So perhaps the process was followed here, as designed; perhaps not. But in principle, if it’s a big deal, it ‘should’ be processed faster than this.
There was no staging-next at that point. It got merged to master about 1 day before the unbound PR got opened. Some very critical fixes were merged directly to master in the past, but that won’t speed up getting the fixes into the big channels too much in comparison to the usual process. (it speeds up the -small channels)
Note that the rebuilds are expensive for the infra. One staging* cycle takes 2 weeks, and almost all of that is blocked by waiting for the infra to process everything. And right now, we’re supporting 3 different branches at once (!) (unstable/master, 26.05 and 25.11), so the worst-case lag is more than 3*2 weeks. Though the overlap only lasts for a few weeks, but even without it there’s about 1 month worst case.
I believe we can improve some things, though. A few relevant points:
unbound in particular. I suspect that if we removed it from gnutls (default) build inputs, we would significantly reduce the amount of rebuilds which it’s causing. There it’s only used for ${gnutls.out}/lib/libgnutls-dane.so* which I suspect to be very niche (would be nice to really confirm) so we could avoid building it in the default version (and e.g. provide a gnutls-dane derivation instead).
x86_64-darwin is being dropped since 26.11 (current unstable/master/staging-next). That will decrease the load on the infra.
in infra chat we recently seem to have a consensus that we wouldn’t build nixosTests anymore. Of course we’d keep those which are chosen to be channel blockers, and perhaps there would be some other aspect to compromise, but currently the costs/benefits of building all of them all the time doesn’t seem favorable. (they take a large fraction of total Hydra’s time)
I’d say that due to the nature of Nix, doing this security fixes quickly is just not possible.
For NixOS, an alternative way could be to maintain a module (something like system.applySecurityPatches) with a list of hotfixes to be applied with system.replaceDependencies. This is updated as soon as the fix PR is merged and eventually removed when the staging cycle is complete.
For the last point, we could alternatively enable auto-allocate-uids (if it’s not already the case?) on the infra and push people to use container tests for the application tests and only use VM tests when strictly required.
We’ve been doing this in some out of tree project, the gains in terms of CI time are pretty massive.
The build infra supports container tests since May 23rd. Container tests can dramatically improve runtime speed, but I think what bogs hydra down is the large closures NixOS tests have. In the classic queue-runner model they are funnelled through the coordinator.
It would be good if a list of hotfixes for key dependencies can somehow be maintained in tree (within Nixpkgs itself) so that it simply becomes transparent to users. However, I don’t know whether this is actually possible.
It’s definitely possibile in theory, but whether it will work in practice is difficult to say.
Maintaining a list of hotfixes needs coordination between the security and infra teams, which I think are already overworked, but maybe some portion of this workflow could be automated.
Fwiw at work we have stack of patches for CVEs we care about that we pop as the relevant nixpkgs channel catches up. I think it a very reasonable workflow, and allows a pretty quick turn around, for us.
Well, if you can apply the patch and rebuild everything locally it means it probably doesn’t cause a mass rebuild. In this case the channel should update quickly, though. Unless, of course, it’s stuck for some other reason.
The real problem is when you indeed have a mass requild (like, you patch a curl vulnerability and tens of thousand of packages need to be rebuilt). These updates must go through staging and they are obviously slower.
Maybe in your case you only depend on a few packages, so even a mass rebuild may be feasible if you have a decent enough machine. However, you can still hit some packages that take ages to build, like chromium. So, I don’t think this is feasible, not for everyone at least.