Postmortem from the Nix team
Timeline
The Nix Team timeline:
- @xokvidium was going through build failures in Nix’s own CI around ~10.00pm UTC.
- At that point Address ifdef problem with macOS/BSD sandboxing (backport #13455) by mergify[bot] · Pull Request #13458 · NixOS/nix · GitHub from DetSys was opened and this was not yet clear to be a security problem.
- The fix is merged and the backport triggered, the Nix Team is made aware of the regression, but none are available at that point to do the point release.
- 6.45am UTC: Matrix room with the Nix and Security Team is set up.
- 7.00am UTC: @edolstra triggers the Hydra job for 2.30.1 and uploads ~7:40am UTC.
- 1.30pm UTC: Nixpkgs nixVersions.git and cachix/install-nix-action update PRs are opened and merged shortly after.
Mitigations and improvements
This incident was largely beyond the control of the Nix team due to the short timeline.
Nonetheless the following process mitigations respectively will/ought to apply:
- Speed up the release process: @Mic92 and @tomberek will work on further automating the release process so that it can be triggered by any Nix team member. In this case @xokvidium would have triggered a release.
- @edolstra now has pager duty in Determinate Systems to quickly respond to incidents like this.
- We urge @grahamc not to disclose a vulnerability before coordinating with the relevant implementations.
This includes cases where the vulnerability is somewhat easy to discover, because a fix was merged.
At that point we believe the most responsible course of action is not to call attention to it just yet.
Relevant implementations may normally include: upstream Nix, Lix, Guix. Perhaps in rare cases Snix or Tvix. We appreciate that the Nixpkgs security team was involved with this. - We have enabled the GitHub branch protection rule that requires a review. In practice, the team normally applies the 4 eyes principle, and it was followed in this instance as well because the change was originally made by different authors. Nonetheless, GitHub’s formulation of the review rule would have restricted the merge of this particular bug.
Besides the review and release process improvements, we will further improve our automated testing and analysis.
- Apply more analysis tools; linting, static analysis, and machine learning
- Improvements to the test suite in terms of tests, and environments in which they can be run
Notes of our discussion, with more technical details, can be found in 2025-07-16 Nix team meeting minutes #235