Give automated feedback when a commit causes breakage

Meta Moderation
1

Hypothesis
A package changing from building to breaking in Hydra is often manually discovered by users of the package. The author of the commit causing breakage often does not know about this breakage until it is manually discovered by someone.

Idea/Solution: Tightening the feedback loop
git bisect (Git - git-bisect Documentation) + Hydra could be used to track down the exact commit causing a breakage. Once the exact commit has been found - a bot could comment on this commit with a link to the first failed Hydra build.

That way, the author of the commit will get automated feedback that their commit broke something as quickly as possible, which means the package might spend less time overall being broken - improving the overall “robustness” of nixpkgs unstable.

Potential downsides

  • This might not be as common a problem as I imagine, and not worth spending a lot of time on
  • Performing these bisections every time something breaks might be too resource intensive on the Hydra infrastructure

I’d love to hear your thoughts on this!

  • Is the hypothesis correct (is this worth solving)?
  • Would it be too resource intensive for Hydra to do by default?
  • Are there some other downsides I didn’t consider here?
2 Likes
2

Hydra has a feature to notify committers (without bisection), but “we” haven’t used it in the past few years. IIRC it was considered (by Eelco) to generate too large amount of e-mail notifications in practice.

3

I’ve also been concerned about this. I went ahead and created such a CI system, called nixpkgs-upkeep, for a subset of packages that I care about. Feel free to submit a PR for a package that you care about!

Things that it does:

  • Builds a set of packages off of nixpkgs master, every 12 hours.
  • Automatically creates issues when those builds fail, cc’ing the package maintainers. Logic exists to avoid creating duplicate issues if a package is broken for more than 12 hours.

Things that it does not yet do (PRs welcome!):

  • Run git bisect to identify the exact commit that caused the breakage. In the future it would be nice for a system that automatically tracks down the breaking change and gently chides people in the breaking PR to run nixpkgs-review in the future.
  • If package A depends on B, and A fails to build due to B failing to build, it will open the issue on package A, not on package B. Not a huge deal, but could be made more precise. See here.

So far it’s been mostly a system for personal use, but I would love for it to be useful to others in the community!

1 Like