I’ve started building a system that runs nixpkgs-review on each PR to nixpkgs on both x86_64-linux and aarch64-linux. It does have some cutoffs that exclude building certain PRs that cause mass rebuilds, but I believe it covers approximately 75% of all PRs currently.
I’m still working on the system, but you can see a live shot of the builds that have completed here: r-rmcgibbo’s gists · GitHub.
It’s currently not activated to actually comment on github, because I wanted to iron out the bugs in the orchestration and ask the community for feedback first.
@Sandro – I know specifically that you review a lot of PRs so I want to confirm that this would be helpful to you before pushing it.
Here’s for example the comment it would have posted on github for one of the 300 PRs that it has reviewed in the past 3 days:
Result of `nixpkgs-review pr 110223` at 7ab66f91 run on aarch64-linux [1](https://github.com/Mic92/nixpkgs-review)
<details>
<summary>1 package built:</summary>
<ul>
<li><a href="https://gist.github.com/83cc5ffd8d63c75638539d33d06be302">xfig</a></li></li>
</ul>
</details>
Please check them carefully before you add them. So far I have no feedback if they worked for anyone else. I try to keep them stable and not break them with every other commit but so far they have zero tests and every change could break something.
FYI: I’ve “turned on” this bot,and it’s now commenting on github. I’ve tried to put in place a few different checks so as to not upload comments to github if they’re completely duplicative (i.e. if someone else has posted a nixpkgs-review bot for x86_64-linux my bot will skip posting for that arch), but these checks are unlikely to be perfect.
Please reach out to me on over email, IRC, or any other mechanism if it’s bothering you or if there are any changes you’d like me to make to it.
I currently have it configured to attempt to build PRs that it believes it can finish in less than 2 hours of build time, and I’m working on adding a feature such that it can attempt to partially build PRs that are are larger, if they’re composed of many derivations.
Numerous improvements to @jtojnar’s nixpkgs-hammering have been landing, so the bot will be getting smarter with suggestions (although there’s lots of room for improvement of course).
Some bugs, areas for improvement and future plans:
I do receive some HTTP 422 errors from the github API when posting comments that I don’t understand.
There are some cases of truncated logs (cc @danieldk who pointed one out this morning to me and @Sandro who pointed out a few as well). I believe, but have not confirmed, that this is relates to timeouts (I have a 1 hour limit per drv in place). The uploaded logs and the “failure message” in the github comment really ought to be more helpful here.
When a build fails because one of its dependencies failed to build (e.g. because of a hash mismatch in the src), the bot often doesn’t upload useful logs showing the failure of the dependency. This needs to be fixed.
Thanks @danieldk and @mic92. Most of the builds are run in fresh spot VMs, so I don’t know how much monitoring would help (it’s not very actionable). I think the best bet is either to skip those builds or provision machines with larger disks.
Note: If anyone – especially core maintainers who file lots of PRs – would prefer to not be notified with a github comment when the build succeeds, just let me know. You can find me on IRC, twitter, or look up my email address.
It’s a 1-line change for me to make the bot refrain from commenting on PRs that you’ve filed unless the build actually fails.
(Other changes to reduce the amount of comment spam like integration with the Github Checks API are possible but more delicate.)