In a previous post, I announced the first iteration of the “r-rmcgibbo” bot, which runs nixpkgs-review against a large proportion of PRs to nixpkgs.
I wanted to give a short update.
The bot has been running (mostly continuously) over the last month. I have been making some updates to its code base and methodology, and I think it’s starting to stabilize. According to my logs, it’s been run on about 4,500 nixpkgs PRs for x86_64-linux and 4,200 nixpkgs PRs for aarch64-linux.
As currently configured, it runs a fork (many of the changes are described here) of nixpkgs-review that adds a few features, and runs nixpkgs-hammering to provide “suggestions”.
Although the vanilla version of nixpkgs-review tries to build every changed nixpkgs attribute in a PR, you might have noticed that the r-rmcgibbo bot sometimes says that it skipped certain attrs because of “time constraints”. This is based on a calculation that tries to find a subset of the modified build graph that can be built within a cutoff (currently set at 1.5 hours) based on build time estimates for each package that were determined based on hydra’s logs, and are now being refined based on r-rmcgibbo’s own logs. With this feature, the r-rmcgibbo bot can be run against PRs to all branches, including staging. (However, sometimes the staging jobs on hydra have are sufficiently backed up that building any staging package requires re-building stdenv, in which case the build is usually predicted to take longer than 1.5 hours and is skipped.)
The bot runs on AWS infrastructure. It has a front-end process that watches Github for new PRs and a back-end that builds PRs, connected via a queue. The backends run on fairly small machine types. These machines are not always capable of building every PR; the bot is programmed to detect OOMs, timeouts, and disk-full scenarios, and cancels its github comments when such an event is detected. All in, the cost to operate it is ~$40-50 USD / month.
My priorities going forward are:
- Open-source all of the code (auditing to make sure I haven’t left my AWS credentials in there anywhere, right now the code and the AWS infrastructure are a bit too intertwined).
- Upstream the appropriate features from my fork into Mic92’s nixpkgs-review itself.
- Discuss with nixpkgs stakeholders if and how this idea could be moved into OfBorg or otherwise run in a community-managed way, rather than just by me.
- Discuss with nixpkgs stakeholders how the results of these runs could be shown using the Checks API (green check mark, etc) rather than as comment.
- I did make a proof-of-concept Github Action that runs nixpkgs-hammering and posts suggestions in-line in the Files tab.
- Any other community priorities based on feedback.
I know that bot-generated comments are not the ideal way to present these build results. They create an email and a notification to all subscribers on the PR, which can be excessive, especially for prolific contributors. Hopefully future integration with OfBorg and/or the Github Checks API can replace the comments and provide a better experience in the future. In the short term, I want to mention the attempts I have made to address these concerns within the limitations of the “comment-posting model”:
- Rather than posting two comments (one for x86_64-linux and one for aarch64-linux) the bot will edit its own prior comment to consolidate both results into a single comment.
- If the PR changes only a single nixpkgs attr, the bot runs the build but does not upload the comment, because OfBorg will have already built these PRs completely itself.
- If a previous github user has already commented with
nixpkgs-reviewresults for a particular platform, the bot will not duplicatively post results for that platform. - If the bot detects various internal errors that might have occurred, most commonly a out-of-memory condition that caused a spurious build failure, it will not post any comment.
- (If the comments are particularly bothering you, you can always reach out to me here or on IRC and I can add you to a “quiet list” so your PRs will not receive confirmation comments in the case that all packages built successfully.)
The output of the bot is often very useful (even though there are sometimes false negatives) and building a reasonable set of dependencies is really the next step beyond/for ofborg.