Steps towards even more PR automation

Development
11

Hi all,

First off I’d like to say I agree in spirit with most of this thread, and it’s
great to see others thinking about it. We need to lean heavily on automation and
“work smarter not harder” to keep up with the world and avoid burning our people
out.

On this front, I have a couple improvements I’d like to call out.

  1. With help from @ryantm, I’ve done work to refactor nixpkgs-update
    by separating out the updater “rewrite rules” from the nuts-and-bolts of
    cloning the repo, doing builds, interacting with Repology, sending a PR, etc.
    The goal here is to make it easy to add a new improvement/rewrite rule that
    gets run when versions upgrade.
    https://github.com/ryantm/nixpkgs-update/blob/321c8c59b2faa1a9ae89224fa125fe081dd56f71/src/Update.hs#L212-L222

  2. Using (1), we’ve added a rewrite rule that can quote
    meta.homepage for https://github.com/NixOS/rfcs/pull/45. Example here:
    thonny: 3.2.6 -> 3.2.7 by r-ryantm · Pull Request #82390 · NixOS/nixpkgs · GitHub
    This is a fairly simple proof-of-concept rewrite rule.

  3. Using (1), we’ve added a rewrite rule that is capable of updating
    Rust application packages automatically, like this:
    rust-bindgen: 0.53.1 -> 0.53.2 by r-ryantm · Pull Request #82384 · NixOS/nixpkgs · GitHub

  4. Using (1), we’ve added a rewrite rule that is capable of updating Golang
    application packages automatically, like this:
    antibody: 4.2.0 -> 4.3.1 by r-ryantm · Pull Request #82492 · NixOS/nixpkgs · GitHub
    For a PR example of adding a new rule, this is what added golang support:
    Automated updates for buildGoModule packages by bhipple · Pull Request #163 · ryantm/nixpkgs-update · GitHub

  5. With help from @mic92, nixpkgs-review is now capable of operating
    non-interactively with a --no-shell cmdline argument. This is a building
    block that can let us hook it into other bots like nixpkgs-update, or
    ad-hoc mass PR verification scripts, or other CI bots, and so on.
    Add --no-shell flag for non-interactive nixpkgs-review by bhipple · Pull Request #84 · Mic92/nixpkgs-review · GitHub

Aside from celebrating the concrete improvements already implemented-and-deployed,
the main reason to emphasize these is that they are building blocks for further
automation, so if you have ideas here it ought to be easier to implement them now!

A quick note on the philosophy of (1): this is not meant to replace treewide
PRs or CI checks. The purview of nixpkgs-update is still in updating
individual package versions, but we often have some “not absolutely urgent but
still nice to have” improvements in the works that we’d like to slowly sweep
through NixPkgs, like RFC 45, or things like removing unnecessary rec, or
using fetchFromGitHub for release tarballs instead of fetchzip, etc. One way to
accomplish these is by having someone do a treewide PR with an ad-hoc
sed/hnix/editor-macro generating the diff, but these can be hard to code review
and create mass rebuilds + merge conflicts, and while appropriate for relatively
important/critical changes are less compelling for small cleanups. Moreover they
don’t keep applying over time.

The above gives that hook to have the updater bot slowly drag in small
improvements over time, spread out as packages themselves get updates and
reviews/testing as part of their normal lifecycle.

Some comments on the above thread:

  1. Based on experience, I’d advocate against having auto-merge bots. Potential
    security vulnerabilities and complexity aside, we’ve had multiple issues
    where the nixpkgs-update bot generated garbage that still passed CI, and was
    only saved by human code review (huge thanks to @jonringer and @marsam for
    reviewing+merging a ton of my PRs, btw). I’d argue we should focus the bulk
    of our effort on PR generation / PR validation with CI, at least initially,
    because it’s lower risk and higher RoI.

  2. The ability to farm out temporary OfBorg capacity would be fantastic. Much of
    the above was an output of me initially writing my own scripts for
    Rust Application Migration to fetchCargoTarball Implementation · Issue #79975 · NixOS/nixpkgs · GitHub. I burnt out the OfBorg bot by
    going too fast, then learned from @grahamc how nixpkgs-update does rate
    limiting and implemented it, which got me thinking about updating
    nixpkgs-update to make these kinds of updates easier in the future. Once I
    added rate limiting, the script spent most of its time sleeping with 72 cores
    idle waiting for OfBorg to catch up. It would have been nice to connect that
    capacity temporarily for PR builds. Capacity on the AWS spot market is
    extremely cheap, but it requires OfBorg to support a dynamic build farm
    gracefully.

    I agree these should never make their way into the official binary cache for
    security reasons, of course.

  3. At the end of the day our humans are our most valuable resource, and probably
    the greatest risk to our ecosystem is that we burn out key maintainers by
    having them do too much repetitive, grindy work that should have been
    automated. To the extent that we can make their lives easier with heavy PR
    CI validation / nix derivation sanitizing we should prioritize it! No novel
    point here; just reiterating that I think improvements suggested in this
    thread are valuable and important for the long-term health of NixPkgs.

6 Likes