Suggestion to help maintainers with Nixpkgs' PRs bloat

Besides all the bugs / issues / rough edges we have in NixOS, I tend to think what’s even worse is the insane amount of issues & PRs open. I was wondering whether the community would be willing to run a GitHub bot that will aid with managing these.

Particularly, I’ve had been boiling up an idea for a bot that might help maintainers focus on PRs whose author show willingless for them to be merged while satisfying the reviewers.

Basically, the idea is to label each issue and PR with a label such as last responded: <role> where <role> can be:

  • Author
  • Reviewer
  • Maintainer

The idea came to my mind after I was browsing the open PRs in Nixpkgs and I saw many which were commented by a reviewer with formatting suggestions etc. and the authors just haven’t responded back, nor fixed the issues pointed out by the reviewer.

Running this bot should benefit the maintainers because it will enable them to filter out reviewed PRs with no respond from the authors - PRs not worth looking at since their authors don’t show a minimal willingless to cooperate; while other PRs - which are reviewed and resolved will be easily traceable.

What I find unfortunate with our situation is the fact that Nixpkgs has around 1500 open PRs (!) and many of which the authors have a strong willingless to satisfy the reviewers and they do eventually, but these PRs get missed from the eyes of those too few maintainers with merge rights.

I feel mistreated as a PR author because I never miss a review and I always respond back with the kindest approach towards resolution. But too often my PRs stay unmerged, even after an approval / review resolution.

BTW Is this one of those ideas which might fit more for the RFCs repo? I wouldn’t have mind propose it there but I’m not sure it would fit and I still didn’t fully get the concept of it.

7 Likes

This is somewhat similar to [RFC 0030] Formalize review workflow by timokau · Pull Request #30 · NixOS/rfcs · GitHub. I’d still like something like this, though I don’t currently have time to follow up on it myself.

Are you implying with the exclamation mark that these are a lot, or even too many? Because if you are, then I disagree, as we merge over 1500 PR’s a month…

1 Like

I also like the people checking and giving feedback to my PRs, but having a little bit more automation which would ease the reviewers’ life would definitely be better.
Because there is currently 1508 open PRs. 727 (~half) of them created before 2019-08-01 (and even 43 of them has their review approved).

I got the same answer @FRidh gave on the matrix channel when I asked about this since I didn’t get any comment on my PRs for some time (also partly because I am still rookie and they have a bad quality :grimacing: ).

I’m not implying that not enough PRs are merged. I’m implying that there are too many and it makes it unfeasable to distinguish between those worth attention and those not.

3 Likes

Some people (@Fridh, among others) are working on these issues, along the lines of what you are talking about.

Until we have such a solution in place I highly suggest pinging people with commit rights on your PRs, using the PRs already reviewed thread, asking for help in IRC, and just being persistent. I have found that if you get the attention of a committer they are usually happy to help.

Keep in mind sometimes threads just go stagnant because it slips a committers mind.

3 Likes

I think there are too many ready PR‘s for the people who can merge (not for the project), otherwise it wouldn‘t be that hard to get a reviewed PR merged. But I think the RFC for automatic merging might be a solution to this:

https://github.com/NixOS/rfcs/pull/50

In my opinion the ~1500 open PR‘s are less of a problem than the amount of work it takes to merge all the PR‘s… I don‘t think commiters just „can‘t find“ PR‘s ready to merge. it‘s rather that they don‘t have time. Or am I wrong on this?

2 Likes

“people who can merge”. Surely there are more than just guys with write access.

7 Likes

You are correct. :smile:

1 Like

Please excuse me and thanks a lot for your correction, I fixed it!

5 Likes

Yeah, we’re going for the merge feature in ofborg to help with this issue a lot.
Hopefully when I get some time to re-review the RFC we can see if we’ve reached a good agreement.

[RFC 0051] Mark stale nixpkgs issues by ryantm · Pull Request #51 · NixOS/rfcs · GitHub to add a triage bot for issue staleness was just accepted, so hopefully I can get that implemented soon and it will help a little.

I agree with @fridh that 1500 is not a lot when it is on the order of what we merge each month.

1 Like

Although 1500 isn’t a huge amount of PRs in a given month. Having to spend less time on non-impactful PRs would help with being able to give attention to more significant PRs.

The 19.09 release was particularly rough with all the ZHF PRs, then for @FRidh and myself, the python3.8 release caused a lot of churn in the python ecosystem.

2 Likes

@timokau I don’t see why should we implement this our selves from scratch within ofborg ([WIP] Labels by Ekleog · Pull Request #216 · NixOS/ofborg · GitHub). From a Google search and a GitHub’s Marketplace search, I’ve found these alternatives which might interest us:

Also for issues, there are some interesting apps:

1 Like

I think the suggestion itself can be very useful for those that are interested in going through PR’s as proposed here and having ofborg or another bot take care of that is I think a good idea.

That of course does lead to the question, would people use this to find PR’s? Speaking for myself, probably not, despite this seeming like a fair approach. Why not? Limited time and interest as well as politics.

Consider e.g. how I divide my time. Issues/PR’s I feel responsible for get most of my attention. These are related to packages I maintain (core Python) or core packages I broke (I don’t care much about leaf packages). Then, there is staging. After that, I devote time to people that I think contribute in such a good way that I want to support that, or because they help me out a lot (e.g. Jon and dotlambda with Python packages). Then there’s the easy fixes such as r-ryantm bot PR’s. After that, if there is time, I may have a look at other PR’s. But how, and in what way? Well, that depends on what I feel like doing. Remember, most of us are doing this in our free time, and so don’t want this to feel like “work” either.

@doronbehar in your case I know you also often take a maintainer role for packages, so I imagine the earlier mentioned merge-bot would be useful for you.

This I still experience as one of the major issues as Nixpkgs is growing. There is just so much noise on the tracker. All these “insignificant” leaf packages hide the more important issues.

2 Likes

So what you are saying is that the problem is mainly a lack of manpower rather then the lack of tooling / automation right? I guess that’s always right, I just hope to ease the maintainers’ life never the less.

BTW, I really wanted to reach the conference this year, in regard of this lack of manpower, was there any progress / new trusts made during the conference? I mean - adding new people to the trusted circuit of maintainers.

IIRC that rfc predates github apps. A custom implementation may still be worth it, its definitely more flexible. But at least as a proof of concept one of those apps would also do. Feel free to open a new RFC or to modify mine :slight_smile:

@FRidh you’re probably not the “typical” case here, since you maintain a whole sub-package set and likely have your hands full with that. I do think that there are some people who would use this.

1 Like

I don’t mean to question any maintainers/committers who aren’t sure if this will help since I’m not really in the loop, but I’ve done something similar-ish (different mechanism, but I think it accomplishes a similar end) on an old MUD I admin for. I’ll just describe what I did and how it worked out; feel free to skip reading.

During a period where a small number of us were working to wrangle a long backlog, one of the changes I made to our in-game issue tracker (which handles bugs/typos, ideas, and admin todos/projects) was to make it possible to assign any issue to feedback, which functionally accomplishes a few things:

  • The default admin view shows issues assigned to the admin, or to any abstract domains over which they have ownership/maintainership. Assigning to feedback yanks issues out of this view and helps shield admins from un-actionable issues.
  • Assigning the issue to feedback also pulls it out of the other main view–the triage queue. This starts saving small slices of time+effort for everyone trying to run triage.
  • It automatically posts a message to the issue notifying all participants that it’ll self-delete in N (42 :wink: ) days without feedback. This happens in 14-day phases (feedback → dismissed → deleted → completed), and there’s an automatic notification for all participants at each of the first 3 phases. Any new comment from the reporter returns it to the triage bin through the first 2 phases; an admin can easily intervene at any point in the first 3 phases.

I found that using an official process to flag an issue as soon as you take the action that puts the ball back in the reporter’s court structures everyone’s expectations and greatly reduces a few kinds of work around languishing issues that nibble away at everyone’s time and energy:

  • Multiple admins running triage regularly re-reviewing many issues to identify the ones that have gone “too long” without feedback.
  • Making subjective, individual calls about what counts as too long.
  • Manually nagging the reporter for more information one or more times until it feels fair/reasonable to dismiss.
  • Interpersonal friction that slowly builds up from having to:
    • prod people for feedback that they don’t give, even if you see them on doing something else
    • personally make a decision to dismiss their issue
    • receive a personal decision to dismiss your issue (causes the most trouble if steps/expectations/decisions aren’t well-communicated; communication quality more likely to sag when triage can’t keep up)
4 Likes

A full-cycle review support would need to integrate pretty tightly with ofBorg builds or something similar, and it looks like third-party CI pipelines are more pain than gain for Nixpkgs codebase…