Weekly leaderboards

My only concern here is that rewarding people in this fashion is optimizing for the wrong thing. Rewarding for “approvals” encourages people to review the easy PRs and skip the hard ones. Rewarding for “requests for changes” encourages people to nitpick PRs for subjective stylistic things. Rewarding for merges is practically the opposite, it encourages people to merge PRs that otherwise could have benefitted from some feedback.

That said, I don’t have any ideas for how to fix this while still retaining the notion of rewarding people, so maybe it’s worth doing this anyway and just keeping an eye on things to make sure this increases contribution overall instead of just redirecting existing contribution. If this produces more contribution overall it may still be a win, with easy PRs getting reviewed and merged faster (which makes everyone happy) without necessarily slowing down hard PRs.

5 Likes

The real question is whether we improve or degrade the zero-hydra-failures effort. Can anyone determine if weekly sprints to fix PRs has a positive impact on hydra?

2 Likes

I find it fun since I don’t think anyone takes it at face value. And I don’t think anyone is rewarded according to his position either ?
For the weekly report, I would prefer to see global metrics: #Merged PR, #closed issues
, a list of new packages (if that can be automated )

1 Like

Approvals

If I sleepwalk into top-11 the week when I was at a conference, maybe this is not such a good metric!

Maybe a separate top for non-committer approvals, as the non-merge reviews by committers are often a separate story?

My only concern here is that rewarding people in this fashion is optimizing for the wrong thing. Rewarding for “approvals” encourages people to review the easy PRs and skip the hard ones. Rewarding for “requests for changes” encourages people to nitpick PRs for subjective stylistic things. Rewarding for merges is practically the opposite, it encourages people to merge PRs that otherwise could have benefitted from some feedback.

Hopefully, rewarding all three brings incentives back into balance.

That said, I don’t have any ideas for how to fix this while still retaining the notion of rewarding people, so maybe it’s worth doing this anyway and just keeping an eye on things to make sure this increases contribution overall instead of just redirecting existing contribution. If this produces more contribution overall it may still be a win, with easy PRs getting reviewed and merged faster (which makes everyone happy) without necessarily slowing down hard PRs.

Really hard PRs are the somewhat-controversial-tradeoff ones. These won’t be a happy story, however we gamify the easy ones.

I don’t expect such a leaderboard to redirect the type of contributions; if it can be used as a source of positive glow for those who spend effort (it looks like everyone who spends significant effort in a specific week gets into the list with the current cutoff — maybe this should be the rule of maintenance of thresholds in the future, instead of fixed top-10), it could be nice.

Having a leaderboard for this would be great. Way too many failures right now :disappointed:

I think that directly depends on the rigor or the reviwers. If people at least follow the instructions in PRs already reviewed, i.e. actually run nix-review and fix the issues it reports, even if it is just marking packages as broken, it is a net positive. If they don’t, merging new PRs will probably break things.

Regarding the concerns of incentivizing wrong things: Its always a risk with gamification. However I think this is harmless here. I see nothing wrong with nitpicks as long as they have at least a little merit. Nothing wrong with a high standard. Gaming merges/approvals without proper review will get noticed by the community in the long run.

There hasn’t been much incentive for approvals until now: committers could just merge immediately, non-committers would get frustrated because their review doesn’t have any effect. Maybe PRs already reviewed will change that. You can already see that in @tomberek’s efforts for example :slight_smile:

I have thought about counting a merge of a PR that doesn’t have an approval by anybody else as an approval. Some committers use the approval feature before merging, others don’t so I think that would be a fairer question.

That would currently drown out non-committer approvals a bit, but maybe that will change. If people start using the distinction between “reviewer” and “merger” more, just listing the merges for committers doesn’t seem fair either.

Here are some more stats, with new PRs included and merge-without-approval counted as approval. Cutoff at 3 this time, to save space and since it is more unique than 10.

Approvals

  1. xeji (100)
  2. matthiasbeyer (63)
  3. timokau (31)
  4. tomberek (25)
  5. worldofpeace (21)

Requests for changes

  1. timokau (17)
  2. aanderse (11)
  3. worldofpeace (7)
  4. xeji (7)
  5. matthiasbeyer (6)

Merges

  1. xeji (112)
  2. timokau (32)
  3. ryantm (22)
  4. worldofpeace (22)
  5. markuskowa (17)

New PRs

  1. dtzWill (21)
  2. matthewbauer (8)
  3. aanderse (7)
  4. romildo (7)
  5. vdemeester (7)

And here is the script I used in case someone wants to tinker. Be warned: its neither pretty nor commentated, but also not very complicated.

Another iteration. This time merges without any prior approval by someone else only count as positive reviews. Merges with prior approval are counted separately. Ranks are now grouped.

Positive reviews

  1. xeji (101)
  2. matthiasbeyer (67)
  3. timokau (31)
  4. tomberek (27)
  5. worldofpeace (21)
  6. pSub (17)
  7. globin (15)
  8. markuskowa, vcunat (10)
  9. Ma27, Infinisil (9)
  10. dotlambda, zimbatm, matthewbauer (7)
  11. 7c6f434c (6)
  12. basvandijk, vdemeester, flokli, aanderse, veprbl, nlewo, dywedir (5)
  13. gebner, thoughtpolice (4)
  14. marsam, LnL7, jtojnar, c0bw3b, andir, etu, aneeshusa, vbgl, grahamc (3)
  15. FRidh, lheckemann, Ericson2314, tadeokondrak, Mic92, primeos, cillianderoiste (2)
  16. bendlas, domenkozar, asymmetric, ryantm, apeyroux, costrouc, andrewrk, michaelpj, kalbasit, NeQuissimus, samueldr, svanderburg, manveru, peterhoeg, WilliButz, romildo, falsifian, bjornfor, hedning, Ekleog, offlinehacker, symphorien, dtzWill, fadenb, hrdinka, ehmry, cpages, peti (1)

Requests for changes

  1. timokau (17)
  2. aanderse (12)
  3. dotlambda (9)
  4. matthiasbeyer (8)
  5. worldofpeace, xeji (7)
  6. kalbasit, markuskowa (3)
  7. flokli, jtojnar, tomberek, ryantm, basvandijk, peti (2)
  8. etu, andrew-d, vbgl, bricewge, pSub, andir, symphorien, Infinisil (1)

Merges of already reviewed PRs

  1. ryantm (19)
  2. xeji (15)
  3. worldofpeace (5)
  4. flokli, markuskowa, timokau (3)
  5. dotlambda, pSub, lheckemann, zimbatm (2)
  6. hedning, romildo, 7c6f434c, jtojnar, andir, peterhoeg, matthewbauer, samueldr, dtzWill, benley, veprbl, globin (1)

New PRs

  1. dtzWill (21)
  2. aanderse, matthewbauer (8)
  3. peterhoeg, basvandijk, matthiasbeyer, romildo, vdemeester (7)
  4. xrelkd, timokau (6)
  5. manveru, teto, worldofpeace, delroth (5)
  6. dhl (4)
  7. volth, markuskowa, aneeshusa, veprbl, ctheune, jabranham, vbgl, pSub, gebner (3)
  8. lsix, lihop, JohnAZoidberg, lilyball, Ma27, FRidh, marius851000, Vskilet, jtojnar, gnidorah, thefloweringash, Mic92, ajs124, dywedir, bricewge, WilliButz, magnetophon, Yarny0, megheaiulian, Taneb, illegalprime, AesaKamar (2)
  9. exarkun, Zimmi48, divi255, couchemar, rasendubi, domenkozar, ThomasMader, NathanielHill, marsam, zookatron, lopsided98, lightbulbjim, jacereda, CohenCyril, hedning, dnaq, andrew-d, mogorman, jonafato, melazyk, bendlas, adrianparvino, c0bw3b, xeji, taku0, suhr, willtim, yrashk, arianvp, minijackson, nomeata, thedavidmeister, fgaz, P-E-Meunier, PombeirP, NickHu, damien-biasotto, mmlb, feijoas, costrouc, justinwoo, jbgi, 239, Izorkin, loewenheim, ckauhaus, nicknovitski, primeos, betaboon, mboes, flokli, talyz, averelld, cdepillabout, thoughtpolice, Thesola10, nh2, gilligan, risicle, Gerschtli, ilikeavocadoes, rhokue, nyanloutre, scalavision, zimbatm, ck3d, Profpatsch, Kaali, aristaeus, marler8997, bb010g, OmnipotentEntity, benley, eyJhb, bbigras, bkchr, witkamp, dotlambda, petrkr, terlar, marzipankaiser, tadeokondrak, eri451, tg-x, midchildan, d10n (1)

(Edit: Fixed my numbers)

1 Like

Publishing leaderboards sounds like a recipe for burnout to me.

3 Likes

A universal incentive with praise and recognition can be desirable.

But a leaderboard can turn into something in which you compete with yourself and others to work yourself more. Similar relationships have been observed with the contributions chart on GitHub.

Care to elaborate?

Something like a “contributor of the month” (or “…week”) could be interesting.
A sticky thread where one can nominate somebody else and list reasons (like “he/she published a cool new tool” , “…made an awesome RFC”, etc.) for the nomination. At the end of the week/month the person with the most likes gets the title for a month. If it is possible to make this visible on Github, IRC and here, it could be something that makes people try a slight bit harder to do things the community likes and have more fun doing it :slight_smile:

I don’t think this would be taking away a lots of time and at the same time have some features like:

  1. possibly bringing a bit a relaxed atmosphere (idk if this is good or necessary, but I think having a bit of a “tradition” is a good thing for the community)
  2. people effectively talk about what is happening (stats say absolutely nothing about what’s happening!)
  3. make people maybe a tiny bit more likely to share their tools, help in a useful way, etc.
  4. Maybe you will discover that one tool you need just right now in a nomination? :smirk:
  5. It’s fun :grin:

That may sound against the idea of a weekly leaderboard, but I still think such a leaderboard is not a bad idea itsself.
It isn’t the best indicator to show who’s “the best” or something like that, but it lists people who invest their time and earn to be honored!
Thanks @timokau for this! :relaxed:

1 Like

Rust does this in an interesting way without turning it into a numbers game: https://thanks.rust-lang.org/. Though given that NixOS doesn’t do releases nearly as often as Rust you’d have to figure out a better metric to track by.

1 Like

I don’t really like the rust approach since they moved to the “thanks” page. Putting it on a separate page kills most of the value imho, since realistically not many people will actually look at it. I’d much rather highlight a smaller selection of people highly visible.

For comparison, here is IPFS (see “Contributors by the numbers”).

1 Like

I agree with you there, I think the most valuable part is that it’s automated. They do a similar thing for Rust Weekly with their quote/crate of the week, you could potentially do something similar for NixOS Weekly.

1 Like

Here’s the latest:

Positive reviews

  1. FRidh (84)
  2. c0bw3b (44)
  3. tomberek (24)
  4. dtzWill (23)
  5. matthiasbeyer, worldofpeace (18)
  6. matthewbauer (15)
  7. Lassulus, aanderse (11)
  8. marsam, jtojnar, NeQuissimus (9)
  9. JohnAZoidberg, Infinisil, peterhoeg (8)
  10. joachifm (7)
  11. teto (6)
  12. gebner, 7c6f434c, fpletz, srhb, Ma27, dotlambda (5)
  13. alyssais, andir, timokau, peti, kalbasit, grahamc, vbgl (4)
  14. primeos, etu, pSub, endgame, dywedir, veprbl, Mic92, c00w, flokli (3)
  15. vcunat, angristan, zimbatm, ryantm, ttuegel, markuskowa, samueldr (2)
  16. LnL7, bjornfor, Ericson2314, yegortimoshenko, thefloweringash, edef1c, rnhmjoj, jwiegley, volth, basvandijk, tilpner, nh2, Profpatsch, MP2E, romildo, hedning (1)

Requests for changes

  1. aanderse (9)
  2. dotlambda (7)
  3. kalbasit (5)
  4. c0bw3b, JohnAZoidberg, jtojnar (4)
  5. matthiasbeyer, tomberek, FRidh, marsam (3)
  6. samueldr, ryantm, Infinisil, peti, srhb, Profpatsch (1)

Merges of already reviewed PRs

  1. ryantm (9)
  2. FRidh, dtzWill (7)
  3. c0bw3b, timokau (6)
  4. peterhoeg (3)
  5. worldofpeace, Lassulus, dotlambda, marsam (2)
  6. 7c6f434c, flokli, gebner, lheckemann, Profpatsch, srhb, aanderse, teto, Mic92, matthewbauer, dywedir, NeQuissimus (1)

New PRs

  1. dtzWill (33)
  2. manveru (17)
  3. marsam (12)
  4. peterhoeg (8)
  5. matthewbauer, angristan (7)
  6. nyanloutre, JohnAZoidberg, risicle, Infinisil, xrelkd (6)
  7. dotlambda, rvolosatovs, c0bw3b (5)
  8. thefloweringash, etu, volth, danieldk, timokau, magnetophon, lheckemann (4)
  9. primeos, elseym, marenz2569, basvandijk, jtojnar, WilliButz, eadwu, bbigras, ambrop72, andir, Tomahna (3)
  10. flokli, callahad, seqizz, romildo, gebner, bignaux, FRidh, minijackson, worldofpeace, wucke13, joachifm, herrwiese, LinArcX, artemist, exarkun, johanot, monsieurp, lasandell, matthiasbeyer, mtrsk, azazel75, kierdavis, arianvp (2)
  11. cawilliamson, rydnr, juaningan, burke, sondr3, asymmetric, aethelz, kwohlfahrt, aanderse, mwilsoninsight, kmicklas, dasJ, klntsky, anpryl, bachp, samdoshi, dawidsowa, vcunat, SRGOM, glesica, chris-martin, booxter, edef1c, Gonzih, benley, 4z3, nbardiuk, xzfc, nshalman, thoughtpolice, winpat, lblasc, cohei, alyssais, Shados, endgame, vdemeester, spacefrogg, malob, anton-dessiatov, 7c6f434c, c0deaddict, jraygauthier, vbgl, Profpatsch, jabranham, BenBals, averelld, schmittlauch, kisonecat, jonringer, kevingriffin, cdepillabout, tex, barakber, uHOOCCOOHu, colemickens, telotortium, ahiaao, dywedir, bhipple, yrashk, stefano-m, thoferon, livnev, neilmayhew, Ma27, srhb, ajs124, Amar1729, sorki, samsonan-it, bgamari, dfordivam, symphorien, nmattia, Luis-Hebendanz, rick68 (1)
4 Likes

I like stats :slight_smile:

@timokau could you build a static website with a CI to update weekly? So if one is interested in the leaderboard can look there.

But that might not be the actual intention since you said it should be visible. So NixOS weekly would be the right place. Maybe just a short (top 5?) and link to static site with top 100? (top 10 visible and more link)

Why have you separated positive reviews and change requests? Why not combine to “PR Reviews”?

Why no stats for New Issues? Is it discouraged to open issues? :smiley:
Thats what i do most of the time and it’s also a lot of effort when you debug to provide helpful information! But it’s easier to open a lot of issues than opening a lot of PRs, so separating them makes sense.

I would like to see most active contributors and new contributors in the NixOS release notes. Just a list sorted after most contributions or newest. So you can see that you are part of the effort.

No, I’m not a web developer so I don’t have either access to servers or any experience in that regard. Its not worth it for me. Feel free to do with https://github.com/timokau/nixpkgs-stats whatever you want :slight_smile:

Yes, that would be the idea. A more link is not necessarily essential in my opinion: currently top 10 would easily cover any significant contributions.

I didn’t really think about it much. They do seem a bit different to me, but for the sake of brevity it may be a good idea to merge the categories.

I wouldn’t say it is discouraged. But what I want to encourage here is reducing the backlog. As long as we have more solutions (PRs) than we can deal with, I wouldn’t explicitly encourage creating new problems (issues). I’m not saying that the issue creator is actually creating problems, but gamifying this aspect could lead to more nitpicky issues just for the stats sake.

If someone is sufficiently annoyed by an issue, there is no need for any more incentive to report it.

That would be a great idea in addition to more frequent weeklies!

1 Like

There is so much activity in nixpkgs that it’s not humanly possible to follow all of it anymore. Having different summarized views into that activity is quite useful in that regard.

I like the term “pulse” that GitHub uses: https://github.com/NixOS/nixpkgs/pulse . The report is not somewhere where I would tend to go, and the data is not tailored to our community though.

The nixos-weekly mailing list is the best source for summarized content for me (thanks @domenkozar!). It really should be on the NixOS homepage. It could also be augmented with automated reports as long as they are useful. Switching the reports once in a while is probably useful as well.

3 Likes

Yeah I feel like the pulse information is not all that interesting because it is not personal.

Here’s a different format, what do you think (kind of surprising to me that there was no comment with multiple reactions):

Community spotlight

The holy trinity for contribution to nixpkgs are direct code contribution, reviews of those contributions and merges of finished pull requests. In the past week

  • 41 pull request were opened by 120 contributors. @marsam alone opened 12 of those
  • 251 pull request were reviewed by 57 contributors. @c0bw3b alone reviewed 49 of those
  • 19 pull request were merged by 20 contributors. @c0bw3b alone merged 7 of those

The most reacted to comment was this one.

You can help! You do not need any special permissions to propose or review changes. Check out this discourse thread if you want to get started with reviews.

1 Like