Weekly leaderboards

#1

I’m playing with the idea of generating “leaderboards” of nixpkgs contribution to give credit where credit is due, help notice new active members and add a little gamification :slight_smile: Maybe to be included in nixos-weekly or posted on discourse?

Anyway, here is some data for the last week (top 10 plus those that have an equal number to no. 10; ordering of people with the same number is arbitrary). What do you think?

Approvals

  1. matthiasbeyer (55)
  2. tomberek (21)
  3. worldofpeace (12)
  4. timokau (8)
  5. vcunat (7)
  6. pSub (6)
  7. vdemeester (5)
  8. Ma27 (4)
  9. 7c6f434c (4)
  10. xeji (4)
  11. aanderse (4)

Requests for changes

  1. timokau (17)
  2. aanderse (8)
  3. xeji (7)
  4. matthiasbeyer (6)
  5. worldofpeace (5)
  6. flokli (3)
  7. kalbasit (3)
  8. markuskowa (3)
  9. ryantm (2)
  10. jtojnar (2)
  11. basvandijk (2)
  12. dotlambda (2)
  13. peti (2)

Merges

  1. xeji (108)
  2. timokau (32)
  3. ryantm (28)
  4. markuskowa (22)
  5. worldofpeace (21)
  6. globin (16)
  7. pSub (13)
  8. Infinisil (10)
  9. flokli (8)
  10. zimbatm (8)
  11. dotlambda (8)

Impressive work @xeji and @musicmatze!

1 Like

#2

My only concern here is that rewarding people in this fashion is optimizing for the wrong thing. Rewarding for “approvals” encourages people to review the easy PRs and skip the hard ones. Rewarding for “requests for changes” encourages people to nitpick PRs for subjective stylistic things. Rewarding for merges is practically the opposite, it encourages people to merge PRs that otherwise could have benefitted from some feedback.

That said, I don’t have any ideas for how to fix this while still retaining the notion of rewarding people, so maybe it’s worth doing this anyway and just keeping an eye on things to make sure this increases contribution overall instead of just redirecting existing contribution. If this produces more contribution overall it may still be a win, with easy PRs getting reviewed and merged faster (which makes everyone happy) without necessarily slowing down hard PRs.

2 Likes

#3

The real question is whether we improve or degrade the zero-hydra-failures effort. Can anyone determine if weekly sprints to fix PRs has a positive impact on hydra?

2 Likes

#4

I find it fun since I don’t think anyone takes it at face value. And I don’t think anyone is rewarded according to his position either ?
For the weekly report, I would prefer to see global metrics: #Merged PR, #closed issues
, a list of new packages (if that can be automated )

0 Likes

#5

Approvals

If I sleepwalk into top-11 the week when I was at a conference, maybe this is not such a good metric!

Maybe a separate top for non-committer approvals, as the non-merge reviews by committers are often a separate story?

0 Likes

#6

My only concern here is that rewarding people in this fashion is optimizing for the wrong thing. Rewarding for “approvals” encourages people to review the easy PRs and skip the hard ones. Rewarding for “requests for changes” encourages people to nitpick PRs for subjective stylistic things. Rewarding for merges is practically the opposite, it encourages people to merge PRs that otherwise could have benefitted from some feedback.

Hopefully, rewarding all three brings incentives back into balance.

That said, I don’t have any ideas for how to fix this while still retaining the notion of rewarding people, so maybe it’s worth doing this anyway and just keeping an eye on things to make sure this increases contribution overall instead of just redirecting existing contribution. If this produces more contribution overall it may still be a win, with easy PRs getting reviewed and merged faster (which makes everyone happy) without necessarily slowing down hard PRs.

Really hard PRs are the somewhat-controversial-tradeoff ones. These won’t be a happy story, however we gamify the easy ones.

I don’t expect such a leaderboard to redirect the type of contributions; if it can be used as a source of positive glow for those who spend effort (it looks like everyone who spends significant effort in a specific week gets into the list with the current cutoff — maybe this should be the rule of maintenance of thresholds in the future, instead of fixed top-10), it could be nice.

0 Likes

#7

Having a leaderboard for this would be great. Way too many failures right now :disappointed:

0 Likes

#8

I think that directly depends on the rigor or the reviwers. If people at least follow the instructions in PRs already reviewed, i.e. actually run nix-review and fix the issues it reports, even if it is just marking packages as broken, it is a net positive. If they don’t, merging new PRs will probably break things.

Regarding the concerns of incentivizing wrong things: Its always a risk with gamification. However I think this is harmless here. I see nothing wrong with nitpicks as long as they have at least a little merit. Nothing wrong with a high standard. Gaming merges/approvals without proper review will get noticed by the community in the long run.

0 Likes

#9

There hasn’t been much incentive for approvals until now: committers could just merge immediately, non-committers would get frustrated because their review doesn’t have any effect. Maybe PRs already reviewed will change that. You can already see that in @tomberek’s efforts for example :slight_smile:

I have thought about counting a merge of a PR that doesn’t have an approval by anybody else as an approval. Some committers use the approval feature before merging, others don’t so I think that would be a fairer question.

That would currently drown out non-committer approvals a bit, but maybe that will change. If people start using the distinction between “reviewer” and “merger” more, just listing the merges for committers doesn’t seem fair either.

Here are some more stats, with new PRs included and merge-without-approval counted as approval. Cutoff at 3 this time, to save space and since it is more unique than 10.

Approvals

  1. xeji (100)
  2. matthiasbeyer (63)
  3. timokau (31)
  4. tomberek (25)
  5. worldofpeace (21)

Requests for changes

  1. timokau (17)
  2. aanderse (11)
  3. worldofpeace (7)
  4. xeji (7)
  5. matthiasbeyer (6)

Merges

  1. xeji (112)
  2. timokau (32)
  3. ryantm (22)
  4. worldofpeace (22)
  5. markuskowa (17)

New PRs

  1. dtzWill (21)
  2. matthewbauer (8)
  3. aanderse (7)
  4. romildo (7)
  5. vdemeester (7)

And here is the script I used in case someone wants to tinker. Be warned: its neither pretty nor commentated, but also not very complicated.

0 Likes

#10

Another iteration. This time merges without any prior approval by someone else only count as positive reviews. Merges with prior approval are counted separately. Ranks are now grouped.

Positive reviews

  1. xeji (101)
  2. matthiasbeyer (67)
  3. timokau (31)
  4. tomberek (27)
  5. worldofpeace (21)
  6. pSub (17)
  7. globin (15)
  8. markuskowa, vcunat (10)
  9. Ma27, Infinisil (9)
  10. dotlambda, zimbatm, matthewbauer (7)
  11. 7c6f434c (6)
  12. basvandijk, vdemeester, flokli, aanderse, veprbl, nlewo, dywedir (5)
  13. gebner, thoughtpolice (4)
  14. marsam, LnL7, jtojnar, c0bw3b, andir, etu, aneeshusa, vbgl, grahamc (3)
  15. FRidh, lheckemann, Ericson2314, tadeokondrak, Mic92, primeos, cillianderoiste (2)
  16. bendlas, domenkozar, asymmetric, ryantm, apeyroux, costrouc, andrewrk, michaelpj, kalbasit, NeQuissimus, samueldr, svanderburg, manveru, peterhoeg, WilliButz, romildo, falsifian, bjornfor, hedning, Ekleog, offlinehacker, symphorien, dtzWill, fadenb, hrdinka, ehmry, cpages, peti (1)

Requests for changes

  1. timokau (17)
  2. aanderse (12)
  3. dotlambda (9)
  4. matthiasbeyer (8)
  5. worldofpeace, xeji (7)
  6. kalbasit, markuskowa (3)
  7. flokli, jtojnar, tomberek, ryantm, basvandijk, peti (2)
  8. etu, andrew-d, vbgl, bricewge, pSub, andir, symphorien, Infinisil (1)

Merges of already reviewed PRs

  1. ryantm (19)
  2. xeji (15)
  3. worldofpeace (5)
  4. flokli, markuskowa, timokau (3)
  5. dotlambda, pSub, lheckemann, zimbatm (2)
  6. hedning, romildo, 7c6f434c, jtojnar, andir, peterhoeg, matthewbauer, samueldr, dtzWill, benley, veprbl, globin (1)

New PRs

  1. dtzWill (21)
  2. aanderse, matthewbauer (8)
  3. peterhoeg, basvandijk, matthiasbeyer, romildo, vdemeester (7)
  4. xrelkd, timokau (6)
  5. manveru, teto, worldofpeace, delroth (5)
  6. dhl (4)
  7. volth, markuskowa, aneeshusa, veprbl, ctheune, jabranham, vbgl, pSub, gebner (3)
  8. lsix, lihop, JohnAZoidberg, lilyball, Ma27, FRidh, marius851000, Vskilet, jtojnar, gnidorah, thefloweringash, Mic92, ajs124, dywedir, bricewge, WilliButz, magnetophon, Yarny0, megheaiulian, Taneb, illegalprime, AesaKamar (2)
  9. exarkun, Zimmi48, divi255, couchemar, rasendubi, domenkozar, ThomasMader, NathanielHill, marsam, zookatron, lopsided98, lightbulbjim, jacereda, CohenCyril, hedning, dnaq, andrew-d, mogorman, jonafato, melazyk, bendlas, adrianparvino, c0bw3b, xeji, taku0, suhr, willtim, yrashk, arianvp, minijackson, nomeata, thedavidmeister, fgaz, P-E-Meunier, PombeirP, NickHu, damien-biasotto, mmlb, feijoas, costrouc, justinwoo, jbgi, 239, Izorkin, loewenheim, ckauhaus, nicknovitski, primeos, betaboon, mboes, flokli, talyz, averelld, cdepillabout, thoughtpolice, Thesola10, nh2, gilligan, risicle, Gerschtli, ilikeavocadoes, rhokue, nyanloutre, scalavision, zimbatm, ck3d, Profpatsch, Kaali, aristaeus, marler8997, bb010g, OmnipotentEntity, benley, eyJhb, bbigras, bkchr, witkamp, dotlambda, petrkr, terlar, marzipankaiser, tadeokondrak, eri451, tg-x, midchildan, d10n (1)

(Edit: Fixed my numbers)

0 Likes

#11

Publishing leaderboards sounds like a recipe for burnout to me.

2 Likes

#12

A universal incentive with praise and recognition can be desirable.

But a leaderboard can turn into something in which you compete with yourself and others to work yourself more. Similar relationships have been observed with the contributions chart on GitHub.

Care to elaborate?

0 Likes

#13

Something like a “contributor of the month” (or “…week”) could be interesting.
A sticky thread where one can nominate somebody else and list reasons (like “he/she published a cool new tool” , “…made an awesome RFC”, etc.) for the nomination. At the end of the week/month the person with the most likes gets the title for a month. If it is possible to make this visible on Github, IRC and here, it could be something that makes people try a slight bit harder to do things the community likes and have more fun doing it :slight_smile:

I don’t think this would be taking away a lots of time and at the same time have some features like:

  1. possibly bringing a bit a relaxed atmosphere (idk if this is good or necessary, but I think having a bit of a “tradition” is a good thing for the community)
  2. people effectively talk about what is happening (stats say absolutely nothing about what’s happening!)
  3. make people maybe a tiny bit more likely to share their tools, help in a useful way, etc.
  4. Maybe you will discover that one tool you need just right now in a nomination? :smirk:
  5. It’s fun :grin:

That may sound against the idea of a weekly leaderboard, but I still think such a leaderboard is not a bad idea itsself.
It isn’t the best indicator to show who’s “the best” or something like that, but it lists people who invest their time and earn to be honored!
Thanks @timokau for this! :relaxed:

1 Like

#14

Rust does this in an interesting way without turning it into a numbers game: https://thanks.rust-lang.org/. Though given that NixOS doesn’t do releases nearly as often as Rust you’d have to figure out a better metric to track by.

0 Likes

#15

I don’t really like the rust approach since they moved to the “thanks” page. Putting it on a separate page kills most of the value imho, since realistically not many people will actually look at it. I’d much rather highlight a smaller selection of people highly visible.

For comparison, here is IPFS (see “Contributors by the numbers”).

0 Likes

#16

I agree with you there, I think the most valuable part is that it’s automated. They do a similar thing for Rust Weekly with their quote/crate of the week, you could potentially do something similar for NixOS Weekly.

0 Likes