I like this distinction between Teams, that are imbued with role and responsibility, and Working Groups that are just an assembly of people with a shared interest. In that model, we would want to favour Working Groups over Teams generally. Unless we see the group would function better with some additional roles and responsibilities (and a WG could then be converted to a Team).
If we can agree on that, then a few things follow:
To encourage WGs, it should be low friction to create and disband them. Right now, this requires creating a Matrix channel and then getting it added to the roster. Or ask an admin to create a Discourse group or GitHub team. But documenting those steps would already be pretty good. Maybe you have some ideas in that area.
It would also make sense to go back to existing teams and see if the team roles and responsibilities are clearly outlined. Let me know if any team stands out to you.
It’s probably hard to do a good job being part of too many teams. I would favour setting a cap to 2 or 3 teams per member. This would also help make more room for participation.
I think the nixpkgs-architecture is a working group instead of a team?
The Nixpkgs Architecture Team depends on the RFC process to give authority to decisions. However it is rather authorative and responsible for implementing and maintaining those decisions (only “rather” because if we end up not properly maintaining those parts, other Nixpkgs committers might just do it instead).
Furthermore, a working group has the notion associated with it that they work towards a specific reachable goal, and then disband once that goal is achieved, without a permanent presence. This does not match the Nixpkgs Architecture Team.
Fully agreed from experience. It’s also hard to do a good job if the team or one’s role has a too broad scope. I’d always strongly recommend writing that down beforehand and checking regularly if it’s still realistic given available time.
I’m not sure about that. I can imagine having a cross compilation working group (it would be a much closer fit than a team for how it currently works), for example, and it’s not like we’re ever going to be done with cross compilation.
If the group is using the RFC process to gain authority, then isn’t it on the same level as other contributors?
My impression is that we just have a different definition of a Working Group. I don’t mind changing the name if that helps. To me, the difference between Working Groups and Teams is more a question of nature. It doesn’t say anything about lifetime and importance.
In fact, per the current discussion, Working Groups seem to be more desirable in general and nixpkgs architecture could be leading that transformation.
What would be nice is if we could agree on that terminology, because then it can be used consistently across the ecosystem, and make it easier for everyone to navigate it.
I don’t really mind changing names (other than it being a pain to do), but changing names doesn’t really do anything by itself. To get more team/group standardisation, somebody should propose and write down the standard, get consensus on it, and then teams can follow it.
In fact, this is why i think this topic came up. The specific incident in question was about the auto-merge-bot and a discussion started. And while that seems to be resolved well, there was a group of people that didnt quite have a clear voice, but should have one. I’ve been thing of this as the “Nixpkgs Daily Operations” team - in contrast with the longer time horizon of the Nixpkgs Architecture Team. And while that set of people should have a voice, that authority also comes with responsibilities. I dont think it is too much to ask that spme formalization take place by the people who want to have a voice when such situations come up. The exact format can vary, but that is the associated responsibility.
To summarize: a team provides a way to give authority coupled with responsibility. Working groups provide organization to get a specific task done, but usually have less responsibilty/authority as a result, which suites that model well.
The next question is how to ensure teams don’t become silos or gatekeepers. This is a real risk. I think that risk should be mitigated by having the teams be responsible for things. Meeting notes; a regular status report; asking for, spending, and reporting a budget; establishing written goals; clarifying processes - these sorts of things keep the team easily accountable.
Next, you need a check/balance mechanism or two. One is the RFC process which can put pressure from a non-team source. The other is the Board. @zimbatm , the board should ensure the teams are healthy, productive, unblocked, and hold team leads accountable. And another is the the other teams; hence the value in some regular team lead sync session.
The final question is if the risks of more teams and more formalized structures outweigh the risks. In the case of the overall Nix project, i think it generally does. I propose that many of our weakest points are due to unclear authority/responsibility and poor accountability.
We did have a clear voice. There were a lot of obviously very active Nixpkgs contributors who all shared the same concerns about the bot.
This sounds like you’re saying that if the people who work on Nixpkgs every day want to have a voice about what happens to Nixpkgs, they need to do less work on Nixpkgs and instead do a bunch of busywork. This is pretty much guaranteed to team membership becoming out of sync with the group of active contributors they’re supposed to represent, because the people on the team are going to be busy running a team, and so the people who still have time for lots of Nixpkgs work (because they’re not wasting their time in meetings) are still not going to have that voice. We’d end up in a strictly worse situation, because it would look like there was a team consisting of those people, but there wouldn’t be.
Skimmed the thread so far, but here are my 0.5 NOK:
Teams have both pros and cons. The pros are shared responsibility, pooling of domain knowledge, and dedicated syncronous communication channels for coordination and as a known place for non team members to reach out to the team.
The primary cons IMO is that it is more common for no one to respond when @nixos/<team> handles are pinged compared to when single individuals are pinged, since no one on the team has been assigned the responsibility of responding to pings and delegating work.
To make forming many teams be worth it, we have to tip the scales such that the pros win out. To this end I propose:
New teams should recieve a matrix channel and an entry in team-list.nix, promtly
The team-list.nix could even link to the relevant matrix channel
We could do a yearly check-in issue on github, to prune or reform dead teams.
Teams should assign a member to respond and delegate when the group is pinged, this responsibility may rotate among the team members
Teams could be assigned a committer or two acting as their fast path to get merged.
This encourages forming more teams of non-comitters where their domain knowledge can be elevated. The team may carry most of the initial PR review, then ping their assigned committer once the group condiders the PR sound. The assigned committer will naturally gain more and more domain knowledge as a result of this
A problem with the CODEOWNERs file is that only committers may be listed there, to my understanding. We could improve ofborg to consider any file touched within a by-name package to be under the perview of (callPackage package.nix).meta.maintainers. PRs that don’t touch package.nix or default.nix don’t cause a review request at the time of writing.
We did have a clear voice. There were a lot of obviously very active Nixpkgs contributors who all shared the same concerns about the bot.
Correct, and they all had to individually express that. And at first it could have been easily missed to check with or gather their opinion because there is not an easily identifiable group. Let’s say right now I have some idea, or some proposal that will impact the very active Nixpkgs contributors - I have to know who they are, I have to know where they communicate, there is no clear way to give that set a people a unified voice. What about situations where that set of people may disagree… how is that resolved? What if not everyone is comfortable expressing their opinions? Or is unavailable when decisions start to be made? In contrast, if I have some concern about security, at least I know who should be responsible or who I should initially reach out to, because there is a list, contact info, some processes written down, etc.
This sounds like you’re saying that if the people who work on Nixpkgs every day want to have a voice about what happens to Nixpkgs, they need to do less work on Nixpkgs and instead do a bunch of busywork.
Not a bunch of busywork, but enough to justify having a voice, yes. “Work on Nixpkgs” includes the coordination, this is stuff that is already happening, but it is difficult to see, difficult to track, difficult for newcomers to understand. Having clear contact points, a defined set of people to contact when things go wrong, a set of written policies, a set of responsibilities, a report about the problems/issues/proposals, requests of support from other teams… hrm… it’s starting to sound like a team. I hate busywork and meetings as much as anyone (sometime ask me why… and how it became a life-and-death situation where the paperwork only made things worse). I am keenly aware of that danger, but we are on the opposite side of that spectrum. We are not suffering from too much organization or cohesion, but too little. I’m NOT suggesting that all of those tasks need to be done or that there need to be mandatory weekly meetings, but that some subset should be, or the appropriate ones decided by the relevant people. “Team membership” (or working group, or community-of-interest… the exact label can vary) should reflect reality of the things that need to be done and not a waste of time. And yes, this does impose some responsibilities on people - that’s the cost of having some authority or a voice in decisions. Asking for a voice without responsibility is just as pathological as having responsibility without a voice.
We’d end up in a strictly worse situation, because it would look like there was a team consisting of those people, but there wouldn’t be.
This is a risk. Let’s be aware of this, mitigate it, monitor for it.
And yet the system worked. And it generally works. I think we’d need some actual examples of the system not working to justify such an invasive change.
If somebody is so removed from the community that they have no idea who the active Nixpkgs contributors are, no idea how to find out, and no idea how to get their attention, they’re probably not the right person at that point in time to be suggesting and driving major changes to how Nixpkgs works. Fortunately, in practice this is quite rare. It’s not necessary to have a complete idea of who needs to be consulted at the change at the start, because you can ask people and find out! And then you’re likely to get a much more accurate answer of who the right people are right now.
How does having a team help with any of this? People are still going to disagree, some people are still not going to feel comfortable expressing their opinions, and people with opinions are still going to be unavailable sometimes.
It might, depending on whether the contact info you look at is one of the bits that is up to date, or one of the bits that is not. Graham is the first person listed as a member of the security team, and yet he’s been inactive on it for months. (FWIW: I agree that having a formal team and written down stuff is important for security, because that’s something where people who’re not actively involved might are likely to still need to get in touch. I don’t think the same applies to large changes to Nixpkgs. But the fact that the security team has fallen into exactly this situation is demonstrative of a problem with teams in general.) If you asked around “Should I report security issues to Graham?”, there’s a good chance people would be passively aware that that’s not the best thing to do at the moment. And yet nobody has updated the team list. Maybe because removing somebody inactive from a team without explicit signoff from them feels rude or even impermissible, maybe because they feel it’s not something for them to do because they’re not a member of the team, or maybe just because they don’t know about the team list. Whatever the causes, they’ve come together to manifest a situation where the collective wisdom has been updated, but the written down stuff hasn’t been updated to reflect that. This sort of situation demonstrates that it’s a real trade-off when it’s best to have written stuff, and when it’s better to rely on people asking for up-to-date information. For large Nixpkgs changes, I think the benefits of the collective wisdom approach outweigh the downsides in a way that doesn’t apply to, say, the security team or the infrastructure team.
Why does the existing work that people do on Nixpkgs not justify having a voice? Of course, it’s great if volunteers want to try to work on making the coordination more visible, etc., but it’s not fair to force people who’re already putting a huge amount of their lives into this to do even more work if they want to have a say in decisions that mostly affect them.
Teams make things worse for newcomers in some ways as well. Often newcomers show up, send a small first PR, then trickle in a couple more, then a trickle becomes a flood as they start gaining expertise with a particular area of Nixpkgs. As time goes on, other people notice and recognise the work they’re doing, and start to treat them as an authority on that stuff. I’ve seen this happen constantly in the years I’ve been working on Nixpkgs. But that nice gradual, natural process doesn’t happen with teams, because there’s a cut-off of whether you’re on the team or not. And if there’s one thing new contributors are not good at, it’s having the self-confidence to ask to be part of some formal structure. People who should have been committers years ago often have to be strong-armed into asking, because they’re not confident enough in themselves or scared of rejection. A team can try to be proactive about recruiting, but then that’s yet another responsibility for the team, and it’s easy for it to fall by the wayside. Instead, we tend to see (less active, especially) teams ossifying, especially when newcomers are not actively asking to join.
Why aren’t we starting with the problems this causes, and discussing from there what to do about them, and considering multiple approaches? I feel like all the discussion about teams in Nixpkgs recently has started from teams as a solution because they’ve sometimes worked for other situations other groups of people have found themselves in, and then going backwards to identify problems that a team would solve, without considering whether there might be any other solution to those problems.
For the last few weeks I’ve spent about half of my working time on staging regressions, because I feel responsible for keeping certain parts of Nixpkgs working, and other people expect and rely on me to do that work. Lots of other active contributors similarly already have lots of responsibility to Nixpkgs, and they take it seriously and live up to the requirements. Why doesn’t that responsibility count?
Lots of other active contributors similarly already have lots of responsibility to Nixpkgs, and they take it seriously and live up to the requirements. Why doesn’t that responsibility count?
I probably monitor the Staging channel more than most people, but I had no idea you had taken on that work and commited to it. If I missed it, I’m sure most people have.
It should count. I would like it to. But because it is invisible and fleeting and informal, it is easily overlooked or ignored. It helps a lot for more people to know that you and others have volunteered to accept that sort of responsibility. It is not trivial at all, I understand that. So it would help if a few people wrote down what they are volunteering to be responsible for, what kind of time they are devoting to it and explain to others what areas don’t have enough help at the moment or are at risk because someone is burning out. Occasionally you’d probably want to give some updates about how things are going, how that work can be expected to change in coming months, set expectations. This is starting to sound a lot like a team. It doesn’t have to be super formal with exclusive membership being required to volunteer.
I’m advocating for this in particular because I want Nixpkgs maintenance to be more prominent and celebrated for its value. I want to write about this system works. I’ve answered questions about how Nixpkgs is maintained and people find it hard to navigate without being in the “in crowd”. I may just happen to know what might work for them, or the right contacts, but this doesn’t scale. I want to be able to get funding from various sources and describe what activities it will support and to whom it should go. I want people to be able to rely upon this ecosystem and to trust it.
I feel like this could possibly be addressed in a descriptive, rather than prescriptive way. A summary of “here’s all the work that’s been going on in Nixpkgs” would help with this, without requiring extra work or commitment from everybody doing the work. This would also avoid the problem of “who’s in the team”, etc. @matklad was talking in another thread about how summaries of discussions have worked well for the Rust project, where people were having trouble keeping up with long discussions spread across lots of different places. Maybe we need summaries of work done as well. These summaries would also be a good place to tell people how to get involved with a particular area, etc. I’m an avid reader of LWN, which is basically work summaries for kernel development, usually written by people who specialise in keeping up with the work and writing about it, rather than the people actually doing the work themselves, and I’d love something like that for Nixpkgs.
I’m good at programming, and keeping up with what’s going on, and I think I’m quite good at the human parts of finding solutions to (reasonably sized) technical problems. I’m not good at keeping a schedule, organising, or writing. I can’t commit to attending regular meetings. If those things become requirements for contributing to Nixpkgs at the level I currently do, I’m not going to be able to do it, and I know it’s not just me who’s in that situation. I feel like when teams get brought up, it’s often with the expectation that the people who work in whatever area will just start doing those things. If we were approaching all this with the idea of people with that second skillset supporting people with the first, that would be quite different to me. But the teams I’ve seen so far have not been that.
Taking making it easier to keep up with staging as an example, I could imagine something like the following working well:
Somebody (or a group) with good writing and organisational skills decides they want to become the Staging Chronicler. Every cycle, they outline the work that’s been going on on GitHub and Matrix. They only need to have a surface level understanding at this point of what that work has actually been, and then they look at who’s been involved with each bit, and they can talk to them to have it explained in more detail until they have a good understanding of whatever they need to understand, and then they can write it up. The end result would probably be much better than if a “Staging Team” consisting of the people most actively working on staging were forced to write this up for themselves, and it doesn’t force extra work onto those people, beyond answering questions about the work they were already doing. The chronicler would have made a huge contribution to making Nixpkgs better, and I’m sure they would be recognised as such for their contribution.
I have been considering joining the marketing team to resume work on the newsletter but targeted to keeping informed the community about Nixpkgs events and I would like to try things to keep very simple, maybe, one-line summaries about things happening in Nixpkgs.
I have seen it for Rust newsletter and as a reader of LWN too, I also feel the need for that in Nixpkgs. I don’t know how far I will go alone on this and maybe other people will join me or help me to centralize the blurbs of what is going on, but I feel like I have a lot of stuff already as an active member and sometimes I have trouble to focus on writing code but I still hang around in those channels and still know what is going on.
I love this conversation because it brings up so many good perspectives.
Another way might be to have a dedicated place to post these things in advance. But we have to be sure that the interested parties are subscribed to it. (please don’t say RFC :-p)
The biggest issue I see is that it takes a lot of effort to make large-scale changes. Look at how much effort Infinisil put into getting pkgs/by-name passed. How long has it been going on, half a year and still going? And Infinisil is the best person I know to handle RFCs. This is for an idea that isn’t controversial.
I feel like we’re falling behind on a lot of large projects, and losing goodwill from contributors because of that friction.
Maybe the issue is that consensus gets exponentially harder with the number of participants. I don’t know if the answer is teams, but I know that with small groups of people, the rate of decision-making can be another order of magnitude faster.
This is how it works in corporations or military structures. In that scenario, people get pressured to perform, or else they don’t get a promotion or get fired. I see that mode of thinking coming more from people who are employees themselves. This also goes with top-down thinking, requiring meetings and fixed resource allocations.
All of this is toxic to voluntary work on some level.
In voluntary work, you work on something because you’re interested in it (for whatever reason tickles you). You meet people because you like interacting with them. And you get people motivated by inspiring them. The incentive structure is reversed.
A good test of your own mindset is if you see the NixOS Foundation at the top of the community, or at the bottom. Should we be beholden to it, or should it be in service to us?
On some level, I feel like this discussion is representative of both of these modes.
I agree that this is an issue. Something I’ve noticed with RFCs is that there’s an expectation that every detail is going to be worked out before the RFC is accepted. For some things that’s important, but it feels to me like maybe we’re missing a way to get an idea approved in broad terms, without making people care about or review the implementation in more detail than they need to. For example, I personally don’t feel that the merge bot necessary needs to go through an RFC level of specification before it’s introduced, and would happily agree to something much shorter and looser as long as it addresses my concerns, like “We will have a bot that allows maintainers to merge PRs by r-ryantm”, without needing to have a whole big document with background, alternatives considered, detailed design, etc., and trust the people involved to come up with the specifics.
Having skimmed at lot of it, I see most of the sticking points being that of scale and scaling from 100’s to 10000’s of developers/contributors.
Scaling a company, service anything can be hard…
Even large mega corps have trouble organising and scaling things… and they have paid staff!
Nix is super inclusive, because it’s just a public git repo at the end of the day…
to get my changes into say , the linux kernel i got be a much closer part of the cabal :-).
When i see what were dealing with, and the growth in the last year, it’s mind boggling.
I do feel that a long drawn out RFC process can be demoralising and tarpitty… i’ve always thought the PR welcome’s culture , where you do extensive work for it to be denied. is tantamount to developer abuse , especially if it’s some complex and took a long time to write.
That’s true for every open source project i’ve ever seen.
Someone mentioned working groups, or Special interest groups, I think the IETF called them BOF sessions too… What I’ve seen of volunteer working groups is it allows those with a common interests or itches to scratch to get to know each other…
don’t under estimate the value of human connection and collaboration to get things done quickly.
Nix is probably the largest open source collaborative engineering project on earth right now… so it’s going to need new methods of working ! it’s much much bigger than the linux kernel.
I’ve seen people I’ve introduced to nix, go from newbie, to contributor in some way in a matter of months, not years …that truly is astounding to having the philosophy of nix into practise. This makes me smile that barrier to entry is low, really low.
It maybe just a tooling problem, not a human one, maybe githubs workflow, on how git works adds to the problem of organisation? A lot of large organisation don’t use git at all, (they are still on rcs/svn) lol!
exciting times. I’ll track this thread. in the meantime…stick with nixfication.
Don’t underestimate what this «still going» means, though. It is a large change. There are a lot of failure modes for it, there was work to avoid some of them during the RFC discussion, and there is work being done now to figure out and avoid some other ones.
And yeah, by now I think that having large changes be slow is a feature not a bug.
Just to give another datapoint: I am a full-fledged member of the Docs Team, but I often don’t have time/capacity to attend the video meetings (there are 2 a week). This means that I don’t get to voice my concerns on many issues in the synchronous, high-bandwidth medium of a group call, but I can still catch up via meeting notes and, if need be, make my voice heard asynchronously via chat.
When I do have the time, or a particularly pressing issue to discuss, I join the calls.
This seems to work fine for the rest of the team (but hey, if not, let me know ), and allows me to participate in a way that is not overly demanding of my time, but still has a level of commitment to a specific group of people.
What being an official team gives you (as @tomberek said in above) is basically a Schelling point, or interface, for things like funding, PR review assignments, and so on.
There are people who do make it to all of the meetings (bless them), and I don’t know if the team could function if everybody was as… flexible as I am, but what I wanted to show with this example is that not everybody has to be in-sync all of the time for a team to be succesful at both being productive, and being an interface.
), and allows me to participate in a way that is not overly demanding of my time, but still has a level of commitment to a specific group of people.