Improving our RFC process

Development RFCs
1

I’ve written and shepherded a handful of RFCs over the years, and noticed that some frustration points about our RFC process keep coming up again. I think that it is now time for us to reflect on our past experiences with this, and discuss how to improve things going forward.

So, let’s share your experiences you’ve had with our RFC process so far.

I’ll start with a minor annoyance, that the FCP process is under-specified.
Who actually declares FCP? Is it the shepherds, or is it the steering committee on behalf of the shepherds? If I read the text correctly, the idea is to have one shepherd propose FCP, and then FCP starts only after the other shepherds signed it off. Does this happen automatically, or does it require an explicit announcement after the shepherds have signed off? If so, who is responsible for that? When do we start counting the 10 days for the FCP? And shouldn’t it be the other way around in the first place? The shepherds all give their agreement, and then FCP is declared

It is also advertised widely, e.g. in NixOS Weekly and through Discourse announcements.

Let’s ignore the fact that NixOS Weekly is pretty much dead, who is responsible for making the Discourse announcement? Is it the shepherds or the steering committee? Is such an announcement “mandatory” in the first place?

A trickier one is the relationship between RFC authors and shepherds. In theory, the author is the main responsible for the RFC, while the shepherds’ job is mostly to guide the discussion. In practice, being a shepherd involves a lot more work than that. Some RFCs have been completely rewritten by the shepherds, and in some cases the original authors even stop participating …

The RFC process also states “The author cannot be part of the Shepherd Team”, which also gets a lot trickier in practice than one might initially expect. Does this also apply to co-authors? What if the shepherds completely rewrite the RFC during the process, do they now count as authors? What if an entire Nix team authors an RFC? On this topic, the steering committee wrote in Update 2023-05-03 · Issue #112 · NixOS/rfc-steering-committee · GitHub :

  • sees this as not much of an issue, partly because shepherding often results in contributing to the text of an RFC anyway;
  • RFCSC shall avoid selecting a shepherd team that does not include any opposing voices when such voices are present in the discussion;
  • Formalising this is not necessary, but if people feel the need and are willing to write an RFC to codify this it will of course be considered like any other RFC.

I do think that updating our process to reflect these questions would be favorable.

All these points above raise questions about the role we want our shepherds to fulfill. Do we select shepherds as proponents of an RFC whose job is to safely bring it through the process? Or do we think of shepherds as mostly neutral helpers, preferably with the shepherd team having diverse opinions on the subject? Because the process reads like it was written with the latter in mind, while the former is more common in many RFCs.

One major point of frustration that I regularly see, but which I fear is not a “technical” problem, is the lack of feedback and participation. Especially for the less controversial and more straightforward yet technical RFCs, finding shepherds and getting forward can be extremely difficult. Sometimes no external opinions are raised at all until FCP gives the RFC some additional visibility. This can be very frustrating and demotivating.

On a closing note, my intention is about iterating and making incremental improvements to our process, not about re-inventing how our RFC process could look entirely different instead. I do not necessarily want to change the lived process, I mostly want the written process to more precisely describe what is already happening in practice.

Also note that I have no plans of writing an RFC for this myself currently, but I do hope that somebody else will be inspired by this discussion to do so.

10 Likes
2

Thanks a lot for this post! I’ve been in contact with @piegames for a number of RFC’s where such problems came up:

  • RFC 101 where the shepherd team effectively took over authorship
  • RFC 127 which only announced FCP in a comment, not on Discourse
  • RFC 140, which struggled to progress because the entire Nixpkgs Architecture Team was listed as authors, therefore having to find 3 other people as shepherds, see here. We eventually did find 3 non-NAT members, but it’s a bit ridiculous to be blocked on an effectively non-controversial idea because of this, when there’s already 5 people agreeing with it.

Something else is that I think RFC’s should be required to get approval by the relevant code owners. E.g. an RFC that changes how Haskell packages in Nixpkgs work should get approval by the people that maintain those packages. Or an RFC that changes the Nix syntax should get approval by the Nix team, etc.

11 Likes
3

Something else is that I think RFC’s should be required to get approval by the relevant code owners. E.g. an RFC that changes how Haskell packages in Nixpkgs work should get approval by the people that maintain those packages. Or an RFC that changes the Nix syntax should get approval by the Nix team, etc.

I disagree with this. The RFC process acts as a check on teams, where they might be making decisions that are not in the wider community’s best interest. It’s the only way we have to allow the wider community to override a team when required, and if this suggestion were followed that would no longer be possible.

11 Likes
4

I’ll add my own main frustration with an RFC process: it’s unclear how decisions should be made on controversial RFCs. The RFC process as designs envisages RFCs gradually converging towards consensus, at which point they’re accepted.

But sometimes, people are just not going to agree on whether something is a good idea, and in those cases, it’s not clear how the decision is made. Theoretically the shepherds make the final decision, but do they get to vote based on their own personal feelings (in which case who ends up being somewhat arbitrarly chosen as shepherd — when I really paid attention to this a couple of years ago, the steering committee tended to just pick the first nominees — can make a huge difference), or are they supposed to try to represent the community, in which case how do we keep them accountable? What happens if shepherds are deadlocked?

If lots of people are in favour of an RFC, but there’s a non-negligible number of detractors, and the prevailing interpretation of the RFC process is that consensus is required, the RFC will just languish as long as the objectors keep objecting. And on the flipside, there have been one or two RFCs that I’ve thought were just flat out bad ideas, and in that situation, it’s very hard for me to have confidence in the RFC process weeding them out, because so much depends on who happens to be nominated for the shepherd team, and I expect shepherd teams bias towards people who are in favour of an RFC, because it’s very draining to dedicate that amount of time to an idea you don’t like.

15 Likes
5

I think it’s not that ambiguous that FCP decisions are made by shepherds and not RFC SC. Any shepherd can launch a vote on the matter (targeting unanimous consent among shepherds).

Announcements are a more annoying thing because a shepherd team might have no people with Discourse access for posting in Announcements category… So some of the shepherds approach the Discourse moderators and asks to post an announcement. I think that if this ends up taking more than a day, it is a reasonable courtesy to count the FCP as started at the vote time and ending ten days after the announcement, but this is indeed not specified.

I could support the idea that this is a feature (if you try to collect all the people who care into a team, find who will write from scratch after an initial discussion, and make all the collective improvements as discussions on an RFC proposal, which will probably be slightly less team-insular), but in any case it’s better to have some guidelines for such a case.

If no consensus can be reached on the RFC but the idea in general is accepted, it gets closed, too.

Presumably, if no consensus is achieved even on the idea, then the correct disposition is also to reject as «currently unclear»

[quote=“qyliss, post:4, topic:28552”]
Theoretically the shepherds make the final decision, but do they get to vote based on their own personal feelings [/quote]

RFC Shepherd Team members use their best judgment in taking this step, and the FCP itself ensures there is ample time and notification for stakeholders to push back if it is made prematurely.

However, sometimes substantial new arguments or ideas are raised, the FCP is canceled

I think the intention is their own best judgement but with an expectation to acknowledge all the arguments raised (but not necessarily agree with them).

On importance — absolutely.

On order-based selection — I wouldn’t characterise it like that, it’s mostly «if after n≥2 weeks the set of nominees is barely enough for a shepherd team, they are indeed all approved». This does look the same in many cases, but I do not see any sign that nominees in the first week are treated differently based on the order.

It’s a bit better when you pre-announce the position «I want to authors to give a convincing explanation why objections a-b-c-d do not overweigh the motivation section» (as it makes it reasonable to defer judgement on some sections until the core idea discussion conversation converges).

It’s true that there is a «pro» bias, heavily controversial things usually get one or two detractors in the ST though. Mildly bad ideas have a chance of passing, sure, but maybe this is an acceptable trade-off.

1 Like
6

Yesterday some people compellingly explained me it’s the exact opposite. Which kind of drives home my point that it is ambiguous, and that we should document this as part of the process.

Having to ask moderators to make posts is not an acceptable workflow—either shepherds get write access to the forum or we delegate that job to the steering committee (or some other entity).

1 Like
7

[RFC SC] only in charge of:

  • selecting the Shepherds unanimously
  • supervising that the Shepherds are carrying out their work
  • committing the final RFC

Final Comment Period (FCP)
A period of ten calendar days, which will be called by the Shepherd Team after the RFC has received ample discussion and enough of the tradeoffs have been discussed.

So FCP is called by shepherds and only the final merging is done by SC. I am willing to say outright that to argue that RFC 36 says otherwise one needs to read it inattentively. Sure, careful reading is an effort thus clarifications are welcome, but this part of the process is specified.

I agree that giving the shepherds (or at least one shepherd in each team) right to post RFC announcements is an improvement.

1 Like
8

This neatly ties in to another point I wanted to bring up:

Minor modifications to accepted RFCs can be done in follow-up pull requests. We strive to write each RFC in a manner that it will reflect the final design of the feature; but the nature of the process means that we cannot expect every merged RFC to actually reflect what the end result will be after implementation.

The divergence between RFC text and initial implementation is rather small usually IME, that’s not the problem. The problem is that in the future things will change over time. And many times, those changes will be small and incremental and not really require a new RFC. Now, how do we deal with them? Do we do a small PR to update the old RFC text? Do we let it diverge over time? And if there actually is a second RFC to change things, does it amend the text of the first one?

And a good example is the RFC process itself. Its definition is now spread over four different RFCs, and counting. So if I want to know some information about the process, I now need to read all of them and merge the information in my head, because I never can be sure that what I’m reading hasn’t been overridden by a later one. But this also applies to other topics like the platform support tier list, or in the future the Nix formatting guidelines.

In my opinion, an RFC text therefore should be a change description of things to do, and a snapshot of the current state and situational context. Once it is merged though, one should never have to look at it again (except for historical questions), and every relevant information should be provided in the official documentation instead.

In the case of the RFC process, the normative specification for me is the description in the repository’s README. I did not read RFC 36, and if I need to in order to answer these questions then the process description has failed its job.

3 Likes
9

At the time when I wrote RFC 0001, there was no decision mechanism, so I cobbled something together. It’s probably time to re-evaluate it and come up with something better.

On top of all the good observations here, I have two major gripes with the current system:

  1. It takes a lot of time and energy to get an RFC trough.
  2. The shepherd’s selection mechanism is not very democratic.

Of course, we want good quality RFCs, but the Shepherd’s team has a weird double role of both helping, and judging the work of the RFC author, and that causes unnecessary friction. In my experience, RFCs tend to generate uncomfortable meetings. It also means that people that have a lot of time on their hands have a disproportionate power to propose decisions. I think we’re missing out on good contributions because of that.

In retrospect, it would be better if RFCs were simply proposed to be voted on. That way, the author can submit the RFC on their own time, when it’s ready, and work with people they are comfortable with, or alone.

In this new approach, I would propose that:

  1. Anybody can submit a RFC to be voted on (if we get too many submissions, we might ask for a number of signatures to be accompanied by the proposal).
  2. The RFC team is responsible for organizing the vote. It has X days to do so.
  3. Anybody can submit a counter-proposal RFC within the window.
  4. The RFC team might identify key stakeholders that the RFC touches, and ask for their opinion. Those opinions are published with the vote.
  5. A community member is defined as somebody who has commit access to Nix or nixpkgs.
  6. Community members are allowed to vote on the RFC.

That way, the community as a whole has a vested interest in participating in the process. Because they might not be domain experts, they can defer to the opinion of (4) to vote. Or even make a counter-proposal.

If there are multiple competing RFCs, we can use one of the advanced stack ranking voting techniques.

As a community, we should probably also be pretty strict and vote NO on low-quality RFCs, even if we agree with the content in principle. The author can always re-submit the RFC after correction.

6 Likes
10

Yes, that’s a good idea, although of course there is the usual incentive mismatch (a lot of changes just happen without RFC anyway, including process changes; RFC often serves as a ground of negotiation; and people who know the content of the resulting — evolved— process don’t know what is clear and what is not).

Actually, if only we had a single repo, we could do the proper thing. An RFC PR should have motivation as a patch to «decision record» and detailed design as a patch to documentation. Not sure how to do it with Nix/Nixpkgs split, though.

The phrases I have quoted are also in the README.md.

1 Like
11

As someone who has taken time as an author to explain in detail to a sceptical but willing to negotiate shepherd what measures alleviate the concerns and which of tne less-obvious benefits get unlocked, I say that a universal deadline is almost surely a bad idea.

3 Likes
12

What do you think of the high-level proposal? This is a reaction to one of the points, which can be changed. Modifying it doesn’t fundamentally change the overall idea.

2 Likes
13

Votes have been discussed in the past for various occasions, but the main problem is, who gets to vote, or how to decide who gets to vote? Also, how to vote? How to protect against manipulation etc.? How to deal with low participation?

Even with these solved, I’d like voting to not be the default, and instead only used on controversial RFCs if not consensus cannot be reached otherwise. Because I think the current consensus based approach tends to work out okay for most RFCs.

3 Likes
14

I think most of the things that are changed in it, are not good. For the specific point I have enough confidence to object with least reservations.

Defining constituency for voting and figuring out all the impacts of a definition is hard. It has been tried multiple times now and we have given up every single time.

I think we should aim to be a coherent project and to be ready to deal that there is a fine mesh of interacting subcommunities. I do not believe we have a true singular community around the project, I do believe it would be undesirable, and thus anything that has a need for «as a community we should» seems questionable to me. Sometimes, with careful negotiations, we achieve an agreement on «as a project process we adopt the following». (But here the desirable thing is not really a mechanical process but indeed an adjustment in community attitudes — across the subcommunities interacting with the project)

4 Likes
15

OK, let me try to be constructive (but speculative)

Is it a good idea to introduce a formal handover: if all the shepherds of RFC X and the author of RFC Y agree that a) RFC Y is a fork of RFC X preserving the direction, b) the shepherd team are willing to be the shepherd team there (either having enough people after loss of the author of RFC Y from the team, or not containing the new author anyway) — RFC SC may announce the intention to agree at the next SC meeting (basically immediately in RFC time), which comes into force if after evaluating the reactions to the announcement at the next SC meeting there is still unanimous agreement.

Basically, if an author has written something rather good but has no time to spare for polishing, make it easy and process-encouraged for someone else to step up for continuing the work.

2 Likes
16

The current reactions are an excellent example of what consensus building looks like in the current RFC process; if your idea is not popular with the in-group, it doesn’t get given proper consideration. Or you get a small vocal minority that rejects or nitpicks the proposal until the original author runs out of energy.

Anyways, I don’t mean to argue, but my impression is that we are losing some interesting contributions and proposals because of that kind of dynamics. That’s why I’m proposing to try something different. And also because for social topics like RFC 98, everybody should have a vote. Voting also isn’t really that hard; Debian has been doing it for years. The issue is that most proposals tend to get nit-picked to death.

But we could try other ways to organize. Sometimes the best thing to do (especially with social stuff) is to try different things and see if they work better. All systems have pros and cons.

11 Likes
17

I agree that making decisions in the broader space is exhausting, which has multiple negative consequences that have been outlined here and elsewhere already (“tyranny of availability”). I’m skeptical of voting though, because, as @piegames pointed out, it opens up even more hard questions that don’t contribute to solving the original problems, which are mostly technical in nature.

I very strongly agree with that. Here’s another angle of attack: Let’s shift our thought process away from RFCs as „make an important-looking document“ and towards „write an elaborate commit message for a far-reaching change“ (implemented as a separate document only because our tooling clumsy) and focus on making incremental changes.

That is why I think @zimbatm‘s proposal is bound to fail making any progress: it’s too broadly scoped. Let’s instead pick apart the underlying issues and resolve them in isolation. This may feel like going excruciatingly slow, but has a much higher chance of finishing anything at all.

Some of these underlying issues as I perceive it:

  • Fuzzy problem statements
  • Hard to navigate technical and social environment, due to lack of information or structure thereof
  • No (or weak) definition of owners, stakeholders, and constituents

I’d welcome @piegames making PRs against the RFC repo in order to make process changes, because who says it can’t be done?

2 Likes
18

Yes yes yes this is exactly what I mean.

I’ve been thinking about that, but I’m unsure to which extent an RFC would be required — I’m currently a bit reluctant to author yet another one at the moment …

I think any clarifications of questions that are technically already described in existing RFCs should be okay, but other things like making RFC documents “less important” might be trickier. I might give it a try though and see how far we can go without having to upgrade it to an RFC

2 Likes
19

Regarding voting, I think that does have some good uses. As a general rule: Polls should have objective answers. So don’t ask “Which solution do you like more?”, rather ask “Which of these use cases are you using?”, and then use this to decide how important it is to continue supporting those use cases, and whether each solution does support them.

A good example (imo) is this poll which I created for RFC 140: Poll: How often do you navigate through directories on GitHub to get to a nixpkgs package file?

1 Like
20

IMO, polling and voting are pretty different concepts. As you described, a poll is used to gather data in order to inform a decision, and usually comes with little to no expectations w.r.t. its outcome or legitimacy. A vote on the other hand is inherently tied to a decision being made, and its outcome is binding within some context.

4 Likes