FCP, Prototypes and Implementations for RFCs

Development RFCs
1

I’m posting here because during RFCSC meetings we frequently see RFCs that seem to have reached agreement but stall out because the authors are working on an implementation and want to wait for that to be ready before merging the RFC. I thought it would be interesting to put this to discussion and see if everyone is on the same page and what the different points of view are.

The question is:

Should implementations block RFCs from entering FCP and being accepted, or should the RFC go to FCP as soon as there is general agreement in the community?

My personal stance is this:

RFCs are a way to document the position of the community. Accepting an RFC means “Given the information we had when when discussing this RFC; the Nix community believes that this is the right path forward.”

Notably they are not a promise that something will be done by anyone in particular (or anyone at all). Someone has to still put in the time and effort to contribute the feature discussed. However they should not expect any resistance to merging the feature once implemented (aside from quality of the implementation itself) as the community has agreed that this is the right direction.

An RFC is also not a law that must be followed forever. In light of new evidence the decision of the RFC may no longer be relevant.

Therefore, I see an implementation as unnecessary for an RFC, and should not block FCP. I do think that prototyping/initial implementation can be helpful to ensure that we have a good understanding of the problem at hand, but once we believe that we understand the problem we should move forward with FCP and accepting the RFC. Worst case we hit an unforseen major issue and a second RFC can be opened explaining why the original was not actually a suitable option. But in the vast majority of cases only small changes need to be made during implementation (that wouldn’t require an RFC).

In fact I may go as far to say that the main point of most technical RFCs should be to ensure that there is agreement on the approach before implementation. If the implementation is already complete you may as well just open a PR and discuss there. (Of course in practice these RFCs with implementations generally grow together, so it is a bit different.) Not every change needs an RFC, they are a tool for avoiding wasting time doing work that may not be accepted.

This general discussion has come up a few times in RFCSC meetings and it seems that in general they agree with my point of view. But I just wanted to take the chance to extend this discussion to more of the Nix community to ensure that we are all on the same page.

If we are on the same page, it may be worth updating the RFC documentation to highlight this. For example it talks about “substantial changes” which I don’t particularly agree with. I see the RFC process more about agreement on potentially controversial ideas. The document does say “RFC documents are intended to be seen as the documentation of a decision and a snapshot of a moment in time” but it is “hidden” in The RFC life-cycle section. In fact the document seems to lack any explanation of what accepting an RFC means.

6 Likes
2

There might be two different questions.

  1. Is it OK with RFC process as defined and as intended to accept an RFC without an implementation and without a clear step-by-step plan towards implementation? — I think the answer is clearly «yes, it’s fine». If it is not fine in a specific case, hopefully someone will raise the question and convince the author or the shepherds.

  2. If the author confirms the knowledge of (1) and still wants to do some prototyping before finalising the text, and the shepherds find the definition of «done» well-defined, is there any process to pressure the author into finalising the text first and implementing afterwards? — Pretty clearly no, and if we want such a process, it needs to be planned very carefully.

3 Likes
3

I don’t have a lot of time to expand on this right now but I strongly disagree with the view of RFCs that they are not “law” or don’t have the power of law.

Sometimes, I see RFCs for which I see many implementation challenges that can have consequences on the RFC text and they are rarely considered if no prototype or implementation is done.

Most of the time, in other standardization committees, implementations/prototypes are a prerequisite for an RFC to be validated.

I will personally continue to ensure that my RFCs possess at least one implementation first because this is the quality standard I set for myself.

3 Likes
4

To expand on this.

RFCs are a way to document the position of the community. Accepting an RFC means “Given the information we had when when discussing this RFC; the Nix community believes that this is the right path forward.”

It seems like that multiple folks have a diverging interpretation of what are RFCs then.

Notably they are not a promise that something will be done by anyone in particular (or anyone at all). Someone has to still put in the time and effort to contribute the feature discussed. However they should not expect any resistance to merging the feature once implemented (aside from quality of the implementation itself) as the community has agreed that this is the right direction.

Yeah, I am afraid the problem that we then merge RFCs with no realistic roadmap to implementation or sometimes RFCs which will diverge from their implementation and then the resistance will happen again because there is no reason to accept an implementation that has significantly (undefined term) diverged from its implementation.

An RFC is also not a law that must be followed forever. In light of new evidence the decision of the RFC may no longer be relevant.

Sure, that’s a complicated one and depends on a lot of parameters, but RFCs can setups themselves in a way to make the cost of change of something high or require at least another RFC to subsume the previous revision and put an end to the RFC style way of doing things on that particular component.

It can be desirable that RFC are the fallback “law” for something in the community and thus are law to be followed until a suitable replacement has been pushed forward.

Therefore, I see an implementation as unnecessary for an RFC, and should not block FCP. I do think that prototyping/initial implementation can be helpful to ensure that we have a good understanding of the problem at hand, but once we believe that we understand the problem we should move forward with FCP and accepting the RFC. Worst case we hit an unforseen major issue and a second RFC can be opened explaining why the original was not actually a suitable option. But in the vast majority of cases only small changes need to be made during implementation (that wouldn’t require an RFC).

Sure, but in my perspective, it is a hard call to do to decide when to do an implementation or a prototype first, this is about setting expectations for folks too IMHO and not creating surprise second RFCs because of unforeseen details. This is also about seriousness to me, sometimes, I see RFCs which are abstract and all the interesting material is in the implementation details, and thus we merge courtesy RFCs.

“only small changes”, I feel like this is a perception statement too. The origin of RFCs is to deal with consensus building and “only small changes” can be as well considered not small changes and requires more work, but when a ship has sailed, things become harder to voice oppositions.

In fact I may go as far to say that the main point of most technical RFCs should be to ensure that there is agreement on the approach before implementation. If the implementation is already complete you may as well just open a PR and discuss there. (Of course in practice these RFCs with implementations generally grow together, so it is a bit different.) Not every change needs an RFC, they are a tool for avoiding wasting time doing work that may not be accepted.

I would say that RFC have, in addition, the property to be the place to look at / subscribe for architectural work in general vs. the usual noise in GitHub issues, as we have today no proper platform (no useful contribution chat platform, no useful forge project management tooling and I will leave at that for today.) to do this work. Hence, we must rely to tricks, RFCs are a nice gimmick to inform the community about such ongoing changes.

Overall, my personal feeling is that I have been disappointed in multiple occurrences of the handling of certain RFCs by their lack of implementation, but I will not ask other people to do it, all I ask is that authors are empowered to do their own call on the RFCs they are working on and the RFCSC does not tell them “what to do” when it comes to implementation or not.

I would be quite offended, and I have been quite offended by some RFCSC behaviors in the past (which I made it abundantly clear to the RFCSC representative who came in my RFCs) which just reinforced my negative opinion of the current way RFCs work in the Nix ecosystem.

I hope my view is clarified.

5

in other standardization committees, implementations/prototypes are a prerequisite for an RFC to be validated.

I think it is important to consider the context. Standardization committees are attempting to make a specification for an interoperable standard. It needs to be air-tight so that independent implementations end up compatible. Having prior implementations to hammer out all of the details and corner cases seems incredibly valuable in that situation.

I don’t think any Nix RFCs aim to do this. nixpkgs is definitely a single implementation so there is no need for a precise standard, Nix is in practice defined by a single implementation (although I can see benefit to changing this that is a different topic, and I think the process we use should reflect the current reality) and many RFCs are about community policy or similar which I think will always need some interpretation applied anyways (and often can’t have an implementation beforehand) as human situations are rarely clear cut.

So I guess the question here is what is the goal of an RFC:

  1. To gather an agreement from the community.
  2. To exactly specify a change that will be made to a Nix project.

I see it as the first, and see less value in the latter given our circumstance.

6

We literally have RFCs about introducing language versioning in Nix or formal documents with versioning…

We have folks working on public API guarantees and trying to ensure that external observation of details won’t lead to long term issues.

Maybe many RFCs are about community policies, but I find the term ‘any Nix RFCs aim to do this’ quite strong and I wonder why is that you feel like this.

But that confirms me a bunch of things about the RFC process, so I appreciate the clarification.

I don’t think that community policies had requests for implementations in the past, so I was always excluding what I was saying to inapplicable contents. If RFCs are only about community policies, let’s make it more explicit this is only about that and have technical RFCs being named something else, this will also solve this whole conversation I assume.

7

I don’t quite see how this implies that RFCs should be implemented and airtight before acceptance. I don’t think it is expected that people are building implementations off of the RFC text, I don’t see RFCs intended as a specification in that way.

What is the downside you see what an RFC about language versioning gets accepted without an implementation? For me I see a few “bad outcomes” but none seem particularly severe.

  1. The proposal has a flaw and either:
    a. A minor change is made without an RFC.
    b. A potentially controversial change is required so an amendment RFC is opened.
    c. The implementation is abandoned.
  2. No one implements the proposal.

1a and 1c and 2 have the downside of possible confusion if people expect the RFCs to be specifications. However the fact that RFCs are not specifications is currently explicitly documented in the RFC README.

RFC documents are intended to be seen as the documentation of a decision and a snapshot of a moment in time, rather than a specification-like normative document.

1b potentially wastes a bit of time.

2 is maybe also a bit bad because it is possible that the reason to abandon may not be documented. Whereas when closing an RFC for a similar reason the original author would likely leave a comment on the roadblock that they hit or if they just didn’t find the time.

But none of these are really emphasized because it is a specification about language versioning or API compatibility. It seems fine to me to agree on an approach to language versioning or policies about API compatibility without specifying every single semantic.

If I am understanding your perspective correctly it comes down to what the purpose of an RFC is. It sounds like you desire a Nix RFC to be a formal specification much like an IETF RFC or a Matrix Specification Proposals. The current RFCs on the RFC process seem to indicate that it is more about making a decision by having a formal discussion, concrete specification only required as necessary to make that decision.

The obviously begs the question do we want to solidify the RFC process to be one way or the other. What are the benefits of the “make a decision” process vs the “produce a spec” process.

8

The biggest downside I see is the classical one in this community: we talk a lot and nothing happens.
This contributes to my burnout factor, this contributes needless negative feelings.

I will add one outcome that you didn’t mention:

  1. Proposal has a flaw
    d. It is implemented because no one has the spoon left after the RFC discussion to reject it even though it has diverged.

And 1d. is equivalent to just send a controversial PR and just merge it to me with more scrutiny because we had the RFC dance before.

Correct, I am told that RFC is the place to do those things, if people consider this is not the place to do so, but I am still fight in PRs, that’s really not cool and just make the whole place less safe.

Either RFC can receive this, either we need to consider (ASAP) to do a new RFC to make a place for this and thus solve the whole question.

Large-scale technical work needs to produce specification documents or architectural decision records that doesn’t become outdated in 3 months because of unforeseen implementation issues, IMHO.

If we cannot do that, this will just result in people getting irritated, at least, I will be, but I will do my best to keep it down :slight_smile:.

9

I don’t really see much difference here. Many RFCs are opened, there are lots of discussion then someone loses interest and it closes. There is still lots of talk and little action whether or not the RFC gets merged.

Can you clarify what “things” you mean?

I think it is good to record these decisions but personally I’m not sure that RFCs are the best cost tradeoff here. We can have discussion on a PR itself with these technical discussion and I don’t see how that is materially worse.

It sounds to me like you want an RFC to reflect the lifecycle of a change:

  1. RFC is opened early to confirm direction before substantial work is done.
  2. Discussion and improvements to RFC occur.
  3. Implementation is done based on direction from in-progress RFC.
  4. RFC is updated to reflect implementation.
  5. RFC is accepted.
  6. Implementation is merged (presumably shortly after).

How I see the current process “decision based” process is:

  1. RFC is opened early to confirm direction before substantial work is done.
  2. Discussion and improvements to RFC occur.
    a. Some prototyping may be done during this time.
  3. RFC is accepted.
  4. Implementation is done based on direction from accepted RFC.
  5. Implementation is merged.

I think there is a notable difference here.

RFC is updated to reflect implementation.

This is missing. This means that we are lacking some documentation on the final implementation. However I wonder if an RFC is the best place for this documentation. RFCs are static snapshots and inevitably they will become outdated. While it may be nice to at least defer that to when the initial implementation lands it doesn’t fundamentally change the fact. I wonder if a large part of what you seek can be accomplished better by requiring better technical documentation as part of implementation (and any other changes) that can be kept up-to-date as the project evolves.

10

Right, I am greatly saddened to read this because I really just want to respond an semi-passive-aggressive line about how this is a big issue in the RFC system and how people have been considering leaving the whole NixOS ecosystem over this and I have been affected by it multiple times without any intervention from the RFCSC.

Yeah, the problem is that it waste time and energy from people trying to do things, and this is a critical problem. There’s no reason for such a process to be THAT painful.

So for people who are actually trying to get somewhere (I won’t make any assumption on people’s level of engagement when they open RFCs), this is very hurtful.

Overarching modifications of Nix(OS)/Nixpkgs with large scale consequences, even if they are a one line change, for example.

Here’s one issue: there’s no place to signal those PRs, so they are lost and stakeholders cannot find them, except… for the RFC repo :slight_smile:.

We agree on the notable difference, sometimes, we may even have a cycle of 2/3/4 because we can have up to N implementations that can enlighten the RFC or even sometimes make the RFC no needed in the first place!

I agree but RFCs can already be used as static snapshots if desired, you can use them with things that can be static snapshots, I don’t see the issue with that.

For things that are dynamic, yes, you need slightly more tooling.

11

I’m sorry, I don’t think I understand what you are trying to express here.

All I am trying to say is that I think it is inevitable that some RFCs won’t go to completion due to lack of personal resources by the author. It is unfortunate but even with fully motivated contributors life can get in the way. The only way I could see around this is requiring a full implementation before even opening an RFC. But at that point the time wasted is on the implementation if it isn’t aligned with what the community wants and the RFC gets rejected.

You can definitely post PR links to merged RFCs. Then lots of interested people will have the opportunity to participate in the implementation. I have seen this done and would personally recommend doing this so that people can follow along with the feature even once the RFC is accepted.

12

I cannot post PRs to RFCs that does not exist? That does not make sense.

When I am working on a new thing, I cannot post a PR link for discoverability in a non-existent RFC and if I am going to do an RFC, I can, as well, go the extra mile and show the implementation before confirming the RFC.

I’m not interested into uninformed consensus building on technical decisions.

You wrote:

Well, my time is precious, and I would like people to respect it, so if I engage in good faith in RFCs on such high profile things like language versioning. There’s no reason for the work to go to waste.

When you say things like those “what are the downsides of talking a lot and doing nothing?” you are telling me “what are the downsides of you losing your time over something that you care about and being powerless about changing things?”, at least, this is what I perceive.

So this is hurtful because this is a lack of respect towards my engagement in the community and something that doesn’t make me want to partake in the RFC process at all.

13

This approach, unfortunately, is how we got the Flakes RFC. And in general it has a high risk of prioritising one, not universally shared, use case — and miss the implications for other cases.

It is true that the process we have has a very strong eventually-do-nothing bias; the previous process version had an even stronger bias in that direction that was insurmountable; but I am not sure how far we can reduce this bias before going too far («large changes never get finished» has its own benefits in terms of downstream churn, and almost all users have to have some downstream Nix code), and thus how to design improvements without overshooting.

14

I’m not sure if this is how we got the Flakes RFC, did we? I never heard that Flakes was implemented as a branch and it was shown before we merged the Flakes RFC.

I saw that the Flakes RFC was integrated in an experimental feature. So I feel like this is an orthogonal point.

Also, overall, I think Flakes is an absolute outlier of all RFCs and should never have taken as an example of anything.

I am still not sure if I have seen RFCs dying because of the implementation difficulty, to be honest.

15

The Flakes RFC based on the documentation was abandoned, not merged; the new shake-everything-over RFC is kind of not just about flakes and kind of doesn’t even say that much how things are supposed to end up.

Given implementation is not a prerequisite for acceptance, it is hard for an RFC to die based just on implementation difficulty. I admit to having abandoned an RFC because the feedback was towards an implementation that I would not like too much even if it appeared on its own, and so not willing to fully figure out to the level of at least writing the RFC on it well (which would be below full implementation)

1 Like
16

Right, I said “before we merged the RFC” as a universal quantification over the empty set, that is, a vacuous truth.

Not sure what I should conclude from that though, maybe it’s a good thing?

1 Like
17

Well this is as close an example as I have to the idea being well-received (it was) but implementation being the stumbling stone.

A full implementation of the original plan would definitely be wasted.

On the other hand, RFCs mostly failing is in a sense a feature not a bug…