I didn’t read the article, nor replies here yet, but just from the title:
If Experimental doesn’t mean unstable, why are we talking about stabilzing flakes? Just release 3.0!
For me there are roughly three grades of stability:
“experimental”: Things may break any time, in the weirdest ways, large and loudly, or silent and subtle.
“unstable”: Most parts are actually stabilized, but no guarantees are given to keep them. There will be some grace or deprecation period and a proper changelog
“stable”: No breaking changes that not also result on a new major release
So, yes, you are kind of right when you say “experimental does not mean unstable”, it does actually mean that you have much more freedom breaking things than the “Nix Team” is willing to do…
I don’t consider this is a problem. It’s just a situation. It has benefits. For example, more stability, accuracy when sharing binary caches…
You can configure nix to duplicate files with hard links instead of copies, so multiple versions of the same dependency not always mean duplicate disk space used. Actually, since I discovered this, I stopped worrying too much about lazy trees (which will be a benefit nevertheless; but it doesn’t seem like a flakes 2.0, but a flakes 1.1).
I’d be wary of transplanting Rust lessons to Nix directly — superficially, Rust and Nix follow similar processes, but, while the form is almost the same, the content is very different.
Both Rust and Nix have teams, but in Rust teams are well-aligned and self-selected, while it seems like there’s a pretty strong disagreement wrt Nix future inside the Nix team itself.
Both Rust and Nix have an RFC process, but in Rust the team responsible for a particular area votes to merge or reject an RFC, while in Nix it seems that the RFC pertaining to a particular team is voted on by a different set of people.
Both Rust and Nix have unstable/experimental features, but in Rust unstable has a very clear meaning (available only on a separately distributed “nightly” channel of Rust, which is a separate binary; attached to a tracking issue which charts the path to stabilization; for big features which are not explicitly tabled, actively moved along to the stabilization). I am not sure what “experimental” means in Nix to be honest! It seems that experimental flag is a much smaller speedbump than rust unstable, the real-world usage is much larger and much more production oriented and, at least a little bit, experimental is a way to route around deadlocks in the decision making process.
Teams in the Nix ecosystem are also self-selected. That there is some disagreement on strategic questions in the Nix team is a fair assessment, but I find that the team is a reasonable approximation to being representative of the wider Nix developer community in terms of opinions. In fact, I take particular care to listen to as many people as possible to collect feedback on what we’re doing.
We’re also getting more effective with deciding on ever larger issues over time. @Ericson2314’s work on pushing forward uncontroversial, incremental changes sets good examples of what can be done to keep moving in the meanwhile.
In fact, we have established a ritual to add an idea approved label to things we agree on, and will make prioritised reviews when someone implements the desired changes. It’s documented in CONTRIBUTING.md, which links to our process documentation, so this information can in principle be found. We may want to add it to the manual as well for better discoverability, or anything else you think would make it easier for contributors to learn about it. Feel free to open a pull request if you want it to happen faster – I will help to get it merged.
In practice I don’t really see RFCs getting past the respective code owners. But they can easily get blocked by other stakeholders, although it’s decided ad hoc who that is and sheperds are usually self-selecting. We may want to write this down to set more realistic expectations. @piegames do you think we can just amend the RFC process description without jumping through all of the hoops?
This is defined in the Nix manual: Experimental Features - Nix Reference Manual
Please open issues or directly submit pull requests if there is something to be clarified about that.
This is only true for flakes and nix-command, which have been heavily advertised for years, by @grahamc, @edolstra, and @garbas among others, instead of being talked or written about in a factual manner. Other experimental commands are surprisingly rarely used, as the upcoming community survey results and issue tracker activity show. Up to recently, most people didn’t even know that some of them existed because there was no listing available.
Thanks, this is super useful and adds a tonne of context!
I think this is a bit different for Rust — I would say it’s a not a goal for a Rust team to represent wider community. Teams listen to community feedback, and in turn are required to make their reasoning public to the community, but decisions are based on within-team consensus, not on the community consensus. Usually the two are aligned anyway, but sometimes teams make decisions which are at odds with some fraction of the community (the three most memorable cases are the syntax for .await, the syntax for ?, and the name resolution rules around rust 2018). Rust is more of benevolent hereditary aristocracy than a democracy.
This is only true for flakes and nix-command ,
I think the experimental flag was added specifically for these two historically. So, yeah, I should say i don’t understand experimental applied to these two particular things.
It depends on what exact change in question is; I haven’t fully followed the discussion so far. Changing the process description to better match what is currently done in practice most of the time should be fine. Changing the process itself may be acceptable for small and uncontroversial things, but generally requires an RFC.
The RFC process currently has no real decision mechanism for controversial topics, and this is a difficult issue to solve. See Improving our RFC process where it has been discussed before. Honestly this is a topic that probably requires a lot more discussion before even making it into an RFC.
I would say it has, but with no guarantees of convergence. If shepherds who originally held opposing views eventually reach agreement (which includes the other shepherds and the author), this does help with legitimisation, and sometimes it works (at least in low-heat issues with diametrically opposed views without too much emotion attached).
If a deadlock is indeed persistent and is not resolved by a few hours of negotiations and a few tens of hours of research and preparation, the process indeed reflects the split.
@matklad I think some of deviations can be explained purely by the order in which things occurred:
Rust
Graydon BDFL
Mozilla-led project
Core Team + RFC process with Core Team review
Other teams
Mozilla steps back, Foundation etc.
Nix
Eelco BFDL
Nixpkgs grows into anarchic thing, Eelco takes step back from it
Very hands-off Foundation
Attempts at Core Team for Nix, but it doesn’t stick
RFCs
Bigger foundation, new teams that stick
The big differences as such are:
While Nix is very consciously imitating Rust with RFCs and experimental features, our RFCs predate teams and so this more informal convening of shepherds process was created instead.
Authority in Rust was centralized to begin with, and so “sovereignty flowed downward” (“God” → “king” → etc.), whereas Nix had ceased to have any clear leadership by the point of establishing this structure, so sovereignty had to flow upward (Community (??) → Teams/Foundation). So it’s very natural our teams would be formed from people who don’t agree on everything — we have to represent different constituencies to be able to claim legitimacy.
As others have said experimental should mean unstable. If I opened mozilla firefox and they had introduced an experimental feature that removed access to certain API or introduced new ones, I wouldn’t build any production-ready software on it. I’d wait until it was out of experimental and declared stable.
nix is the only thing I’ve encountered that is too afraid to actually take advantage of “experimental” and break things that are behind an experimental feature flag.
This entire situation could possibly be resolved by using semantic versioning, IMO. Version each flakes change appropriately, add a “version” to the flakes file (like docker-compose.yaml, pyproject.toml, etc.) and fail when the wrong flakes version is being used. Then provide a fallback mechanism that downloads and executes the nix binary with the correct flakes version.
And if a dependency in the dep tree uses an unsupported flakes version, then also fail.
Personally, until flakes are declared stable and it really means stable, I won’t use it.
There needs to be a clear pathway, a roadmap where all current issues are listed and described properly so people can effectively create pull requests to solve them. Once all of these major issues have been resolved, it can be marked as stable. The primary problem currently is that everything is all over the place, and because of this mess there is basically no progress made.
How about a roadmap on the GitHub nixpkgs repository, that would be a great way to display the issues and monitor the progress.
I think we should just start doing stuff, and stop discussing forever. This discussion is already going for years now and is stopping everyone. @Infinisil already layed out what needs to be worked on in a great way. So that could be used as a starting point for the roadmap.
What kind of project management does the nix foundation have anyway? Github (finally) has projects with a kanban board and other views (something gitlab has had for years). It should be possible to create a “flakes” project that’s managed by flakes maintainers, add tickets on there and let people volunteer to work on them to drive them forward.
Yes, that is what I meant. A simple roadmap with the major issues that need to be addressed to stabilize flakes. That way work can be done effectively.
If nothing speaks against that, I would open up an issue for a flake stabilization roadmap on the Nix GitHub repository tomorrow. Further details could be discussed there.
I see, but I think migrating over to the new GitHub Projects feature would be a good thing, because this is exactly what it is made for. Then in addition to that, an announcement post on the Discourse could be made to present the new flake stabilization tracker. I guess this would motivate a few more people to start helping out.
The real project is not shiny-driven like «specifically Flakes», the real project defined and methodically carried out by the Nix team is the entire new CLI structure stabilisation, which includes cleaning up the layering and other consistency improvements, as well as checks for omissions.
Thank you for putting this in writing; I have been using flakes a lot and started encountering the same problems, but I thought I was doing something wrong. Given your standing in the community, seeing you make these points validates my suspicions.
