I agree that from a user experience point of view flakes are conceptually needed even when not activated as a command, one example is of course the new semantics of nix search, but is keeping the 2.4 release “in limbo” a good answer to that?
The people that want to use the 2.3 release can do it by explicitly installing it.
From my point of view there are so many new things in the 2.4 release that some breakage was inevitable and from what I can understand this is due to the exceptionally long time since the last major release.
The other thing I wish I knew better is the “economics and funding” of the Nix development and its relation with the NixOS foundation, if they are related at all. I’m at the margin of the community but sometimes it seems to me like there isn’t a common goal between the people who spend their time developing Nix (for which I’m very grateful) and what the community mumbles it’s better but maybe without having the resources in terms of (paid?) time and knowledge and skills to sit down and actually make it happen.
Maybe a better funding could help in steering the development towards a better common goal?
This captures really concisely something I noticed about Flakes when I first tried them, prior to the 2.4 release:
Because the Flakes interface requires you to write many more things explicitly, having a concise and portable flake.nix file requires you to experience Nix-as-code to a much greater degree than pre-flakes Nix configurations, where if your needs are simple, Nix can really feel like a mostly inert configuration language (even though, of course, it isn’t).
Imo the interface presented by flake-utils-plus goes 60-70% of the way toward restoring that balance where when your needs are simple, Nix can feel like a simple configuration language again.
(I say all of this as an overall fan of the Nix language (and lately, of Flakes, too), and as someone who picked up Nix in college when I had little to no FP experience.)
I’m considering myself a newbie. When I (recently) switched to NixOS my only requirement was that it should “just work” out of the box, but actually this wasn’t the case. From my POV Nix always was an experiment and I doubt this will change in a foreseeable future.
I’ve started from its UI, then I needed to learn its language and now I’m even reading edolstra’s PhD. I think it’s worth that time investment. After I’ve upgraded to a previous version of NixOS I’ve already required to adapt my config.nix to account for incompatible changes anyway. I’m still considering myself an end-user, but I don’t think that anyone could potentially avoid such breaking changes anyway. That’s how NixOS development is done. Just look at GitHub issues and PRs, many are untouched for years.
I think that too rosy expectations are just unrealistic. You just can’t beat QA of a well financed Linux distribution. It’s already amazing how much is done with so little financial support.
Just my two cents.
That’s actually a very good point.
It’s interesting to note that most of the “big” recent Nix changes (the “new and exciting features”) happened because some company funded the work (Flakes initial design and work by Target, CA derivations first by Groq and then Tweag and I think the IPFS foundation, Eelco’s current work on ACLs by Flox, etc.).
Bug getting funding for just the grunt maintenance work is a much harder thing to get because it’s not really something that most companies will be willing to spend their own money on. I think that most of the companies that actually employ the ppl working on Nix also give them some time to work on some more generic Nix maintenance (at least that’s what Tweag does), and that’s probably how most of the Nix maintenance happens, but these are generally small-ish consultancies that only have a limited amount of available money.
Maybe that’s something where the community could help indeed, like what’s been done with Nix 🖤 macOS - Open Collective
I kind of agree with @thufschmitt that maintenance is an issue. I don’t think that money is a solution to the problem at all right now. At least not until we solve various other issues.
The primary issue from my perspective is that there aren’t enough people empowered to help move the Nix project forward.
We rely on a very small set of people, where only a few are actually merging changes that aren’t typos or CI fixes.
Guiding contributors through the process of making a change is crucial. If the experience isn’t great, they will not stick around. One of the key features of NixOS that I often hear is how easy and welcoming it is to contribute (in comparison). When I started out, that we certainly true. GitHub is one ingredient that currently makes it very easy to contribute to both Nix and Nixpkgs (and I’m usually the one favouring Emails over PRs…) but it isn’t a silver bullet.
The biggest issue right now is human bandwidth. We are essentially always waiting for Eelco to approve anything. That isn’t a feasible long-term situation to be in. Making everything depend on him isn’t in the best interested of the project. (This is not because of distrust, but because we can’t clone him.)
The following points are what I think is crucial right now (in rough order of priority)
- First and foremost the most critical part: We need a team of empowered maintainers that have the capacity & capability to look at changes, ideas, issues and are individually in a position to accept those. This means merging changes that meet the contribution criteria, considering a bug a major blocker and so on. This team has to be aligned about the future of nix and the contribution guidelines they enforce.
- Contribution guidelines have to be documented. Right now, we only have documentation on how to build Nix in various ways. What are the standards that are aimed for? What is a valid change? Should I open an issue before implementing a feature? What should my commit messages look like? Is “Fix issue.” a valid commit message? What checks should I run before submitting a change. Are all platforms equally important? Do I have to write tests as part of each change? What do I do if my change hasn’t been reviewed for a while? Whom can I ping? Documenting requirements and goals allows involving more people, as you can always refer to the documents when in doubt. Changes can always be reverted when they aren’t actually aligned. Not formalizing what the common goals are might lead to chaos and stagnation instead.
- Project vision should be communicated in regular intervals. What is going well with the project? Which issues are (high) on the radar of the maintainers? What are the features that we identified are most significant to be added next? Keep the community involved. Be vocal about where the journey is going. It might encourage someone to step up and look into specific topics.
- Enforce peer-review for all changes that are made. This is the part where the aligned maintainer team will be reviewing each other and “outside” contributors. Everyone should adhere to the contribution standards and follow the same process. The response times here are as crucial as for “external” contributions as they are for changes within the team. There shouldn’t be two classes of contributions (ones that just push/merge and those that always have to wait for feedback or approval).
- Define the maintainer team lifecycle. Who decides who is in? What are the requirements? When does someone drop out of the team?
- Define a stability policy for nix. The policy should answer what parts can be relied on to work across all versions of Nix. For example, if the language and the built-ins are permitted to break between releases or if the CLI is permitted to change or if expressions are supposed to be reproducible, what the … Which kinds of changes are valid release blockers? This will also help users understand what they can rely on between releases and what might change. This will also guide maintainers in deciding whether a change is a valid fix or breaking compatibility guarantees.
I admit this will be a lot of work, but it is work we can’t postpone forever. Nix has a lot of potential but is really missing a governing body. Most of the time we focus on the technical issues, but we should also consider the human and community side of things. To some degree, Nix is victim to its own success…
Yeah I basically agree with @andir. There is actually a virtuous cycle where newly deputized maintainers will feel more comfortable merging bug fixes and “uncontroversial” changes and fancy features work, and more people will contribute once the lead time in getting such things merged is less bad.
The new Haskell Foundation planning on taking a more active role than the NixOS foundation is perhaps something we could persue next, but first we need to empower more outside contributors to Nix. GHC has a hellish CI to get PRs past, but even so has more outside contributors than Nix itself. We should at least reach that level of diversified contribution.
As I think has been outlined in some of the responses, it might be more accurate to say that flake.nix has been designed to accommodate a smarter Nix CLI, by defining a schema for the kind of metadata it takes to support a smarter CLI. (It’s probably fair to say that this has involved some entanglement, as flakes and the new CLI have been designed together.)
@thufschmitt emphasized this here:
I think that if we think of flakes essentially as metadata designed to support (1) pure evaluation and (2) a better CLI, it becomes much easier to motivate flakes.
Because I think the discussion in this thread is intended to be more focused on process issues that are not necessarily specific to flakes and its implementation, I’ve split the elaboration of this thought into a separate thread. (Mods, feel free to merge it if putting it in a separate thread takes up too much space.)
I don’t think what I have in mind is especially novel or non-obvious, but I do think it’s potentially relevant and I’d be grateful for any thoughts on it, especially from regular Nix contributors and the signatories of the OP.
This is my exact sentiment. I started using Nix/NixOS fully around the time that flakes were proposed at NixCon 2019, so I started my configs fully with nix 2.3. I had a hard time understanding the general concepts (let alone, what was a bug an what was just a user-induced error). I just started to learn and use pinned nixpkgs in my system configs when I decided to move to flake configurations. So I never really finished even leaning the way to deal with the old interface.
But the flakes/pure approach to configurations was a much, much much more simple introduction to nix, especially given introductions like nixflk/DevOS. Before flakes, there were 15 (exaggerated) different ways to pin and reproduce configurations and no official answer to the problem, so all new users would have their own factions depending on their choice.
I think the conclusion is flakes sucks for master pre-flake Nixers, but is a blessing to new users introduced to nix.
However, I agree that we shouldn’t pull the rug from under all of the pre-flake nix users with a stable release of NixOS defaulting to nix 2.4. The RMs made a good decisions with this.
Now let me preface this, by saying that Nix has come a long way, especially in the last two years and especially in terms of knowledge transfer to, and communication with that part of the user-base that stays relatively silent, does not live and breathe the nixpkgs repo and does not spend every free second on discourse, discord and irc.
Just to add my voice, I think you guys are doing great and the “breaking changes” in 2.4 might have been blown a bit out of proportion as seen from the perspective of such a mere mortal. Nix 2.4 took many steps in the right right directions and flakes are very useful, not to mention the manual and the website. If I started out with nix today, I’d be in a much better place, than 5 years ago, when I actually did. A bit of breakage was to be expected and will be sorted out, some stuff just needs to hit the broader public, which in turn will create pressure or take up arms for it to get sorted out. That is fine! Things take time and problems are necessary to make progress.
However, that quote here resonated with me, though:
The primary issue with nix, from my point of view as a long-time silent user, is that it is really hard become one of these empowered users. While it is true that the bus-factor is very low and the whole project seems to stand and fall with maybe 20 people pouring their blood and youth into it (who seem to burn out and rotate quiet quickly, I might add), that just seems like a symptom to me.
You see that quiet often in the software industry. When a software company grows too large for it’s own shoes, most of the time, that is evidenced by growing workload, difficulty in hiring, overworking their employees and not making much progress despite everyone being very busy. That also means the turnover is rising as demand grows but profit shrinking. In short, things just don’t quite scale as they used to and everything gets inefficient. That is mostly due to lack of switching to processes that scale better, lack of early knowledge transfer and failing to reduce complexities right as they occur at the start, i.e. technical debt “we’ll do it later” type of things.
I’m not saying that’s the case with nix, or that I have a solution. I’m just saying that if I was someone to pour my free Friday into the nix project, I’d fail, because there’s no one who’d have the time to mentor me and no track or program in place that would make me effective quickly, let alone painlessly. There’s just so many inter-dependent features and construction sites to look out for, in short, it’s such an enormous monolithic project, that it’s hard to sink your teeth into just a small part and learn that. And the people in the know, who could make sense of it, are putting out fires and doing cutting edge (read niche) stuff, when maybe training apprentices, writing tutorials and implementing processes would be more conducive. Nix has exploded technically, but organization, planning, training and communication has not not kept up. Just like tech start-ups can not keep being start-ups forever, nix can not keep being a cool tech-demo of loosely organized individuals forever. But you people know that, and you’re making progress with nix 2.4, or the summer of nix, or the marketing team, docu and website. These efforts just need to keep pace with the technical side of things.
Edit:
Let me link Graham’s epitaph here.
With the next release of NixOS on the horizon, I’d like this group to take a look at where we are and start to discuss if the correct fixes have been put in place, or to put some effort to fix the things mentioned in the original post. I’d like to avoid a last-minute objection for 22.05. Are there any concrete steps that need to be taken or working groups established or RFCs written?
Also, we forgot to update NixOS release notes, which still misleadingly feature Nix 2.4 for NixOS 21.11.
Just to bump this thread (and reiterate @tomberek’s last query). Whats the state of Nix going to be in 22.05 ?
I don’t currently use flakes (haven’t had a need,) but am not opposed to a bit of effort to bring my configurations up to spec if need be. I’m just curious what the upgrade path is going to look like?
I for my part am still on a version of nix 2.3 with custom hand picked (and ported) fixes. In my experience it is the only stable version of NIx we currently have. I occasionally try newer versions (across devices) and usually just run into issues non-stop.
I’ll not go through the list of things that we initially pointed out. Most of what I cared about hasn’t changed. I would put in the time but project governance is just terrible.
Is this still valid? Will 22.05 ship a newer nix?
yes, nix 22.05 will ship with the latest nix. Most of the people concerned with the things in this post had voiced their complaints, but have not seemed to champion any of the issues to get them addressed. See @tomberek 's post from almost 3 months ago above. I think this was mostly just voicing unhappiness with how things were trail-blazed forward with flakes.
With 22.05 using nix > 2.3, does this also effect the use of the current standard commands (nix-env/shell/build/etc.) or configuration (“configuration.nix”) ? Or do these remain unaffected? i.e. Is this next upgrade going to require configuration changes?
nix-* commands stay the way they were, this only affects experimental nix command.
Also, for future reference, all the flake-related commands are still firmly marked as experimental, and may even be scrapped entirely: Experimental Commands - Nix Reference Manual
There was a timeline published on how to get those things to maybe become stable eventually, but it’s not a 6 month development deal 
Maybe in a year or two.
Even then, a deprecation of the old commands isn’t even in the timeline, and would be very clearly announced beforehand. I wouldn’t expect anything like that anytime soon. It would be called the 3.0 release too, the nix command is marked as nix-3 here and there, after all.
There might still be some fallout from the upgrade, just as last time. The nix version used by default in nixpkgs is now some 5000 commits out of sync with mainline. Eventually the upgrade will need to happen, once that band-aid is pulled off things will probably get smoother (not least because recent version changes have been much smaller in scope).
I think the nix team deserve a lot of credit for their continued maintenance of the 2.3 branch, they’ve been maintaining it alongside other development for 3 years now. If newer nix versions have issues, report them so that they’re known, and if that doesn’t move quickly enough for you, try to help out - it’s not exactly a Google sized team working on this.
Good stuff, as long as the age old adage “it worked before, it should work now” continues to hold true [with regards to standard nix-* utilities] there shouldn’t be a problem.
@TLATER Thanks for clearing that up. As long as the “new” nix command is marked experimental, standard unstable guarantee’s (what guarantee’s ? 
) should be expected.
Its possible you may experience one of the regressions in the original post here, probably related to fetchgit or fetchtarball, but it looks like there’s a work around or the issue could be addressed…