Nix release schedule and roadmap

[edolstra]

Nix release schedule and roadmap

TLDR: We’ll release Nix 2.4 soon and then move to a 6-weekly release schedule.

Release schedule

The last major Nix release, 2.3, came out in September 2019. Since then, there has been a huge amount of work on the Nix master branch, including major improvements to the new command line interface and new experimental features such as flakes, the content-addressable Nix store, and recursive Nix. This forces more and more users to use unstable Nix releases. So a new release is very much overdue.

However, the master branch is not quite in a releasable state yet. Therefore we’ll stabilize master in the next few weeks and then cut a 2.4 release. If you want to help out, please have a look at the 2.4 milestone on GitHub.

Going forward, we want to avoid having such long gaps between releases, since it’s bad for both contributors and users that there is an unbounded amount of time before a new feature shows up in a stable release. The thing that has historically caused long gaps between Nix releases is new experimental features landing in master that we weren’t quite sure about, and doing a new release meant having to support these features indefinitely. However, Nix 2.4 introduces an experimental-features setting that enables us to add such features in a way that they can be changed or removed, while still getting feedback from adventurous users. So long as experimental features don’t cause breakage in stable features, it’s fine to merge them into master and make a new release.

Therefore, after the Nix 2.4 release, we’ll switch to a 6-weekly release schedule, meaning that we’ll cut a new release from the master branch every 6 weeks. (The 6-week interval may be changed in the future if it turns out to be too long or short.)

Releasing regularly means that we have to keep the master branch in a releasable state at all times. For contributors, this means that PRs with user-visible changes (such as new features) should update the release notes.

One open question is what versioning scheme to use. For now we can stick with the current scheme (i.e. only bumping the major version if there are incompatible changes or major non-experimental new features), but in the future we could switch to date-based versions (e.g. Nix 21.07).

Roadmap

Currently the Nix project doesn’t have a clear roadmap. Roadmaps are hard to make because they depend on what people are willing and able to work on, which is not always easy to predict. However, it’s still useful to have a roadmap to provide a “north star” for the community. So with the appropriate caveats, here is a rough roadmap for the next year or so:

• 2.4
  • Flakes (experimental).
  • Overhauled nix CLI (still marked as experimental, but we’ll try to avoid major changes after this release).
  • Content-addressable Nix store (CAS) (experimental).
• 2.5, …
  • 6-weekly releases as needed.
• 3.0
  • Flakes and nix CLI declared stable.
• 4.0
  • CAS semantics and interfaces declared stable.

This is really just a starting point for a roadmap. Other items that could be added in the future (depending on what people are interested on working on) are UX improvements, rootless Nix, Nix as a build tool, Nix for HPC, Nix modules and supporting new configuration languages such as Nickel or TOML.

Any comments you may have, especially ideas about what you consider important for the Nix roadmap, are very welcome!

[abathur]

@edolstra

…we’ll stabilize master in the next few weeks and then cut a 2.4 release.

…after the Nix 2.4 release, we’ll switch to a 6-weekly release schedule , meaning that we’ll cut a new release from the master branch every 6 weeks.

Great news on both counts, I think.

I’m not sure it needs to be in a milestone or on the roadmap, but since it may help with keeping master release-ready: I’ve been trying to fish around for anyone interested in leveraging the installer test support merged this March to build an installer test suite:

Installer test suite? small project(s), high-leverage, help wanted

@domenkozar landed work in March that establishes basic Nix installer tests on Ubuntu and macOS (how to set them on your fork). This is already a Big Deal in my eyes, but there’s still a lot of low-hanging fruit here. If you know anyone interested in working on (or promoting/boosting) a small high-leverage project, I think this is a good candidate. It has a few distinct bits, so I think a few people could make short work of it without tons of coordination: Add single-user install tests. I…

[jonringer]

A major dilemma for producing more nixpkgs videos was what cli to use. Once it’s stable, I’ll have to find another excuse…

[nrdxp]

@ryantm could you pin this?

[vcunat]

@edolstra

If you want to help out, please have a look at the 2.4 milestone on GitHub .

When I look at the milestone, it does not feel like all that could be resolved “soon” (meaning within a couple months). I’d like to be wrong. Or perhaps I’m imagining stricter sense of the milestone than intended?

[edolstra]

@vcunat

it does not feel like all that could be resolved “soon”

There are a lot of flake-related issues that are not really blockers (since they’re not regressions), so they could be moved to the next milestone. There are also some possible CLI changes (like renaming nix shell) that are not essential but would be nice to have in 2.4 so we don’t have to have breaking changes again afterwards.

[mkg]

@edolstra

This is really just a starting point for a roadmap. Other items that could be added in the future (depending on what people are interested on working on) are UX improvements, rootless Nix, Nix as a build tool, Nix for HPC, Nix modules and supporting new configuration languages such as Nickel or TOML.

I attempted to use Nix for HPC and had problems because it wasn’t rootless. (I used proot and it interacted poorly with the LMod module system already on the clusters.) I’d like to see both of those improved in the future. I am also interested in Nix as a build tool, especially for cross-compilation.

[emdash]

This is really good news.

+1 for date-based versions, and a predictable, periodic release schedule.
+1 keeping master in a releasable condition at all times – too many open source projects make the mistake of using master (or whatever the default branch is) as a development branch, leaving the user to sort out for themselves which branch or tag represents the latest stable source tree.
+1 for feature branches and / or PR-based workflows.
+1 for opt-in to experimental features without having to switch to unstable.

If you’re willing to commit to 6 week releases, great! If not, even a 6 month release schedule is better than the traditional “we’ll release when it’s ready” release policy.

Suggestion: rather than a global “experimental features” flag, require that new features be placed behind their own feature flag. Make any dependencies between features explicit. I’m totally fine with getting an error message like “–frobulate-bloblings requires --confabulate-glamifactors”. And then I can at least make an informed choice. But I get nervous when I have to enable a global “experimental” flag just to try one feature I’m not sure about. If you want to provide a global “–experimental-features” flag, it should map to some documented set of features. And don’t allow anything to depend on “–experimental-features” directly, or you’ll end up back where you started.

Suggestion: PRs need to provide any changes to documentation before they can be merged.

Suggestion: Just don’t have a roadmap: Whatever gets merged between releases is what goes into the next release. Have a backlog of issues, triaged in ways that help a potential contributor identify what they might want to work on. Use tags to help. Things like “good first issue”, for example. For large units of work, or cross-cutting concerns, use “epics” or adopt an RFC system, and for these it makes sense to have a target release.

Suggestion: Adopt a policy and introduce a mechanism for deprecating features.

For an individual package, such as a tool or a library, semantic versioning might make some sense. But for a distro (or meta-distro), which aggregates hundreds to thousands of packages, regular releases are vital, and date-based versions are easier for everyone to understand. It’s far more important and useful to communicate how old a release is than anything else. I much appreciate not having to look up the release to figure this out.

You’re competing against hundreds of other distros, and believe it or not, these little things are hugely important. Anything which creates friction is going to cause people to move on. Anything you can do to reduce friction will help drive adoption.

[emdash]

I’ll go one step further and say that you should release a new stable release sooner rather than later. Don’t wait for things to be “fixed” or “ready”.

Instead, introduce the feature flags / feature gate mechanism. Then place anything that’s still in development behind those flags. Then release. Then spend the next six weeks fixing whatever issues begin to emerge. If you’re really going to release every six weeks, it’s fine if some stuff ends up being broken at first.

It’s not so much that we expect things to be pain-free. But you want to feel like the pain you’re experiencing is temporary, and that your bug reports are being at least acknowledged. That you’re not stepping on the same trap that hundreds of other people have already stepped on. In short, that your pain is useful to the developers.

[colemickens]

If I understand the ask, that’s already --experimental-features works. It takes a list of features to enable.

From a script of mine:
--experimental-features "nix-command flakes recursive-nix"

[j-k]

@edolstra

rootless Nix

portable-nix is great for now but official rootless support will be big (at least for what I’m doing)

[davidak]

@edolstra

One open question is what versioning scheme to use.

The obvious answer is SemVer. It is a well known standard and tells the user something about the update.

CalVer only makes sense for special cases, like a linux distro.

Don’t use any other scheme without a very good reason!

Does anyone has a different opinion?

[Ericson2314]

Very happy to see we’re going to exericse on a regular release cadence (and get a ton of regression bugs to fix! :)) before stabilizing any new features!

[loafofpiecrust]

I’m just reading this fantastic news. I definitely agree with @davidak on using semantic versioning for Nix itself (while calendar versioning makes sense for NixOS) so that we can distinguish between major and minor changes. The Rust release model is a great one to follow; they use the concept of editions which are calendar based to segment the usage of breaking changes, while following semantic versioning for the compiler broadly. This produces pretty solid forward and backward compatibility in the compiler.

Overall, +1 to release schedule (rust has been going strong with this!); +1 to master always releasable, that’s what it’s for IMO.

[andir]

How will the new release schedule be compatible with NixOS stable versions? Can it be expected that the version that we have on branch-off will be supported by Nix developers for an entire NixOS release cycle? Do we have to upgrade Nix from release to release in the stable NixOS release (which might potentially deprecate/change Nix feature within a NixOS release)?

[edolstra]

@andir Good questions. To hash out these questions I created an RFC about the Nix release schedule here: [RFC 0106] Nix release schedule by edolstra · Pull Request #106 · NixOS/rfcs · GitHub. I think we should do bug fix releases for at least the Nix release used by the latest NixOS release.

[jonringer]

As promised, I made a video on using flakes in out-of-tree code