tl;dr: Nix maintainers are working towards stabilising
nix storecommands, and plan to break down theflakesexperimental feature into components that can be stabilised one by one.
Background
The Nix team was founded in October 2022 in order to support development of the Nix C++ codebase.
The team started out on general topics to build trust and establish an effective work process, and we have seen good progress. We are meeting twice a week, and work on the agreed-upon issues in-between meetings. Those general efforts have been a proving ground to determine if we are able to address larger issues, as we have described in our first half-year report.
After RFC 136 was merged, we started to work on incrementally stabilising the new command line interface (CLI).
Users and developers have long identified this topic as critical to easing Nix adoption. We recognise that the lingering dissonance between the classic and the new experimental CLI is a distraction from maintenance and feature development. The situation does not help with easing and guiding the work of contributors, either — a need that emerged clearly from the 2022 community survey, and this team took on serving that need as one of our primary responsibilities.
While work on the new CLI presents a substantial effort — and on the team we continue to debate over many open design questions, as we all really care about getting it right — we agree that:
- The current narrative on the state of the CLI causes confusion.
- We want to conclude the experiment by
- Incorporating the vast amount of feedback we already collected.
- Stabilising those new features we feel are settled.
We all have a strong interest in getting this done. Therefore we prioritised this narrow subject in the past months, and plan to continue on that path in the foreseeable future.
Planned steps
The current stabilisation process is as follows:
-
Formally stabilise the “hierarchical command line structure” first, without any commands in it.
This would amount to updating the manual to remove the warning for the main
nixcommand, and highlighting experimental subcommands in the overview explicitly.nix --helpandnix replare have never been guarded by the experimental feature, and all other commands will remain experimental for now. -
Prepare stabilisation of commands controlling the Nix store (following RFC 136), and several low-complexity commands.
The goal is to make each command in the set of candidates ready for removal of the “experimental” flag. During this step we will focus on ensuring meaningful and correct behavior, including interactions with options, environment variables, and files specific to the given command. While we want to keep feature mappings between the new and the classic CLI in mind, feature parity is currently not a priority.
We will also gather issues on global concerns to follow up on in the next step. Examples are flags that apply to many commands, such as
--print-build-logs, or cross-cutting concepts such as installables. -
Stabilise the global options and settings relevant to the set of candidates prepared in the previous step.
Those options and settings apply to multiple commands and effectively can’t be changed in substantial ways after stabilisation.
-
Stabilise the prepared candidate commands in succession.
This will be preceded by reviewing remaining issues and evaluating all the feedback collected in the process so far.
We’re tracking overall progress in NixOS/nix#7701, expecting many additional issues and pull requests to be opened as a consequence. We anticipate some breaking changes to experimental nix commands, and will take great care not to break the stable CLI.
We don’t expect to do this perfectly, and instead intend to keep the disruption to a minimum while carefully balancing the conflicting goals of getting a large portion of it done and avoiding long-term maintenance burden. Therefore, we ask for your help with reviews, both of the process as well as the actual changes. We will devote time to review and merge code contributions that address any substantial issues that arise.
We hope that using the approach outlined above, we will avoid getting sidetracked by unrelated considerations.
How does this relate to flakes?
To be clear, “flakes” is a bundle of behaviors subsumed under the flakes experimental feature flag.
At the moment, the implementations of the new CLI and flakes are somewhat intertwined. For now we mainly focus on stabilising the new CLI. We expect the process to clarify and strengthen architectural separation of concerns in the code, and help us find ways to proceed with flakes.
We acknowledge that flakes have been around for a long time and that right now it may not look like we’re any closer to stabilisation than we were two years ago. Yet, we’re determined to resolve the ambiguity around flakes as quickly as possible, following the roadmap outlined in RFC 136.
We have to be honest with what to expect here: Due to the nature of our work, we can’t commit to a timeline. But we can commit to a process – such as working on the topic together once a week, and asynchronously as far as time allows – and to intermediate goals that are realistically achievable given our limited resources.
From experience, as a team and as a community, we have been successful with moving forward in small steps, making uncontroversial changes we can all agree to support long-term. We particularly care about long-term stability: Nix expressions that worked ten years ago should continue working today and into the future. (We’re yet to formally define what backward compatibility means though, and to find pragmatic trade-offs to account for the development effort associated with it.)
Our approach is therefore to reduce the surface of experimental features incrementally. We already started that process in the past weeks, and reviewed builtins.fetchTree to be stabilised on its own. It has been the underlying implementation for most builtins.fetch* functions for a long time now, such that much of its logic was already stable despite the interface being experimental. It also serves as the backend for flake inputs. Thus, by stabilising fetchTree, we’ll stabilise part of flakes, and do it in a way that this part can be used in isolation.
As a first step, the next Nix release will see a separate fetch-tree experimental feature that users can test on its own. Documentation still needs improvement, and we have to sort out some design issues in a way that leaves room for future evolution.
The envisioned stable interface of fetchTree will initially only support structured parameters – but not the URL-like syntax – and the file and tarball source types. The other source types will still be guarded by an experimental feature flag, to be stabilised one by one. The URL-like syntax will likely take more time to converge, as it’s a more superficial aspect of the system.
Our tentative plan is to gather experience with that process, and then apply it to the other constituent parts of flakes to be considered for stabilisation, among others:
- the lock file format and semantics
- the
flake.nixfile format - the URL-like syntax for remote sources
- the flake outputs used by the
nixcommand
How you can help
For far-reaching changes such as with CLI and flakes stabilisation we’re going slowly because we don’t want to commit to things that are prone to become an untenable maintenance liability in the future. The delays seen so far are not only due to implementation but also due to decision making. Right now the Nix team is busy with both making design choices and implementing them.
There is still only a handful of people who understand the C++ code and deeper implications of particular changes well enough to work with it effectively. We want to use the stabilisation review as an opportunity to share knowledge within the team and the community of contributors.
If you want to help us moving more quickly, here is how you can help right now:
-
Work on issues with labels
idea approvedorRFCNote that label links in this post may get out of date. Please refer to
CONTRIBUTING.mdin the Nix GitHub repository for up-to-date guidance on making contributions. -
Help find solutions to blocking issues and open design problems.
We currently don’t track these explicitly. Please check our meeting notes to follow of the state of discussion. Feel free to comment on the relevant Discourse posts or GitHub issues.
This setup is not ideal, but we’ll be working on promoting a more structured design process as we go. -
Contact us if you have a significant amount of time or resources you can devote to Nix development.
If you are a decision maker in an organisation, consider sponsoring a Nix maintainer or trusted community member to enable them doing focused work.
Thanks for your patience and for your support!
@edolstra @Ericson2314 @fricklerhandwerk @thufschmitt @roberth @tomberek