An incremental strategy for stabilizing flakes

Development
1

I think flakes brings us some good things (pure evaluation by default, better evaluation caching, and a much-improved UX for the Nix CLI), and now that Nix 2.4 is out, the question of how to bring those good things to full fruition in stable releases of Nix (without gating them behind an experimental flag) is more relevant than ever.

I’d like to ask whether or not the following resembles what community members and Nix developers themselves have in mind for that process.

I wrote up a long post several months ago, originally for this thread as a step toward a new RFC for flakes, but never posted it because I realized I didn’t really have the determination and skills to see it through or the social capital required to rally important community members behind it. But I think it’s still relevant, and maybe it’s worth putting out there at this point. It’s abridged and adapted below.

The basic idea is if we break flakes into pieces, the motivation for each piece of flakes is a subcommand of the new Nix CLI.

One of the difficulties with the original flakes RFC was its extremely broad scope. Flakes were designed to support the whole range of uses cases covered by the new functionality of the nix command, as well as new functionality in nixos-rebuild and nixops. This makes it a lot harder to answer the question

What are flakes and what are they for?

in a straightforward and concise way. I think if we let the CLI features drive the flakes implementation (as they’ve done in fact anyway), one incremental path forward is obvious.

Proposal: an incremental stabilization/release strategy for Nix flakes

  1. Stabilize a syntax and CLI usage for lockable Nix resource URIs¹
  2. Stabilize a syntax for the inputs schema
  3. One-by-one, stabilize the portion of the flake.nix outputs schema corresponding to the functionality of some flakes-aware nix subcommand
    • optionally stabilize the interface of nix subcommand which relies on it at the same time
    • the first flakes-enabled version of Nix needs to include usable equivalents to at least the following:
      • nix search
      • nix-env -i
      • nixos-rebuild
      • nix-channel

The suggested minimal subset of functionality above requires the stabilization of the flake.nix outputs schema only for:

  • ‘legacy packages’, i.e., packages that are assumed to be organized and named however Nixpkgs currently happens to organize and name them (for nix-env -i and nix search, and the equivalent of nix-channel)
  • hosts (or what is called a ‘system flake’, e.g., in flake-utils-plus) (for nixos-rebuild (and later, nixops, nix-darwin and friends))

Caveats

Obviously all of this can only be a useful suggestion if it’s attractive to the people working on Nix, and especially the people implementing flakes, i.e., Eelco.

But I wanted to put it out there as a way that flakes could move forward piecemeal and democratically, so that the question ‘what is this trying to accomplish?’ is easy to answer at each step.

Questions

Does this kind of release strategy for an official, ‘stable’ implementation of flakes and related tooling seem relevant to this discussion?

Is this kind of incremental release strategy agreeable to Nix and Nixpkgs contributors who are happy to have seen a new Nix release (I am!) and understand the importance of enabling pure evaluation by default, but share the concerns of the signatories in the OP?

Do Nix contributors who are invested in flakes as part of Nix’s future see the incremental release strategy outlined above as helpful to or aligned with their vision for Nix and the way forward, post-2.4?

What concerns related to flakes (other than the long-awaited and then all-at-once release process) are not addressed by an incremental strategy like the one outlined above?

¹: A lockable Nix resource URI is any URI:

  • that the Nix CLI tools and Nix language interpreter can understand as a primitive
  • whose syntax either
    • fully determines the referent of the URI irrespective of context, or
    • which, given a context, can be reduced by Nix to a URI which meets the condition above
2 Likes
2

First of all the definition of pure evaluation itself should be carefully discussed…

4 Likes
3

Can I ask you to be a little more explicit about your concern with the form of pure evaluation currently tied in with flakes, or to refer me (and other readers of this thread) to discussions I may have missed?

Would you like to open a thread about what requirements you think make sense for pure evaluation mode in Nix, or to start that discussion here?

Does what conception of pure evaluation is necessary for Nix impact the appropriateness of the process outlined above?

4

Sort of? It puts a bullet point before all the process.

I separately do not think that the process has a high enough chance to untangle the mixing of concerns in flakes.

(Yes, I have zero interest in flakes achieving anything at all, only in them not breaking things in irreplaceable ways; for example, I expect the platform handling of flakes to be worse than the current meta.platforms approach from the point of view of the fact that things are not always known good or known bad and working around the «no idea» will get harder)

It looks like (not sure if there is documentation) a lot of stuff is defined in terms of Git, without for example trusting all relative paths withing the same Nix store path or something.

Then apparently the command-line argument passing is going to be restricted too? Or not? Or what?

1 Like
5

I don’t agree with that characterization. The scope of the flakes RFC was pretty limited: it provides a standard way to define multi-repository Nix projects. This consists of the flake.nix structure, a way to define dependencies on other repos (inputs), and a lock file format. What it didn’t do:

  • Define any flake outputs. (This is up to tools such as nix and Hydra.)
  • Define a flake URL format. (I.e. the “Nix resources”.)
  • Define a registry.
  • Replace any existing configuration/composition mechanisms in the Nix ecosystem, such as Nixpkgs overlays and NixOS modules.
  • Make any language changes.
  • Add a type system for defining the type of outputs. (IIRC this was part of Shea’s original flakes proposal.)
  • Add a module system.

To quote the RFC: Flakes allow hermetic, reproducible evaluation of multi-repository Nix projects; impose a discoverable, standard structure on Nix projects; and replace previous mechanisms such as Nix channels and the Nix search path.

Step 3 seems to ignore the work done in the last couple of years. Nix 2.4 is the first flakes-enabled version of Nix and it has flake-enabled versions of nix search, nix-env -i and nix-channel. (nixos-rebuild is in another project.)

In case you didn’t see it, there is a proposed roadmap that suggests stabilizing flakes and the new CLI in the next few 2.x releases, and doing a 3.0 release when we’re prepared to call them stable.

4 Likes
6

Thanks for replying!

I guess that’s right, and what I remembered was criticism or pushback that wondered about the motivating use cases for flakes, and how there seemed to be many (like the many features of the new Nix CLI).

The idea with Step 3 is just that it might be feasible to make some of the new Nix subcommands available without the experimental-features flag before others, with the suggested minimal set of commands being those required to install software.

It was written with reference to commands that didn’t require the experimental-features option in Nix 2.3 because those are the commands that people who haven’t been using flakes yet would already know well and depend on. I thought it went without saying that this would be a matter of seeking feedback on the existing flake-enabled Nix subcommands, touching them up as needed, and then formalizing their status as official. It also seemed to me like referring to old commands in proposals is a good way to help ensure that changes in behavior are documented and understood, and kinda gives a starting point for questions about whether/when the replacements support behavioral equivalence, if that makes sense.

I’m using those replacements today (nix profile install for nix-env -i, nix search for its namesake Nix 2.3 command, etc.) on the system that I’m typing from. I had them in mind when I wrote Step 3. :slight_smile:

The thought was that because each Nix subcommand has (after your two years of work!) a concrete implementation and immediate use case for end users, submitting the portion of the flake.nix schema required to enable a given subcommand for community feedback and approval individually might invite user and community input in a way that is more focused and constructive than some of the broad discussions of flakes seemed to have been in the past.

Thanks! I’d glanced at it but I probably should have reviewed it before creating this thread.

Because one of my concerns with flakes when I originally wrote what I adapted here was how long it had taken any flakes features to make it into a stable release, even remaining behind the gate of the experimental-features option, what I also hoped for out of the incremental approach outlined in Step 3 was a way of making it possible for downstream Nix tooling to start counting on parts of the flake.nix schema as quickly as possible without being hasty about finalizing the whole thing.

Naively, it seems to me like an incremental approach to declaring portions of the flake.nix schema stable and whitelisting Nix subcommands might fit well with the time-based release schedule on the current roadmap. It sounds like the current plan is just to declare the flakes schema and the Nix CLI stable all at once, together, after all of the pieces are ready. Is that because you anticipate changes in one subcommand requiring changes in other subcommands, or for other/additional reasons?