Improving flakes

[pxc]

@payas

how much of a churn will [the addition of a feature which makes it easier for Nix codebases to refer to one another inflict] on the ecosystem?

@payas

I always imagined flakes as eventual future where entire Nixpkgs is converted to flakes for individual packages.

I think exhaustive monorepos of build recipes are a great basis for feeding package managers. There are tons working examples (besides Nixpkgs and Guix): Gentoo Portage, FreeBSD Ports, NetBSD/SmartOS/OpenIndiana pkgsrc, MacPorts, Homebrew, and Void Linux at least. I understand that Eelco expressed some doubts about the scalability of that approach on the original Flakes RFC, and I disagree with that. I also know that our disagreement doesn’t really matter.

There is no possible implementation of Nix flakes that makes Nix incapable of operating on a Nixpkgs-like monorepo of packages, or indeed Nixpkgs as we have it right now. The thing that powers Nixpkgs in a flakes-based future is the same thing that powers it now: the work of its maintainers. As long as there are people who find it useful and gratifying to maintain a unified set of packages with as much sharing of dependencies as possible, there will be one.

The concern that making it easier to use smaller, independent repositories of Nix packages will fracture Nixpkgs raises the question: what is keeping current Nixpkgs maintainers and contributors invested in the repository? Is it really the fact that splitting their packages out would be cumbersome? I doubt it for several reasons, but even if it were the case… isn’t the problem of a package set maintained by half-willing hostages of technical inconvenience way, way more dire than any imaginary outcome in a flakes-based future? The solution, in any case, seems to be in making sure Nixpkgs is as convenient and maintainable as possible, not in shunning features that enable other ways of composing collections of packages.

@payas

I imagine a lot of people have overlapping sets and subsets of expectations from Flakes in general. How about we start listing them, and then go through them for prioritising?

Something like this is exactly what I had in mind.

@payas

I would like the sub-flake for sub-module story sorted, but I’m not fully clear in this.

As you can imagine, I’d rather punt on this. But anyway: do you mean for Nixpkgs, or just something like ‘here’s a sane way you could use subflakes in a project if you wanted to, including a package set as large as Nixpkgs’?

@payas

90%+ flakes have Nixpkgs as input, and I’m pretty sure 90%+ searches through “nix search” are for nixpkgs, it is still not one of the defaults

I think there are good ways to give Nixpkgs a less uniquely privileged place in the Nix ecosystem without making it any less convenient to use, or threatening its survival. In this case, would the following simple solution work for you? I think it’s something like what I want (I think with option (b) for the defaults)

• When nix search or nix profile install is run without arguments that specify a flake, Nix searches a bunch of flakes according to configuration. By default, it could search (a) all flakes in the registry, (b) only flakes that are pinned in the registry (since that’ll be fast), or (c) no flakes.

@DianaOlympos

Probably and that is not clearer than the first time i read it. We can already do it and yes it is painful. But if that is all we are achieving with this for that much cost and complexity…

I don’t know what ‘cost’ means here, but if it means time going into writing the code that powers the current experimental version of flakes, it doesn’t matter— that cost has already been paid (and not by you or me) regardless of whether the flakes stuff gets revised, replaced, dropped entirely, or released as-is.

At any rate, the exercise I’ve started in this thread is to generate a proposition that potentially involves less complexity for end users than fully grokking flakes as they exist now by finalizing a portion of the flakes feature that does fewer things. So ‘that much complexity’ need not be part of the proposition.

@DianaOlympos

@pxc

Everyone who is using flakes right now is using the very same Nixpkgs we all already know and love. So in a way, the question of what flakes are for if not transforming the structure of Nixpkgs is very easy to answer: look at how flake users are already using flakes, and you’ll see uses that don’t involve any such change.

I did try and i have to disagree, i see a lot of work to be done to be able to get a setup that work, as was pointed multiple time.

This addresses something other than the point I was making, which is about usage of flakes requiring changes to Nixpkgs, and in particular dissolving the monorepo. Of course using flakes in a Nix project or a NixOS configuration is going to require changes within that project.

@DianaOlympos

@pxc

My hope with those prompts was to use the fact that flakes has an implementation that is already used to discard vague, open-ended questions like ‘what are we trying to do [in the end]’.

But that is not true.

This is confusing. I don’t make a truth claim in the quotation you cite, and at any rate whether or not flakes already has users is a matter of fact and anyone with a web browser that can reach GitHub can see that it’s true.

@DianaOlympos

[W]e are [still] talking of what [flakes] solve. We got given an implementation and told that was the way to go and now everyone is trying to find why the hell [did] we do this.

So,

why did we do this

and

what problems are solved by flakes?

are not the same question. They’re also not the same kind of question— why we have widely used but unofficial (and somewhat controversial) implementation of flakes is a social and historical question, and what problems are solved by flakes is a technical question. Recently, @grahamc wrote a blog post addressing the historical question and a discussion was opened up on Discourse here.

The technical question is answerable in terms of the actual present usage of flakes regardless of whether you, or I, or anyone else likes the current implementation. (It’s also independent of the question of whether flakes uniquely solves those problems. But the fact that flakes have been chosen by many users despite being a bit cumbersome does indicate that those users perceive it to solve at least some of their problems better than existing alternatives do.)

My purpose in emphasizing existing usage right now is not to marginalize or turn away from the concerns of people who just don’t use flakes and haven’t bought in at all. Once we have a clear picture of the common portion in the needs of existing flake users, then we can try to integrate the goal of meeting those minimal, common feature completeness needs of existing flake users with the ergonomic needs of people who are less invested in built-in input management/locking functionality. This could be a smaller and less controversial task than trying to conform every ambition that the current flakes implementation and its users have so far had for the feature to those same ergonomic constraints. To get that scope reduction for the proposition of what to standardize and release, we can look at the existing usage. Once we have that, we can think clearly about what unobtrusive implementations of it would look and feel like. Then we could take a step forward without either totally ripping the more complete, experimental implementation of flakes out from under existing users, or presenting non-flakes users with new interfaces whose complexity is excessive or motivated by use cases which haven’t been practiced yet.

Toward clarifying the needs of people who worry that a finalized implementation of flakes (even, potentially, one that is smaller/simpler than the existing implementation) will get in their way, a survey for those users:

• what operating systems do you currently use Nix on?
• do you currently pin any Nix package sources?
  • if so, is that done inside a project for an application, a development environment, a NixOS configuration, or what?
• do you currently do anything to configure NIX_PATH or directories in it (e.g., nix.nixPath on NixOS, setting up local checkouts, modifying /etc/bashrc on Debian or macOS, etc.)?
• if you use nix-channel, how do you use it?
• do you store some or all of your Nix configurations in source control? which and how?