I didn’t see a thread specifically for this, so I’ll just add my feedback here. [Mods: feel free to move this post to another thread if it fits better]
Background: I started using Nix about 9 months ago, and had to decide to start out using flakes or using classic Nix. I looked into flakes and decided to learn classic Nix first. One reason was to experience the pain points for myself, before seeing the solution. The other was because it seemed like flakes required a lot more overhead and boilerplate than just writing nix expressions (and keeping careful control of nixpkgs by pinning or another method). One main source of “overhead” was system: why should I have to worry about this if I’m not cross-compiling? I don’t in classic Nix.
Feedback: system makes much more sense as an input. I strongly support this proposal.
Thanks, Tom! I hope it was still useful for me to get my general thoughts out, but you’ve inspired me to take the plunge so that I can revise those thoughts and round them out with more Immediately actionable questions/problems.
I finally started rewriting my Nix configs last night, starting with a simple, flake-based NixOS setup. I’m still copying over bits of my old config and working out how I want to organize it for non-NixOS hosts, but I’ll start a new thread (and probably link it here) soliciting feedback on my configuration once it’s more or less complete.
So far I’ve chosen to use flake-utils-plus to keep that boilerplate outside of my flake.nix file, and I’ve been happy with that choice. I still might not have tried flakes without it. After I have something I like, I’ll test my intuitions by producing an equivalent flake.nix without it, but I really like the concision of my flake.nix so far. This strikes me as a very good proposal.
people are used to using nix-shell for both build-time debugging, quick test of programs, and development shell. It helps to clearly split these consideration; flakes help here, though it can be non-obvious. ( nix develop and nix shel
I’ve tried moving from nix develop to nix shell but the fact that nix shell only manipulates PATH instead of arbitrary environment variables makes it unfit for my purpose.
For example, I want to set XDG_DATA_DIRS to make bash auto completion to work automatically for things in my shell; and similarly I want to have man pages work as soon as I enter the shell (MANPATH).
For it to replace my mkShell usage it will need to implement that feature at least…
[M]aybe we should implement more of Flakes in Nix. Factoring out lib/ and such things to make a standard library of sorts so we still have some agility. But at that point, what are flakes? Just something to manage inputs / lockfile without much extra structure?
This seems to me to be the single most urgent need that flakes meet.
Is there any opposition among people (who might be skeptical about flakes in general) to getting such a minimal implementation of flakes merged and stabilized as quickly as is feasible? (@payas? @DianaOlympos ?)
How might releasing a minimal version of flakes in that way ‘box in’ Nix developers so that the other goals of flakes are harder to achieve in the future?
What are those ‘other goals of flakes’? I can think of some, but maybe y’all can help me enumerate more. It seems that among them are:
defining a more fixed, conventional structure than default.nix for Nix projects of all sorts
letting Nix the package manager better understand applications as opposed to raw collections of files
integrating some kind of registry for repositories of software packaged with Nix (the current ‘global’ registry)
That seems to be a proper issue. Do you know if this is in-progress or has a GitHub issue? Or does nix shell intentionally not manipulate XDG vars as part of the design?
Until then you should be able to achieve what you need with devShell and shellHook.
I mean my main question if we do that is “What are we trying to do” and “what are flakes”
Right now flakes only make sense to me if they are taken as a whole as a “nixpkgs 2.0” basically. I may be wrong and i am happy to be. I think a solution here would be to open a small RFC describing the minimalistic goal we want to achieve as a first step. And that may require discussions here or elsewhere about what would go into it.
Is there any opposition among people (who might be skeptical about flakes in general) to getting such a minimal implementation of flakes merged and stabilized as quickly as is feasible?
I don’t see a problem with that, and agree on this speeding up the path to 2.4-3.0.
But your 2nd point is tangentially related with a separate doubt of mine, in that how much of a churn will it be on the ecosystem?
@DianaOlympos put it better in above comment. I always imagined flakes as eventual future where entire Nixpkgs is converted to flakes for individual packages. And I imagine a lot of people have overlapping sets and subsets of expectations from Flakes in general. How about we start listing them, and then go through them for prioritising?
I’ll start with mine.
Dependency pinning/lockfiles by default. I like this one being a built in thing.
Saner file structure. Right now, default.nix is a convention, while flake.nix becomes a standard.
I would like the sub-flake for sub-module story sorted, but I’m not fully clear in this.
I understand the implicit understanding and explicit design decisions towards that is eventual reduction of/on reliance of Nixpkgs. Which is why, even though 90%+ flakes have Nixpkgs as input, and I’m pretty sure 90%+ searches through “nix search” are for nixpkgs, it is still not one of the defaults. In short, I happen to like nixpkgs the monorepo, and would like to hash its future out under flakes.
Per the quotation in comment you are replying to, the answer is:
[j]ust something to manage inputs / lockfile without much extra structure
Is that more or less clear, or not really? Are the editorial substitutions I made I clarifying your question fair?
Everyone who is using flakes right now is using the very same Nixpkgs we all already know and love. So in a way, the question of what flakes are for if not transforming the structure of Nixpkgs is very easy to answer: look at how flake users are already using flakes, and you’ll see uses that don’t involve any such change.
For example, personal use case #1 for me is pinning the inputs (including nixpkgs) for NixOS systems that I manage locally with nixos-rebuild, without a bunch of kludge. (Alternatives include having to tell NixOS to set up nixpkgs sources in /etc, point NIX_PATH to them, and run nixos-rebuild twice every time I update the pinned version of Nixpkgs, and cloning a bunch of copies of Nixpkgs as submodules into my personal Nix configuration repo. Both kinda suck.)
Every deployment tool that supports flakes likewise presents a similar use case.
Another use case which doesn’t involve any changes to Nixpkgs at all is better facilitating the use of other people’s custom modules and packages. For example, the only reason the Nix User Repository really needs to exist is to use the overlay system to wrangle a bunch of repositories of Nix code into a usable namespace. Flakes make it really easy to pull in another flake user’s packages or modules, which is great because not everything that is usable is also suitable for inclusion in Nixpkgs, or has an author who wants to do the work to see it merged into Nixpkgs.
My hope with those prompts was to use the fact that flakes has an implementation that is already used to discard vague, open-ended questions like ‘what are we trying to do [in the end]’. We don’t have to answer that question to begin standardizing/releasing flakes if we look at the existing uses, find the most popular ones, and come to agree on some overlapping subset of them, like ‘ok, flakes are at least XYZ’. The candidate I’ve suggested for ‘XYZ’ here is managing and locking inputs, as suggested on the GH issue mentioned earlier.
Probably and that is not clearer than the first time i read it. We can already do it and yes it is painful. But if that is all we are achieving with this for that much cost and complexity, i would strongly advise to consider flakes like at best a spike at a possible solution and to go back to the drawing board.
I did try and i have to disagree, i see a lot of work to be done to be able to get a setup that work, as was pointed multiple time.
See that is interesting. Because i feel the opposite. It makes it harder. Channels have a ton of problems, but they are really easy to use. I have yet to be able to use flakes as well.
[quote=“pxc, post:30, topic:12831”]
My hope with those prompts was to use the fact that flakes has an implementation that is already used to discard vague, open-ended questions like ‘what are we trying to do [in the end]’. [/quote]
But that is not true. What we have is an implementation that a large part of the community feel does not solve the problem or make it worse. That is exactly why we are talking of what they solve. We got given an implementation and told that was the way to go and now everyone is trying to find why the hell do we do this.
In that case, i will say that i consider flake an interesting experiment on the path toward that, but something that is not solving the problem enough while introducing enough pain to not be an acceptable solution. We should go back to the drawing board and establish a plan on how we want to explore that set of problems more and offer steps forward. Your answer is “what people that want flakes use it is what we want from it”. Except the people that do not use it do not want that. We are back to a fork.
I am aware of nix-shell and mkShell… mkShell even has XDG_DATA_DIRS built in these days (if you provide your tools through nativeBuildInputs ). This is what I was using before…
However the discussion is about the fact that nix-shell was never meant for ad-hoc dev environments. It’s really meant to debug nix builds. It sets all kinds of extra env variables one is mostly not interested int. This is why Nix 2.4 moved nix-shell to nix develop and for the ad-hoc environments came up with a new command called nix shell.
However it isn’t featureful enough to replace how I used to use nix-shell/mkShell. That’s what the whole my point was about.
In order for nix shell to replace nix-shell/mkShell it should have the option to set more environment variables than just PATH
However; today I Came up with the following .envrc file to emulate the usage I want:
{
description = "A very basic flake";
outputs = { self, nixpkgs }: {
defaultPackage.x86_64-linux = nixpkgs.legacyPackages.x86_64-linux.buildEnv {
name = "dev-shell";
paths = with nixpkgs.legacyPackages.x86_64-linux; [
kubectl
(terraform_0_13.withPlugins (p: [ p.vultr p.ct ]))
kubernetes-helm
kustomize
];
};
};
}
I think exhaustive monorepos of build recipes are a great basis for feeding package managers. There are tons working examples (besides Nixpkgs and Guix): Gentoo Portage, FreeBSD Ports, NetBSD/SmartOS/OpenIndiana pkgsrc, MacPorts, Homebrew, and Void Linux at least. I understand that Eelco expressed some doubts about the scalability of that approach on the original Flakes RFC, and I disagree with that. I also know that our disagreement doesn’t really matter.
There is no possible implementation of Nix flakes that makes Nix incapable of operating on a Nixpkgs-like monorepo of packages, or indeed Nixpkgs as we have it right now. The thing that powers Nixpkgs in a flakes-based future is the same thing that powers it now: the work of its maintainers. As long as there are people who find it useful and gratifying to maintain a unified set of packages with as much sharing of dependencies as possible, there will be one.
The concern that making it easier to use smaller, independent repositories of Nix packages will fracture Nixpkgs raises the question: what is keeping current Nixpkgs maintainers and contributors invested in the repository? Is it really the fact that splitting their packages out would be cumbersome? I doubt it for several reasons, but even if it were the case… isn’t the problem of a package set maintained by half-willing hostages of technical inconvenience way, way more dire than any imaginary outcome in a flakes-based future? The solution, in any case, seems to be in making sure Nixpkgs is as convenient and maintainable as possible, not in shunning features that enable other ways of composing collections of packages.
Something like this is exactly what I had in mind.
As you can imagine, I’d rather punt on this. But anyway: do you mean for Nixpkgs, or just something like ‘here’s a sane way you could use subflakes in a project if you wanted to, including a package set as large as Nixpkgs’?
I think there are good ways to give Nixpkgs a less uniquely privileged place in the Nix ecosystem without making it any less convenient to use, or threatening its survival. In this case, would the following simple solution work for you? I think it’s something like what I want (I think with option (b) for the defaults)
When nix search or nix profile install is run without arguments that specify a flake, Nix searches a bunch of flakes according to configuration. By default, it could search (a) all flakes in the registry, (b) only flakes that are pinned in the registry (since that’ll be fast), or (c) no flakes.
I don’t know what ‘cost’ means here, but if it means time going into writing the code that powers the current experimental version of flakes, it doesn’t matter— that cost has already been paid (and not by you or me) regardless of whether the flakes stuff gets revised, replaced, dropped entirely, or released as-is.
At any rate, the exercise I’ve started in this thread is to generate a proposition that potentially involves less complexity for end users than fully grokking flakes as they exist now by finalizing a portion of the flakes feature that does fewer things. So ‘that much complexity’ need not be part of the proposition.
This addresses something other than the point I was making, which is about usage of flakes requiring changes to Nixpkgs, and in particular dissolving the monorepo. Of course using flakes in a Nix project or a NixOS configuration is going to require changes within that project.
This is confusing. I don’t make a truth claim in the quotation you cite, and at any rate whether or not flakes already has users is a matter of fact and anyone with a web browser that can reach GitHub can see that it’s true.
So,
why did we do this
and
what problems are solved by flakes?
are not the same question. They’re also not the same kind of question— why we have widely used but unofficial (and somewhat controversial) implementation of flakes is a social and historical question, and what problems are solved by flakes is a technical question. Recently, @grahamc wrote a blog post addressing the historical question and a discussion was opened up on Discourse here.
The technical question is answerable in terms of the actual present usage of flakes regardless of whether you, or I, or anyone else likes the current implementation. (It’s also independent of the question of whether flakes uniquely solves those problems. But the fact that flakes have been chosen by many users despite being a bit cumbersome does indicate that those users perceive it to solve at least some of their problems better than existing alternatives do.)
My purpose in emphasizing existing usage right now is not to marginalize or turn away from the concerns of people who just don’t use flakes and haven’t bought in at all. Once we have a clear picture of the common portion in the needs of existing flake users, then we can try to integrate the goal of meeting those minimal, common feature completeness needs of existing flake users with the ergonomic needs of people who are less invested in built-in input management/locking functionality. This could be a smaller and less controversial task than trying to conform every ambition that the current flakes implementation and its users have so far had for the feature to those same ergonomic constraints. To get that scope reduction for the proposition of what to standardize and release, we can look at the existing usage. Once we have that, we can think clearly about what unobtrusive implementations of it would look and feel like. Then we could take a step forward without either totally ripping the more complete, experimental implementation of flakes out from under existing users, or presenting non-flakes users with new interfaces whose complexity is excessive or motivated by use cases which haven’t been practiced yet.
Toward clarifying the needs of people who worry that a finalized implementation of flakes (even, potentially, one that is smaller/simpler than the existing implementation) will get in their way, a survey for those users:
what operating systems do you currently use Nix on?
do you currently pin any Nix package sources?
if so, is that done inside a project for an application, a development environment, a NixOS configuration, or what?
do you currently do anything to configure NIX_PATH or directories in it (e.g., nix.nixPath on NixOS, setting up local checkouts, modifying /etc/bashrc on Debian or macOS, etc.)?
if you use nix-channel, how do you use it?
do you store some or all of your Nix configurations in source control? which and how?
We have a good discussion here, but its kind of stalled now. @nrdxp@pxc@ryantm and others, any way we can move to possible RFC target as mentions in OP with what we already have, and then continue to hash out larger outline under it?
Yeah, so for me, I’m still trying to understand the best way to move forward with a proposal that will actual garner approval from all (or most) stateholders. Since the original flakes RFC was closed, I’m left wondering if the RFC process is even the appropriate place to suggest changes to flakes at all, and if not, then where do we have these dicussions in a more formalized fashion?
If it is deemed appropriate to go ahead and open an RRC on this, or perhaps we could start a separate RPC process specifically targeting experimetal features, then here’s what I’m thinking it could include:
It appears the most controversial feature of flakes, by far, is the registry. So perhaps, in an effort to get flakes merged faster, we just drop this feature entirely. Really, it doesn’t seem like we lose all that much without it, other than the shorthand for common flakes. Perhaps to mitigate this loss, we can simply add a configuration option to specify desired shorthands, with nixpkgs included as the default entry.
Also, it seems the major difficultly with merging flakes as they stand right now, is that their implementation is kinda tied up with the overhaul of the nix command in general, which is a quickly moving taget. Depending on how difficult this would be to architect, perhaps we could simply leave out the major changes to the UI, and only include the nix flake subcommand, leaving all other additions to continue as experimental features until the devs are content with the redesign.
The biggest hurdle to this might be introducing the notion of the flake uri to the existing stable nix * commands. I’m not familiar enough with the codebase to know if this would be a feasible task.
Other than that, perhaps we should leave out any major changes to flakes for a separate RFC, and simply begin to identify a list of bugs/todos that exist in the current implementation, and start closing them up. Once done, we can just merge as is, and iterate on the design with each point release.
Of course, since I lack experience with the core nix codebase, my suggestions may end up being less than practical, but I guess that’s what a request for comments is for anyway right
No! Even though I do not like or understand why or what is in the registry, please do not remove that feature. Beeing able to write nixpkgs#hello rather than having to write github:nixos/nixpkgs#hello is a time safer!
Though I’d appreciate if usage of the registry would be restricted from a flake.nix, and neither implicit inputs nor flake: URLs would be allowed.
As a matter of structuring the process appropriately, I think packaging ideas into flysheets (similar to the system one proposed by @domenkozar & already referenced above) and let them stick around for a while is a valid option to prepare the field for consensus. They are also easy to x-link and graduate into a full-blown RFC.
Since not everyone uses https://notion.so, here are two alternatives:
We just need to say that flake-url without a prefix: points to a configurable location, with default github:nixos/. We get the short invocation syntax without having to maintain a registry.
What we lose is the ability to have more “top-level names” (e.g. nix run blender-bin), but do we invoke that so often to require a registry?
Sounds like a local only registry. I’m totally fine with a registry that does no remote look ups, but works locally only, as long as it gives the same freedom as the current one.
And I’d be even happier if it wasn’t available from within a flake.nix but only from the CLI.
Looking at the linked thing, I do not see anything beyond Discourse Wiki posts, except complete lack of content accessibility without Javascript and broken PgDn. And I would expect the timescale of discussions to not need parallel real-time editing of hackmd…
But anyway: do you mean for Nixpkgs, or just something like ‘here’s a sane way you could use subflakes in a project if you wanted to, including a package set as large as Nixpkgs’?