How can we get @edolstra to weigh in on the subject of a roadmap for flakes?
2 Likes
What are these? (Edit, to clarify, that’s not direct at anyone in particular, nor am I asking folks to speak on someone else’s behalf. It’s just that I also have gotten this impression, but my understanding of it is somewhat nebulous and I’m eager to understand better in the hopes of there being a best path forward.)
Well I left this part out explicitly, so as not to call anyone out. I’ve heard various comments ranging in depth from “I hate flakes” to more involved arguments about how they will cause a split in nixpkgs, which is probably the concern I’ve heard echoed the most. I was hoping to encourage those with strong opinions to weigh in themselves, rather than attempting to speak on their behalf.
There are also a lot of warts, like the few I mentioned up top, and I’ve got a few more I’ll probably add later on. It looks like the lack of a clear roadmap is a big one for many people, myself included. I was focused more on technical shortcomings in my OP so I’m glad you brought this up as it’s a really important point.
4 Likes
You mean a package source (repo) which hasn’t a flake.nix file i suppose?
I’ve had a discussion about Flakes with a few people in private.
To maximize the chances of Flake feedback be taken into the account, I’d really suggest splitting the discussion to one specific aspect/point. If everything is brought up in one issue, it speaks about the Flakes in general instead of limiting the discussion to a specific issue.
Writing the system input proposal and talking to people it feels to me that the underlying issue about Flakes is that flakes are the first attempt of “convention over configuration”, but only in the aspect of the CLI, but not also the specification of the Nix file.
That gives Flakes a huge design space, that each of us will find a different way, as when things get opinionated, there are lots of opinions.
My take would be: let’s get Nix 2.4 release out and make another iteration of the design in the upcoming months. This will provide feedback from even more users and allowing us to make another round of improvements. It’s a lot better to learn from current design now, rather than hold on to it forever (sunk cost fallacy).
The fact that we are reluctant to advise the use of flakes is by design, it’s marked as experimental and still unreleased. What would help is to define what experimental flag means so that expectations can be held accordingly.
16 Likes
Preface: I like some of the changes that the current Nix unstable development provides but I fear the whole changes that “we” are trying to get into 2.4 are too big already. We’ve had several parts of the code changed in drastic ways. The bug budget for a stable release has IMHO been exhausted a while ago.
I had these comments for a while already but only ever brought them up in many private conversations but never put it out there in “public”.
Here it goes:
I for one think we should circle back a bit on what flakes are supposed to fix and identify issues with the current solutions (random default.nix files…) and what approaches we can take to make the situation better. As Domen pointed out this also means defining what an experimental feature is.
A first step to agreeing on a proper Flakes implementation is to decide what it should solve. Solving “all of the problems” at once will probably not lead to an ideal situation. What are those problems anyway? Once there is an agreement on what it is we want to solve we can start experimenting with solutions and incrementally make the world a better place.
The whole RFC did go sideways and didn’t have a clear direction. The implementation changed as the RFC was going forward. Changing code while discussion is ongoing causes lots of churn on those involved. This is equally true for those trying to follow the RFC as well as those brave souls implementing it.
Flakes, as implemented right now, is trying to do many things but it doesn’t feel well rounded for usage from beginner to advanced scenarios. What was the goal here? Is a huge monorepo (with sub(-sub*)-flakes) a valid use case? Is anyone not using any of the support libraries that are required to write half-maintainable flake.nix files for more than just x86_64-linux? What does it actually include?
Since defining and implementing a standardized file format for flakes isn’t enough there is also the whole side-quest of overhauling the CLI. I am aware that flakes - in its current form - want to be as hermetic as possible and thus sets a few evaluator options to make it so. For an experimental initial implementation we might as well implement flake build as a wrapper around nix-build flake.nix --option ... -A .... until the file format and the general direction of flakes are decided and agreed upon.
In order to properly iterate on the Flakes implementation quickly (with everyone in the community having a chance) it would have been much better to start out by implementing Flakes logic in pure Nix instead of the Nix C++ codebase.
Nothing would have been wrong with import <nix/experiements/flakesN.nix> { inputs = { ... }; outputs = { ... }; until we are happy with UI & UX of flakes. For reasons of efficiency and future UX improvements it could then be moved into the Nix C++ code - if required at all. Before the actual implementation is “stable” we could have bundled experimental flake files with Nix in point releases. Importing those would also “unsupported” and should throw a warning on every single use.
Features such as lockfiles can and could be implemented as well. We already have command line tooling to generate “lockfiles” (nix-prefetch-git .. | jq .. > lock.json). Obviously in the end the tooling should be integrated with the Nix installation.
When iterating - locally - on the flakes format all you would have to do is exchange that path that the flake definition is imported from. Currently you must be fluent in C++ and understand all the indirections the code, the fetchers, the new multi-inheritance CLI classes, … are taking. At this point it should be obvious that contributing non-trivial changes to Nix requires a lot of dedication whilst we can see in Nixpkgs that we have plenty of people that are willing to write Nix.
Perhaps the lockfiles story should have been tackled after the initial version anyway. Nix users are used to pin their dependencies explicitly. Pinning dependencies also relates to the whole story around Nix having it’s own (flake) registry. Who does it serve? Does it just safe us the github:nixos/ prefix when referring to nixpkgs? Is that really worth the feature? There was a whole sub-discussion on the RFC that was not taken seriously as it was officially not part of it but yet is an essential part of how flakes fetches inputs. What is gained by introducing yet another notation for declaring inputs? Couldn’t we fix whatever was wrong with the fetchers? What is gained by having the inputs be an attribute set instead of actual Nix code? Why do we have to take them into the inputs = { ... }: function instead of using regular Nix code that used to be good enough? I feel the overall community doesn’t know why that was done or even if it is a good way it is now. It certainly serves the new CLI.
As a general note I am impressed on how well the Rust community does this. There have been RFCs to define a general direction of a feature and then there is follow-up work to actually flesh out the implementation AFTER the direction is clear. What happened here is nothing like that. We just get an RFC while we discuss an already existing implementation just to amend the implementation.
24 Likes
I definitely agree that it can be overwhelming and somewhat unproductive to discuss all the issues with the current implementation all at once. But at some point we’ve got to distill these issues down into some form of RFC so, as @andir mentioned very eloquently, we can have a clear direction for the devs to move toward.
I myself have been very frustrated with my own person lack of C++ experience, and have tried and mostly failed to address some outstanding issues. The best I can hope for right now is to push the conversation forward and I’d be more than happy to help draft an RFC to bring everything together once we work out a clear direction for flakes.
It has been difficult to discern where best to try and address these issues since not all of the core Nix devs seem to frequent discourse, but I thought it might be a good place to start. The current situation of flakes being unstable for several years seems to have caused a lot of confusion, exhaustion and frustration for all parties, and I’d really love to see some concrete steps taken to finally move towards a merged release.
Even if the ultimate decision is simply to completely scrap flakes and move forward with a completely different solution, I’d like to see that codified as soon as possible.
After all, the longer we wait, the bigger that diff is going to grow…
9 Likes
So this is going to be from the pov of someone that at this point consider a hard fork like the only way forward. I have zero hope of this conversation getting anywhere sadly.
I think part of the problem is that the nix community has 2 different type of users. One one hand you get the people that seem mostly to come from the Haskell side of the community. What they want is to sandbox better, deal with impurity and limit the reproducibility problems.
On the other hand, you have people coming from a more ops or other dev community pov who mostly want a nice way to dive a dev, build and deploy environment. Something that is particularly hard to get out there outside of Dockerfiles, which has all kind of problems.
This second population has yet to understand any advantage of flakes. This is a population that is generally really happy about the 2nix, FOD or fetch stuff and that mostly see flakes as a way to make them all kind of work that would not be sustainable. How much of that is true is left as an exercise to the reader. It also a community that mostly do not want to use Git at a distribution format.
When reading the Flakes proposal and the development since then, the feeling of that second part is mostly that they are ignored. Is that true ? Maybe not but it is the feeling.
This is not helped by the fact that Flakes were presented as a “fait accompli”, with an implementation already in the core repository, with a fully fledged implementation. Especially knowing how hard it is to approach and contribute to the Nix codebase itself. This felt like a “this is how things are going to be, deal with it”.
I don’t know what steps can be taken from there. I would advise to do a build flag next time instead of a runtime flag to run experimental. Do not make people feel like this is a decided way and we are now just stabilising, as it is how it felt this time.
I do think that Nix will have to decide what they want to be on the long run. The community as it is right now is too diverse with too diverse needs. Unless major work and money is poured into building a codebase that can deal with them all, hard forks and explosion into more granular communities is probably a better way forward. The main reason that has not happened yet is mostly that the Nix codebase is considered mostly intractable.
5 Likes
I suck. I’m embarrassed to show my face in the community because I have been a ‘ghost’ maintainer (might still be listed in Nixpkgs, idk), and I used to make minor contributions every now and then, and I don’t anymore even though I still use NixOS as my main desktop OS, I use it at work, and I follow developments in the community closely. Every criticism I have related to Nix is directed, one way or another, at people who give more to the community, have more expertise, and are more responsible than I am, which makes me want to keep my thoughts on a lot of things to myself. (I’d also like to note, like I think others have in this thread or similar threads, that the situation with Nix flakes is so painful in part precisely because the existing implementation really does solve a bunch of problems and facilitate valuable new ways to use Nix.)
But I’ve been thinking about this a lot lately, I wanna get it out, and I think some of what I have to say might be useful.
When I first got into Nix, I enthusiastically Nixified my setup, creating a config repo that I used to take my tools with me on NixOS, foreign Linux, and macOS. I used git submodules for checkouts of Nixpkgs, which essentially worked to pin my setup, but it was/is cumbersome. For a hell of a long time now, I’ve told myself that it’s time to get back into Nix, starting by rewriting my Nix configs in order to simplify them, so that I can pin Nixpkgs and other sources of Nix expressions in a more lightweight way, and so that I can work on NixOS modules without keeping a fork of Nixpkgs around just to test and develop them. After that I’d like to fix some things I broke ages ago, if they’re still broken, and work on some new configuration stuff for fish and other shells. I’d like to be able to conveniently do the latter outside of the Nixpkgs tree because I’d like for others to be able to use it regardless of whether I ever try to get it merged into Nixpkgs.
Every time I’ve looked into it, I’ve been put off by the state of flakes and other available options for pinning the local system on NixOS. For my personal use, I hardly even care about the flakes interfaces (the one for Nix code or the CLI) being stable, but it’s still never really looked like a good option for me. The problems have instead been as follows.
-
The Nix ecosystem involves a ton of tooling that’s more or less third-party, or independent. Because the flakes feature is considered unstable/experimental and might disappear, support for flakes has lagged in many of these tools, and has arrived at various times. On different occasions when I looked into switching, different tools that were part of my setup or that I wanted to be able to use, like NixOps or Home Manager, didn’t support flakes. I looked into branches and forks where support had been added, but it was more steps and I didn’t want to deal with it. This kind of thing is still a problem for some tools.
-
Actual flake.nix files for NixOS configurations are complicated. Besides boilerplate syntax (tons of
flake = false;) there’s also often some logic involved in them many of them. This is the equivalent of a general purpose package manager’s sources list, and many configurations I might want to model mine off of include bits of code for things like:
• dealing with all the different target systems in the way the flakes interface forces you to
• managing NIX_PATH or some kind of backwards compatibility
• patching Nixpkgs
• emulating the Nixpkgs flake by setting up legacyPackages so that the convenient nix profile install syntax works
• setting up overlays from various pinned package sources
• configuring Hydra jobs or other CI
(I guess some of this stuff is what Nix flakes are for, and I can see why it makes sense to put that stuff in flake.nix. But truthfully, I mostly just wanna pin Nixpkgs and have it work with nicely with nixos-rebuild, and the best examples I have to look at are all over 100 lines long (often several hundred lines!), and it’s not unusual for them to pull in whole libraries like flake-utils or flake-utils-plus in order to be a little neater. I want to get started using flakes to pin the existing dependencies for my Nix setups (NixOS, macOS, foreign Linux) in like 20 minutes or less without dumping a bunch of helper functions in my flake.nix or building from a template whose conventions may be motivated than much more complicated setups than mine or an assumption of NixOS-only. There’s still no obvious starting point for that, and it reinforces my procrastination/hesitancy.)
- The Nix flake registry doesn’t have good backwards compatibility with Nix channels and NIX_PATH. It feels incomplete and inconvenient to use until (a) the Nix registry presents its entries as pseudo-channels for tools or commands that still expect NIX_PATH, and (b) there’s some way to configure a systemwide/multi-user Nix registry on non-NixOS (or imperatively, via the command-line), like you can using
root’s channels.
There are other issues, but those are the reasons I’m not using flakes right now, even though it’s clear to me that they solve important problems and I’d basically like to be using them already.
I think some of the ‘extra’ stuff flakes do is interesting and might be necessary, but if nixos-rebuild and nix-env worked as well with Niv as they do with nix-channel, I’d just start using Niv for my NixOS configurations tomorrow.
So that’s what’s holding me back or turning me off when it comes to flakes, but the limbo state of flakes (and the other new features in nixUnstable) also puts me off recommending Nix to others. 
You only get so much interest and patience when you’re trying to get someone to try new things, and I don’t want to waste the one chance I might have with someone recommending a version of Nix which is only almost there. Here’s what I mean:
-
Nix is making its way into the repositories of package managers for other distros (it’s already in Debian, Gentoo, and FreeBSD Ports, while it’s on the way in NetBSD’s pkgsrc). This is awesome, because it means trying Nix can be easier, safer and more manageable. It could put it in a similar position to Flatpak or Snap for users who want to get versions of packages or whole packages that aren’t available in their base distribution, where its presence in the base distribution makes it feel like a natural addition rather than grafted on, the way a glorified
curl|bashinstaller can seem. But, for example, the Debian package doesn’t set up NIX_PATH the same way as NixOS or our tarball installer. But why deal with holding someone’s hand in figuring out what their NIX_PATH is and where it comes from, especially when I know that ‘any day now’ Nix won’t depend on NIX_PATH anyway? It also doesn’t add any channels by default at all, so even setting aside the impurity,nix-envwon’t know how to install anything out of the box. Best, it seems, to just wait a bit before recommending Ubuntu usersapt install nix-bin nix-systemd-setup. -
Same thing for recommending NixOS for local desktops or simple server setups when the selling point is reproducibility and the best tool we have for pinning NixOS setups is unreleased/unsupported and kinda incomplete. I want to wait at least until we have something like devos a little more settled so that I can tell someone: ‘here’s how you set up NixOS and pin it to a commit, here’s where you put the config for each host, here’s where you put any custom modules you want, here’s where you put custom packages— and you can start thinking about how that stuff works and whether you want to rearrange it as you become comfortable’, knowing that the starting point for them is simple and unlikely to change from what I use or know in the near future.
-
Similarly, all of the CLI improvements in nixUnstable, the fact that those will still change, and the massive performance difference yielded by flakes thanks to the evaluation caching it supports all make me want to put off encouraging people from really trying and learning the Nix CLI until after that stuff is released. It also doesn’t seem great to just introduce them to the existing/old CLI when I know that something much better is fairly well developed, mostly documented, and fairly widely used. Some of this is stuff that the community can work out independent of the first stable release of Nix that enables flakes, as seems to be going on inside devos and flake-utils-plus. But to some extent, projects like this that aim to simplify or settle conventions for the use of flakes inherit this problem of volatility (likelihood to undergo breaking changes in the near or moderate term) from the flakes feature itself.
Overall, the sense that later (but soon) will be a MUCH better time to recommend Nix seems like a real problem to me. And it’s a problem that might stand in the way of the development of future Nix goodies. You can even imagine something like a deflationary spiral except the investment being disincentivized is developer time in the Nix ecosystem. Perhaps this worry is overstated. Lots of things are definitely still moving forward in the Nix ecosystem and it’s exciting to see. But I can’t help but wonder if we’re missing out because of this effect.
I think maybe I’d rather have a flakes interface that’s settled and sucks, or see a promise of stability actually get broken, than see Nix spend another year or two in almost-there limbo for the basic functionality of pinning package sources and incorporating third-party repositories in a way that every tool in the Nix universe is expected to support. If we have to start out with an initial version of flakes that’s kinda clunky, I know I’ll figure out a way to deal with it and I bet the rest of the community will too. But the expectation that this implementation could disappear or radically change some time soon makes me (and, I assume, others) hesitate to bother really figuring out how to do that now. :-\
edited to make it a little clearer, more readable, and more polite (no important differences)
16 Likes
Generally I’ve liked having flakes as an opinionated structure. I’ve eschewed the wrapper utilities and have used them to pin, to create Hydra jobs, NixOS configurations, etc.
I’m not sure how to address all of the problems people have discussed, I only know how to solve problems: one at a time. So if people want to work with flakes, get something working, or even to help me understand what difficulties people are having, please put an hour or two in my office hours.
4 Likes
As a mostly passive observer and Nix/OS user for more than 2 years who has just started contributing to nixpkgs, I am barely at a stage to have a fully informed opinion, but thanks to @nrdxp we are having a much needed discussion on this.
Personally, I feel Flakes are starting to hold Nix back.
- We have bunch of nice improvements in Nix unstable, including new CLI and CAS. Some of these are entangled with Flakes, and are holding back wider adoption because flakes are so underbaked.
- I remember being excited about flakes way back after watching NixCon session by Eelco. But then the RFC went nowhere. This has many issues. I wanted to try out flakes immediately, but the documentation was so poor that for a Nix newbie it was too much. I can already see the rift happening, because nix already has sad documentation story and even then ongoing efforts are split between flakes and legacy way.
- Latest NixCon brought possible future of TOML flakes, which means whatever work I have put in, may get deprecated.
- I agree on the point that RFC did not discuss and pan out implementation details. we got an implementation and then there was discussion on how it could be take forward.
- We still lack any kind of transparency on what the future looks like, and from the history of this feature, even if we get one, I’m afraid it will be like RFC where we’ll be told what side we are already on.
The list may look like its all over the place and its partially my fault for being bad at communication, but here is what I would personally like to see.
- Flakes solve real problems, lets first decide what they are and in what order they should be solved. In short, scope definition.
- A fully open discussion on the above.
- A timeline on the above. It can stretch for 5 years and will be understandable because such changes take time.
- Consistent releases that solve individual parts from above. Let the small feature permeate slowly, so the community gets time to adjust and test and hash out finishing touches going forward.
I fell in love with Nix/OS the first time I tried it. I hope to continue using it, and I hope to be able to use it professionally some day. But as it stands, its already niche, and further ongoing lack of clarity is really going to hurt long term. Let this not be Python3 moment for us.
14 Likes
I’m closing this thread to encourage people to split their improvement points into single issues like @domenkozar asked. If you have concerns about this, please direct message me.
3 Likes
I received a bunch of feedback that I shouldn’t have closed this, so I’m reopening it.
8 Likes
Some notes from office hours:
- builtins.getFlake is not well-known. With a new enough Nix it allows non-flake uses to find and refer to flake-enabled packages/overlays/etc.
- recommend not using flake-utils or other “flake wrappers” until after one is comfortable with making a few explicit flakes
- it helps to at first only use flakes with
devShell = import ./shell.nix {inherit pkgs;};anddefaultPackage = import ./default.nix {inherit pkgs;}as a way to ease adoption and support both legacy and new (once comfortable one can adopt flake-compat) - avoid
nix profile(vicenix-env) until one is comfortable - people are used to using
nix-shellfor both build-time debugging, quick test of programs, and development shell. It helps to clearly split these consideration; flakes help here, though it can be non-obvious. (nix developandnix shell) - apps != packages
8 Likes
I didn’t see a thread specifically for this, so I’ll just add my feedback here. [Mods: feel free to move this post to another thread if it fits better]
Background: I started using Nix about 9 months ago, and had to decide to start out using flakes or using classic Nix. I looked into flakes and decided to learn classic Nix first. One reason was to experience the pain points for myself, before seeing the solution. The other was because it seemed like flakes required a lot more overhead and boilerplate than just writing nix expressions (and keeping careful control of nixpkgs by pinning or another method). One main source of “overhead” was system: why should I have to worry about this if I’m not cross-compiling? I don’t in classic Nix.
Feedback: system makes much more sense as an input. I strongly support this proposal.
9 Likes
Thanks, Tom! I hope it was still useful for me to get my general thoughts out, but you’ve inspired me to take the plunge so that I can revise those thoughts and round them out with more Immediately actionable questions/problems.
I finally started rewriting my Nix configs last night, starting with a simple, flake-based NixOS setup. I’m still copying over bits of my old config and working out how I want to organize it for non-NixOS hosts, but I’ll start a new thread (and probably link it here) soliciting feedback on my configuration once it’s more or less complete. 
So far I’ve chosen to use flake-utils-plus to keep that boilerplate outside of my flake.nix file, and I’ve been happy with that choice. I still might not have tried flakes without it. After I have something I like, I’ll test my intuitions by producing an equivalent flake.nix without it, but I really like the concision of my flake.nix so far. This strikes me as a very good proposal.
3 Likes
Thanks! There’s Provide builtins.currentSystem as a input to flakes or make system a parameter · Issue #3843 · NixOS/nix · GitHub for further discussion around the proposal.
3 Likes
people are used to using
nix-shellfor both build-time debugging, quick test of programs, and development shell. It helps to clearly split these consideration; flakes help here, though it can be non-obvious. (nix developandnix shel
I’ve tried moving from nix develop to nix shell but the fact that nix shell only manipulates PATH instead of arbitrary environment variables makes it unfit for my purpose.
For example, I want to set XDG_DATA_DIRS to make bash auto completion to work automatically for things in my shell; and similarly I want to have man pages work as soon as I enter the shell (MANPATH).
For it to replace my mkShell usage it will need to implement that feature at least…
5 Likes
@Ericson2314 writes on that issue:
[M]aybe we should implement more of Flakes in Nix. Factoring out
lib/and such things to make a standard library of sorts so we still have some agility. But at that point, what are flakes? Just something to manage inputs / lockfile without much extra structure?
This seems to me to be the single most urgent need that flakes meet.
-
Is there any opposition among people (who might be skeptical about flakes in general) to getting such a minimal implementation of flakes merged and stabilized as quickly as is feasible? (@payas? @DianaOlympos ?)
-
How might releasing a minimal version of flakes in that way ‘box in’ Nix developers so that the other goals of flakes are harder to achieve in the future?
-
What are those ‘other goals of flakes’? I can think of some, but maybe y’all can help me enumerate more. It seems that among them are:
- defining a more fixed, conventional structure than
default.nixfor Nix projects of all sorts - letting Nix the package manager better understand applications as opposed to raw collections of files
- integrating some kind of registry for repositories of software packaged with Nix (the current ‘global’ registry)
- and what else?
- defining a more fixed, conventional structure than
3 Likes