Improving flakes

I definitely agree that it can be overwhelming and somewhat unproductive to discuss all the issues with the current implementation all at once. But at some point we’ve got to distill these issues down into some form of RFC so, as @andir mentioned very eloquently, we can have a clear direction for the devs to move toward.

I myself have been very frustrated with my own person lack of C++ experience, and have tried and mostly failed to address some outstanding issues. The best I can hope for right now is to push the conversation forward and I’d be more than happy to help draft an RFC to bring everything together once we work out a clear direction for flakes.

It has been difficult to discern where best to try and address these issues since not all of the core Nix devs seem to frequent discourse, but I thought it might be a good place to start. The current situation of flakes being unstable for several years seems to have caused a lot of confusion, exhaustion and frustration for all parties, and I’d really love to see some concrete steps taken to finally move towards a merged release.

Even if the ultimate decision is simply to completely scrap flakes and move forward with a completely different solution, I’d like to see that codified as soon as possible.

After all, the longer we wait, the bigger that diff is going to grow…

9 Likes

So this is going to be from the pov of someone that at this point consider a hard fork like the only way forward. I have zero hope of this conversation getting anywhere sadly.

I think part of the problem is that the nix community has 2 different type of users. One one hand you get the people that seem mostly to come from the Haskell side of the community. What they want is to sandbox better, deal with impurity and limit the reproducibility problems.

On the other hand, you have people coming from a more ops or other dev community pov who mostly want a nice way to dive a dev, build and deploy environment. Something that is particularly hard to get out there outside of Dockerfiles, which has all kind of problems.

This second population has yet to understand any advantage of flakes. This is a population that is generally really happy about the 2nix, FOD or fetch stuff and that mostly see flakes as a way to make them all kind of work that would not be sustainable. How much of that is true is left as an exercise to the reader. It also a community that mostly do not want to use Git at a distribution format.

When reading the Flakes proposal and the development since then, the feeling of that second part is mostly that they are ignored. Is that true ? Maybe not but it is the feeling.

This is not helped by the fact that Flakes were presented as a “fait accompli”, with an implementation already in the core repository, with a fully fledged implementation. Especially knowing how hard it is to approach and contribute to the Nix codebase itself. This felt like a “this is how things are going to be, deal with it”.

I don’t know what steps can be taken from there. I would advise to do a build flag next time instead of a runtime flag to run experimental. Do not make people feel like this is a decided way and we are now just stabilising, as it is how it felt this time.

I do think that Nix will have to decide what they want to be on the long run. The community as it is right now is too diverse with too diverse needs. Unless major work and money is poured into building a codebase that can deal with them all, hard forks and explosion into more granular communities is probably a better way forward. The main reason that has not happened yet is mostly that the Nix codebase is considered mostly intractable.

5 Likes

I suck. I’m embarrassed to show my face in the community because I have been a ‘ghost’ maintainer (might still be listed in Nixpkgs, idk), and I used to make minor contributions every now and then, and I don’t anymore even though I still use NixOS as my main desktop OS, I use it at work, and I follow developments in the community closely. Every criticism I have related to Nix is directed, one way or another, at people who give more to the community, have more expertise, and are more responsible than I am, which makes me want to keep my thoughts on a lot of things to myself. (I’d also like to note, like I think others have in this thread or similar threads, that the situation with Nix flakes is so painful in part precisely because the existing implementation really does solve a bunch of problems and facilitate valuable new ways to use Nix.)

But I’ve been thinking about this a lot lately, I wanna get it out, and I think some of what I have to say might be useful.

When I first got into Nix, I enthusiastically Nixified my setup, creating a config repo that I used to take my tools with me on NixOS, foreign Linux, and macOS. I used git submodules for checkouts of Nixpkgs, which essentially worked to pin my setup, but it was/is cumbersome. For a hell of a long time now, I’ve told myself that it’s time to get back into Nix, starting by rewriting my Nix configs in order to simplify them, so that I can pin Nixpkgs and other sources of Nix expressions in a more lightweight way, and so that I can work on NixOS modules without keeping a fork of Nixpkgs around just to test and develop them. After that I’d like to fix some things I broke ages ago, if they’re still broken, and work on some new configuration stuff for fish and other shells. I’d like to be able to conveniently do the latter outside of the Nixpkgs tree because I’d like for others to be able to use it regardless of whether I ever try to get it merged into Nixpkgs.

Every time I’ve looked into it, I’ve been put off by the state of flakes and other available options for pinning the local system on NixOS. For my personal use, I hardly even care about the flakes interfaces (the one for Nix code or the CLI) being stable, but it’s still never really looked like a good option for me. The problems have instead been as follows.

  1. The Nix ecosystem involves a ton of tooling that’s more or less third-party, or independent. Because the flakes feature is considered unstable/experimental and might disappear, support for flakes has lagged in many of these tools, and has arrived at various times. On different occasions when I looked into switching, different tools that were part of my setup or that I wanted to be able to use, like NixOps or Home Manager, didn’t support flakes. I looked into branches and forks where support had been added, but it was more steps and I didn’t want to deal with it. This kind of thing is still a problem for some tools.

  2. Actual flake.nix files for NixOS configurations are complicated. Besides boilerplate syntax (tons of flake = false;) there’s also often some logic involved in them many of them. This is the equivalent of a general purpose package manager’s sources list, and many configurations I might want to model mine off of include bits of code for things like:
    • dealing with all the different target systems in the way the flakes interface forces you to
    • managing NIX_PATH or some kind of backwards compatibility
    • patching Nixpkgs
    • emulating the Nixpkgs flake by setting up legacyPackages so that the convenient nix profile install syntax works
    • setting up overlays from various pinned package sources
    • configuring Hydra jobs or other CI

(I guess some of this stuff is what Nix flakes are for, and I can see why it makes sense to put that stuff in flake.nix. But truthfully, I mostly just wanna pin Nixpkgs and have it work with nicely with nixos-rebuild, and the best examples I have to look at are all over 100 lines long (often several hundred lines!), and it’s not unusual for them to pull in whole libraries like flake-utils or flake-utils-plus in order to be a little neater. I want to get started using flakes to pin the existing dependencies for my Nix setups (NixOS, macOS, foreign Linux) in like 20 minutes or less without dumping a bunch of helper functions in my flake.nix or building from a template whose conventions may be motivated than much more complicated setups than mine or an assumption of NixOS-only. There’s still no obvious starting point for that, and it reinforces my procrastination/hesitancy.)

  1. The Nix flake registry doesn’t have good backwards compatibility with Nix channels and NIX_PATH. It feels incomplete and inconvenient to use until (a) the Nix registry presents its entries as pseudo-channels for tools or commands that still expect NIX_PATH, and (b) there’s some way to configure a systemwide/multi-user Nix registry on non-NixOS (or imperatively, via the command-line), like you can using root’s channels.

There are other issues, but those are the reasons I’m not using flakes right now, even though it’s clear to me that they solve important problems and I’d basically like to be using them already.

I think some of the ‘extra’ stuff flakes do is interesting and might be necessary, but if nixos-rebuild and nix-env worked as well with Niv as they do with nix-channel, I’d just start using Niv for my NixOS configurations tomorrow.

So that’s what’s holding me back or turning me off when it comes to flakes, but the limbo state of flakes (and the other new features in nixUnstable) also puts me off recommending Nix to others. :frowning:

You only get so much interest and patience when you’re trying to get someone to try new things, and I don’t want to waste the one chance I might have with someone recommending a version of Nix which is only almost there. Here’s what I mean:

  1. Nix is making its way into the repositories of package managers for other distros (it’s already in Debian, Gentoo, and FreeBSD Ports, while it’s on the way in NetBSD’s pkgsrc). This is awesome, because it means trying Nix can be easier, safer and more manageable. It could put it in a similar position to Flatpak or Snap for users who want to get versions of packages or whole packages that aren’t available in their base distribution, where its presence in the base distribution makes it feel like a natural addition rather than grafted on, the way a glorified curl|bash installer can seem. But, for example, the Debian package doesn’t set up NIX_PATH the same way as NixOS or our tarball installer. But why deal with holding someone’s hand in figuring out what their NIX_PATH is and where it comes from, especially when I know that ‘any day now’ Nix won’t depend on NIX_PATH anyway? It also doesn’t add any channels by default at all, so even setting aside the impurity, nix-env won’t know how to install anything out of the box. Best, it seems, to just wait a bit before recommending Ubuntu users apt install nix-bin nix-systemd-setup.

  2. Same thing for recommending NixOS for local desktops or simple server setups when the selling point is reproducibility and the best tool we have for pinning NixOS setups is unreleased/unsupported and kinda incomplete. I want to wait at least until we have something like devos a little more settled so that I can tell someone: ‘here’s how you set up NixOS and pin it to a commit, here’s where you put the config for each host, here’s where you put any custom modules you want, here’s where you put custom packages— and you can start thinking about how that stuff works and whether you want to rearrange it as you become comfortable’, knowing that the starting point for them is simple and unlikely to change from what I use or know in the near future.

  3. Similarly, all of the CLI improvements in nixUnstable, the fact that those will still change, and the massive performance difference yielded by flakes thanks to the evaluation caching it supports all make me want to put off encouraging people from really trying and learning the Nix CLI until after that stuff is released. It also doesn’t seem great to just introduce them to the existing/old CLI when I know that something much better is fairly well developed, mostly documented, and fairly widely used. Some of this is stuff that the community can work out independent of the first stable release of Nix that enables flakes, as seems to be going on inside devos and flake-utils-plus. But to some extent, projects like this that aim to simplify or settle conventions for the use of flakes inherit this problem of volatility (likelihood to undergo breaking changes in the near or moderate term) from the flakes feature itself.

Overall, the sense that later (but soon) will be a MUCH better time to recommend Nix seems like a real problem to me. And it’s a problem that might stand in the way of the development of future Nix goodies. You can even imagine something like a deflationary spiral except the investment being disincentivized is developer time in the Nix ecosystem. Perhaps this worry is overstated. Lots of things are definitely still moving forward in the Nix ecosystem and it’s exciting to see. But I can’t help but wonder if we’re missing out because of this effect.

I think maybe I’d rather have a flakes interface that’s settled and sucks, or see a promise of stability actually get broken, than see Nix spend another year or two in almost-there limbo for the basic functionality of pinning package sources and incorporating third-party repositories in a way that every tool in the Nix universe is expected to support. If we have to start out with an initial version of flakes that’s kinda clunky, I know I’ll figure out a way to deal with it and I bet the rest of the community will too. But the expectation that this implementation could disappear or radically change some time soon makes me (and, I assume, others) hesitate to bother really figuring out how to do that now. :-\


edited to make it a little clearer, more readable, and more polite (no important differences)

16 Likes

Generally I’ve liked having flakes as an opinionated structure. I’ve eschewed the wrapper utilities and have used them to pin, to create Hydra jobs, NixOS configurations, etc.

I’m not sure how to address all of the problems people have discussed, I only know how to solve problems: one at a time. So if people want to work with flakes, get something working, or even to help me understand what difficulties people are having, please put an hour or two in my office hours.

4 Likes

As a mostly passive observer and Nix/OS user for more than 2 years who has just started contributing to nixpkgs, I am barely at a stage to have a fully informed opinion, but thanks to @nrdxp we are having a much needed discussion on this.

Personally, I feel Flakes are starting to hold Nix back.

  • We have bunch of nice improvements in Nix unstable, including new CLI and CAS. Some of these are entangled with Flakes, and are holding back wider adoption because flakes are so underbaked.
  • I remember being excited about flakes way back after watching NixCon session by Eelco. But then the RFC went nowhere. This has many issues. I wanted to try out flakes immediately, but the documentation was so poor that for a Nix newbie it was too much. I can already see the rift happening, because nix already has sad documentation story and even then ongoing efforts are split between flakes and legacy way.
  • Latest NixCon brought possible future of TOML flakes, which means whatever work I have put in, may get deprecated.
  • I agree on the point that RFC did not discuss and pan out implementation details. we got an implementation and then there was discussion on how it could be take forward.
  • We still lack any kind of transparency on what the future looks like, and from the history of this feature, even if we get one, I’m afraid it will be like RFC where we’ll be told what side we are already on.

The list may look like its all over the place and its partially my fault for being bad at communication, but here is what I would personally like to see.

  • Flakes solve real problems, lets first decide what they are and in what order they should be solved. In short, scope definition.
  • A fully open discussion on the above.
  • A timeline on the above. It can stretch for 5 years and will be understandable because such changes take time.
  • Consistent releases that solve individual parts from above. Let the small feature permeate slowly, so the community gets time to adjust and test and hash out finishing touches going forward.

I fell in love with Nix/OS the first time I tried it. I hope to continue using it, and I hope to be able to use it professionally some day. But as it stands, its already niche, and further ongoing lack of clarity is really going to hurt long term. Let this not be Python3 moment for us.

14 Likes

I’m closing this thread to encourage people to split their improvement points into single issues like @domenkozar asked. If you have concerns about this, please direct message me.

3 Likes

I received a bunch of feedback that I shouldn’t have closed this, so I’m reopening it.

8 Likes

Some notes from office hours:

  • builtins.getFlake is not well-known. With a new enough Nix it allows non-flake uses to find and refer to flake-enabled packages/overlays/etc.
  • recommend not using flake-utils or other “flake wrappers” until after one is comfortable with making a few explicit flakes
  • it helps to at first only use flakes with devShell = import ./shell.nix {inherit pkgs;}; and defaultPackage = import ./default.nix {inherit pkgs;} as a way to ease adoption and support both legacy and new (once comfortable one can adopt flake-compat)
  • avoid nix profile (vice nix-env) until one is comfortable
  • people are used to using nix-shell for both build-time debugging, quick test of programs, and development shell. It helps to clearly split these consideration; flakes help here, though it can be non-obvious. (nix develop and nix shell)
  • apps != packages
8 Likes

I didn’t see a thread specifically for this, so I’ll just add my feedback here. [Mods: feel free to move this post to another thread if it fits better]

Background: I started using Nix about 9 months ago, and had to decide to start out using flakes or using classic Nix. I looked into flakes and decided to learn classic Nix first. One reason was to experience the pain points for myself, before seeing the solution. The other was because it seemed like flakes required a lot more overhead and boilerplate than just writing nix expressions (and keeping careful control of nixpkgs by pinning or another method). One main source of “overhead” was system: why should I have to worry about this if I’m not cross-compiling? I don’t in classic Nix.

Feedback: system makes much more sense as an input. I strongly support this proposal.

9 Likes

Thanks, Tom! I hope it was still useful for me to get my general thoughts out, but you’ve inspired me to take the plunge so that I can revise those thoughts and round them out with more Immediately actionable questions/problems.

I finally started rewriting my Nix configs last night, starting with a simple, flake-based NixOS setup. I’m still copying over bits of my old config and working out how I want to organize it for non-NixOS hosts, but I’ll start a new thread (and probably link it here) soliciting feedback on my configuration once it’s more or less complete. :slight_smile:

So far I’ve chosen to use flake-utils-plus to keep that boilerplate outside of my flake.nix file, and I’ve been happy with that choice. I still might not have tried flakes without it. After I have something I like, I’ll test my intuitions by producing an equivalent flake.nix without it, but I really like the concision of my flake.nix so far. This strikes me as a very good proposal.

3 Likes

Thanks! There’s Provide builtins.currentSystem as a input to flakes or make system a parameter · Issue #3843 · NixOS/nix · GitHub for further discussion around the proposal.

3 Likes

people are used to using nix-shell for both build-time debugging, quick test of programs, and development shell. It helps to clearly split these consideration; flakes help here, though it can be non-obvious. ( nix develop and nix shel

I’ve tried moving from nix develop to nix shell but the fact that nix shell only manipulates PATH instead of arbitrary environment variables makes it unfit for my purpose.

For example, I want to set XDG_DATA_DIRS to make bash auto completion to work automatically for things in my shell; and similarly I want to have man pages work as soon as I enter the shell (MANPATH).

For it to replace my mkShell usage it will need to implement that feature at least…

5 Likes

@Ericson2314 writes on that issue:

[M]aybe we should implement more of Flakes in Nix. Factoring out lib/ and such things to make a standard library of sorts so we still have some agility. But at that point, what are flakes? Just something to manage inputs / lockfile without much extra structure?

This seems to me to be the single most urgent need that flakes meet.

  1. Is there any opposition among people (who might be skeptical about flakes in general) to getting such a minimal implementation of flakes merged and stabilized as quickly as is feasible? (@payas? @DianaOlympos ?)

  2. How might releasing a minimal version of flakes in that way ‘box in’ Nix developers so that the other goals of flakes are harder to achieve in the future?

  3. What are those ‘other goals of flakes’? I can think of some, but maybe y’all can help me enumerate more. It seems that among them are:

    • defining a more fixed, conventional structure than default.nix for Nix projects of all sorts
    • letting Nix the package manager better understand applications as opposed to raw collections of files
    • integrating some kind of registry for repositories of software packaged with Nix (the current ‘global’ registry)
    • and what else?
3 Likes

That seems to be a proper issue. Do you know if this is in-progress or has a GitHub issue? Or does nix shell intentionally not manipulate XDG vars as part of the design?

Until then you should be able to achieve what you need with devShell and shellHook.

2 Likes

I mean my main question if we do that is “What are we trying to do” and “what are flakes”

Right now flakes only make sense to me if they are taken as a whole as a “nixpkgs 2.0” basically. I may be wrong and i am happy to be. I think a solution here would be to open a small RFC describing the minimalistic goal we want to achieve as a first step. And that may require discussions here or elsewhere about what would go into it.

2 Likes

Is there any opposition among people (who might be skeptical about flakes in general) to getting such a minimal implementation of flakes merged and stabilized as quickly as is feasible?

I don’t see a problem with that, and agree on this speeding up the path to 2.4-3.0.

But your 2nd point is tangentially related with a separate doubt of mine, in that how much of a churn will it be on the ecosystem?

@DianaOlympos put it better in above comment. I always imagined flakes as eventual future where entire Nixpkgs is converted to flakes for individual packages. And I imagine a lot of people have overlapping sets and subsets of expectations from Flakes in general. How about we start listing them, and then go through them for prioritising?

I’ll start with mine.

  1. Dependency pinning/lockfiles by default. I like this one being a built in thing.
  2. Saner file structure. Right now, default.nix is a convention, while flake.nix becomes a standard.
  3. I would like the sub-flake for sub-module story sorted, but I’m not fully clear in this.
  4. I understand the implicit understanding and explicit design decisions towards that is eventual reduction of/on reliance of Nixpkgs. Which is why, even though 90%+ flakes have Nixpkgs as input, and I’m pretty sure 90%+ searches through “nix search” are for nixpkgs, it is still not one of the defaults. In short, I happen to like nixpkgs the monorepo, and would like to hash its future out under flakes.
3 Likes

Per the quotation in comment you are replying to, the answer is:

[j]ust something to manage inputs / lockfile without much extra structure

Is that more or less clear, or not really? Are the editorial substitutions I made I clarifying your question fair?

Everyone who is using flakes right now is using the very same Nixpkgs we all already know and love. So in a way, the question of what flakes are for if not transforming the structure of Nixpkgs is very easy to answer: look at how flake users are already using flakes, and you’ll see uses that don’t involve any such change.

For example, personal use case #1 for me is pinning the inputs (including nixpkgs) for NixOS systems that I manage locally with nixos-rebuild, without a bunch of kludge. (Alternatives include having to tell NixOS to set up nixpkgs sources in /etc, point NIX_PATH to them, and run nixos-rebuild twice every time I update the pinned version of Nixpkgs, and cloning a bunch of copies of Nixpkgs as submodules into my personal Nix configuration repo. Both kinda suck.)

Every deployment tool that supports flakes likewise presents a similar use case.

Another use case which doesn’t involve any changes to Nixpkgs at all is better facilitating the use of other people’s custom modules and packages. For example, the only reason the Nix User Repository really needs to exist is to use the overlay system to wrangle a bunch of repositories of Nix code into a usable namespace. Flakes make it really easy to pull in another flake user’s packages or modules, which is great because not everything that is usable is also suitable for inclusion in Nixpkgs, or has an author who wants to do the work to see it merged into Nixpkgs.


My hope with those prompts was to use the fact that flakes has an implementation that is already used to discard vague, open-ended questions like ‘what are we trying to do [in the end]’. We don’t have to answer that question to begin standardizing/releasing flakes if we look at the existing uses, find the most popular ones, and come to agree on some overlapping subset of them, like ‘ok, flakes are at least XYZ’. The candidate I’ve suggested for ‘XYZ’ here is managing and locking inputs, as suggested on the GH issue mentioned earlier.

2 Likes

I have a flake.nix that XDG_DATA_DIRS is exported inside a mkShell:

{
  description = "My Python application";

  inputs = {
    nixpkgs = {
      url = "github:nixos/nixpkgs/nixos-unstable";
    };
    flake-utils = {
      url = "github:numtide/flake-utils";
    };
    poetry2nix-src = {
      url = "github:nix-community/poetry2nix";
    };
  };

  outputs = { self, nixpkgs, poetry2nix-src, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};

        poetry2nix = import poetry2nix-src {
          inherit pkgs;
          poetry = pkgs.poetry;
        };

        customOverrides = self: super: {
          # Overrides go here
        };

        my_env = pkgs.poetry2nix.mkPoetryEnv {
          projectDir = ./.;
          preferWheels = true;
          overrides =
            [ poetry2nix.defaultPoetryOverrides ];
        };

      in
      {
        devShell = pkgs.mkShell {
          buildInputs = with pkgs; [
            poetry
            my_env
            glib
            gsettings-desktop-schemas
            gtk3
          ];

          nativeBuildInputs = with pkgs; [
            wrapGAppsHook
          ];

          shellHook = ''
            export XDG_DATA_DIRS="$GSETTINGS_SCHEMAS_PATH:${pkgs.gsettings-desktop-schemas}/share/gsettings-schemas/${pkgs.gsettings-desktop-schemas.name}:${pkgs.gtk3}/share/gsettings-schemas/${pkgs.gtk3.name}:${pkgs.hicolor-icon-theme}/share:${pkgs.gnome3.defaultIconTheme}/share"
          '';
        };
      });
}
1 Like