Improving flakes

Is there any opposition among people (who might be skeptical about flakes in general) to getting such a minimal implementation of flakes merged and stabilized as quickly as is feasible?

I don’t see a problem with that, and agree on this speeding up the path to 2.4-3.0.

But your 2nd point is tangentially related with a separate doubt of mine, in that how much of a churn will it be on the ecosystem?

@DianaOlympos put it better in above comment. I always imagined flakes as eventual future where entire Nixpkgs is converted to flakes for individual packages. And I imagine a lot of people have overlapping sets and subsets of expectations from Flakes in general. How about we start listing them, and then go through them for prioritising?

I’ll start with mine.

  1. Dependency pinning/lockfiles by default. I like this one being a built in thing.
  2. Saner file structure. Right now, default.nix is a convention, while flake.nix becomes a standard.
  3. I would like the sub-flake for sub-module story sorted, but I’m not fully clear in this.
  4. I understand the implicit understanding and explicit design decisions towards that is eventual reduction of/on reliance of Nixpkgs. Which is why, even though 90%+ flakes have Nixpkgs as input, and I’m pretty sure 90%+ searches through “nix search” are for nixpkgs, it is still not one of the defaults. In short, I happen to like nixpkgs the monorepo, and would like to hash its future out under flakes.
3 Likes

Per the quotation in comment you are replying to, the answer is:

[j]ust something to manage inputs / lockfile without much extra structure

Is that more or less clear, or not really? Are the editorial substitutions I made I clarifying your question fair?

Everyone who is using flakes right now is using the very same Nixpkgs we all already know and love. So in a way, the question of what flakes are for if not transforming the structure of Nixpkgs is very easy to answer: look at how flake users are already using flakes, and you’ll see uses that don’t involve any such change.

For example, personal use case #1 for me is pinning the inputs (including nixpkgs) for NixOS systems that I manage locally with nixos-rebuild, without a bunch of kludge. (Alternatives include having to tell NixOS to set up nixpkgs sources in /etc, point NIX_PATH to them, and run nixos-rebuild twice every time I update the pinned version of Nixpkgs, and cloning a bunch of copies of Nixpkgs as submodules into my personal Nix configuration repo. Both kinda suck.)

Every deployment tool that supports flakes likewise presents a similar use case.

Another use case which doesn’t involve any changes to Nixpkgs at all is better facilitating the use of other people’s custom modules and packages. For example, the only reason the Nix User Repository really needs to exist is to use the overlay system to wrangle a bunch of repositories of Nix code into a usable namespace. Flakes make it really easy to pull in another flake user’s packages or modules, which is great because not everything that is usable is also suitable for inclusion in Nixpkgs, or has an author who wants to do the work to see it merged into Nixpkgs.


My hope with those prompts was to use the fact that flakes has an implementation that is already used to discard vague, open-ended questions like ‘what are we trying to do [in the end]’. We don’t have to answer that question to begin standardizing/releasing flakes if we look at the existing uses, find the most popular ones, and come to agree on some overlapping subset of them, like ‘ok, flakes are at least XYZ’. The candidate I’ve suggested for ‘XYZ’ here is managing and locking inputs, as suggested on the GH issue mentioned earlier.

2 Likes

I have a flake.nix that XDG_DATA_DIRS is exported inside a mkShell:

{
  description = "My Python application";

  inputs = {
    nixpkgs = {
      url = "github:nixos/nixpkgs/nixos-unstable";
    };
    flake-utils = {
      url = "github:numtide/flake-utils";
    };
    poetry2nix-src = {
      url = "github:nix-community/poetry2nix";
    };
  };

  outputs = { self, nixpkgs, poetry2nix-src, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};

        poetry2nix = import poetry2nix-src {
          inherit pkgs;
          poetry = pkgs.poetry;
        };

        customOverrides = self: super: {
          # Overrides go here
        };

        my_env = pkgs.poetry2nix.mkPoetryEnv {
          projectDir = ./.;
          preferWheels = true;
          overrides =
            [ poetry2nix.defaultPoetryOverrides ];
        };

      in
      {
        devShell = pkgs.mkShell {
          buildInputs = with pkgs; [
            poetry
            my_env
            glib
            gsettings-desktop-schemas
            gtk3
          ];

          nativeBuildInputs = with pkgs; [
            wrapGAppsHook
          ];

          shellHook = ''
            export XDG_DATA_DIRS="$GSETTINGS_SCHEMAS_PATH:${pkgs.gsettings-desktop-schemas}/share/gsettings-schemas/${pkgs.gsettings-desktop-schemas.name}:${pkgs.gtk3}/share/gsettings-schemas/${pkgs.gtk3.name}:${pkgs.hicolor-icon-theme}/share:${pkgs.gnome3.defaultIconTheme}/share"
          '';
        };
      });
}
1 Like

Probably and that is not clearer than the first time i read it. We can already do it and yes it is painful. But if that is all we are achieving with this for that much cost and complexity, i would strongly advise to consider flakes like at best a spike at a possible solution and to go back to the drawing board.

I did try and i have to disagree, i see a lot of work to be done to be able to get a setup that work, as was pointed multiple time.

See that is interesting. Because i feel the opposite. It makes it harder. Channels have a ton of problems, but they are really easy to use. I have yet to be able to use flakes as well.

[quote=“pxc, post:30, topic:12831”]
My hope with those prompts was to use the fact that flakes has an implementation that is already used to discard vague, open-ended questions like ‘what are we trying to do [in the end]’. [/quote]

But that is not true. What we have is an implementation that a large part of the community feel does not solve the problem or make it worse. That is exactly why we are talking of what they solve. We got given an implementation and told that was the way to go and now everyone is trying to find why the hell do we do this.

In that case, i will say that i consider flake an interesting experiment on the path toward that, but something that is not solving the problem enough while introducing enough pain to not be an acceptable solution. We should go back to the drawing board and establish a plan on how we want to explore that set of problems more and offer steps forward. Your answer is “what people that want flakes use it is what we want from it”. Except the people that do not use it do not want that. We are back to a fork.

6 Likes

I am aware of nix-shell and mkShell
mkShell even has XDG_DATA_DIRS built in these days (if you provide your tools through nativeBuildInputs ). This is what I was using before…

However the discussion is about the fact that nix-shell was never meant for ad-hoc dev environments. It’s really meant to debug nix builds. It sets all kinds of extra env variables one is mostly not interested int. This is why Nix 2.4 moved nix-shell to nix develop and for the ad-hoc environments came up with a new command called nix shell.

However it isn’t featureful enough to replace how I used to use nix-shell/mkShell. That’s what the whole my point was about.

In order for nix shell to replace nix-shell/mkShell it should have the option to set more environment variables than just PATH

However; today I Came up with the following .envrc file to emulate the usage I want:

{
  description = "A very basic flake";
  outputs = { self, nixpkgs }: {
    defaultPackage.x86_64-linux = nixpkgs.legacyPackages.x86_64-linux.buildEnv {
      name = "dev-shell";
      paths = with nixpkgs.legacyPackages.x86_64-linux; [
        kubectl
        (terraform_0_13.withPlugins (p: [ p.vultr p.ct ]))
        kubernetes-helm
        kustomize
      ];
    };
  };
}
nix build .

PATH_add $PWD/result/bin

export XDG_DATA_DIRS=$PWD/result/share:$XDG_DATA_DIRS

[[ -f .envrc.local ]] && source_env .envrc.local

it works well.

2 Likes

I think exhaustive monorepos of build recipes are a great basis for feeding package managers. There are tons working examples (besides Nixpkgs and Guix): Gentoo Portage, FreeBSD Ports, NetBSD/SmartOS/OpenIndiana pkgsrc, MacPorts, Homebrew, and Void Linux at least. I understand that Eelco expressed some doubts about the scalability of that approach on the original Flakes RFC, and I disagree with that. I also know that our disagreement doesn’t really matter.

There is no possible implementation of Nix flakes that makes Nix incapable of operating on a Nixpkgs-like monorepo of packages, or indeed Nixpkgs as we have it right now. The thing that powers Nixpkgs in a flakes-based future is the same thing that powers it now: the work of its maintainers. As long as there are people who find it useful and gratifying to maintain a unified set of packages with as much sharing of dependencies as possible, there will be one.

The concern that making it easier to use smaller, independent repositories of Nix packages will fracture Nixpkgs raises the question: what is keeping current Nixpkgs maintainers and contributors invested in the repository? Is it really the fact that splitting their packages out would be cumbersome? I doubt it for several reasons, but even if it were the case… isn’t the problem of a package set maintained by half-willing hostages of technical inconvenience way, way more dire than any imaginary outcome in a flakes-based future? The solution, in any case, seems to be in making sure Nixpkgs is as convenient and maintainable as possible, not in shunning features that enable other ways of composing collections of packages.

Something like this is exactly what I had in mind. :smiley:

As you can imagine, I’d rather punt on this. :laughing: But anyway: do you mean for Nixpkgs, or just something like ‘here’s a sane way you could use subflakes in a project if you wanted to, including a package set as large as Nixpkgs’?

I think there are good ways to give Nixpkgs a less uniquely privileged place in the Nix ecosystem without making it any less convenient to use, or threatening its survival. In this case, would the following simple solution work for you? I think it’s something like what I want (I think with option (b) for the defaults)

  • When nix search or nix profile install is run without arguments that specify a flake, Nix searches a bunch of flakes according to configuration. By default, it could search (a) all flakes in the registry, (b) only flakes that are pinned in the registry (since that’ll be fast), or (c) no flakes.

I don’t know what ‘cost’ means here, but if it means time going into writing the code that powers the current experimental version of flakes, it doesn’t matter— that cost has already been paid (and not by you or me) regardless of whether the flakes stuff gets revised, replaced, dropped entirely, or released as-is.

At any rate, the exercise I’ve started in this thread is to generate a proposition that potentially involves less complexity for end users than fully grokking flakes as they exist now by finalizing a portion of the flakes feature that does fewer things. So ‘that much complexity’ need not be part of the proposition.

This addresses something other than the point I was making, which is about usage of flakes requiring changes to Nixpkgs, and in particular dissolving the monorepo. Of course using flakes in a Nix project or a NixOS configuration is going to require changes within that project.

This is confusing. I don’t make a truth claim in the quotation you cite, and at any rate whether or not flakes already has users is a matter of fact and anyone with a web browser that can reach GitHub can see that it’s true.

So,

why did we do this

and

what problems are solved by flakes?

are not the same question. They’re also not the same kind of question— why we have widely used but unofficial (and somewhat controversial) implementation of flakes is a social and historical question, and what problems are solved by flakes is a technical question. Recently, @grahamc wrote a blog post addressing the historical question and a discussion was opened up on Discourse here.

The technical question is answerable in terms of the actual present usage of flakes regardless of whether you, or I, or anyone else likes the current implementation. (It’s also independent of the question of whether flakes uniquely solves those problems. But the fact that flakes have been chosen by many users despite being a bit cumbersome does indicate that those users perceive it to solve at least some of their problems better than existing alternatives do.)

My purpose in emphasizing existing usage right now is not to marginalize or turn away from the concerns of people who just don’t use flakes and haven’t bought in at all. Once we have a clear picture of the common portion in the needs of existing flake users, then we can try to integrate the goal of meeting those minimal, common feature completeness needs of existing flake users with the ergonomic needs of people who are less invested in built-in input management/locking functionality. This could be a smaller and less controversial task than trying to conform every ambition that the current flakes implementation and its users have so far had for the feature to those same ergonomic constraints. To get that scope reduction for the proposition of what to standardize and release, we can look at the existing usage. Once we have that, we can think clearly about what unobtrusive implementations of it would look and feel like. Then we could take a step forward without either totally ripping the more complete, experimental implementation of flakes out from under existing users, or presenting non-flakes users with new interfaces whose complexity is excessive or motivated by use cases which haven’t been practiced yet.

Toward clarifying the needs of people who worry that a finalized implementation of flakes (even, potentially, one that is smaller/simpler than the existing implementation) will get in their way, a survey for those users:

  • what operating systems do you currently use Nix on?
  • do you currently pin any Nix package sources?
    • if so, is that done inside a project for an application, a development environment, a NixOS configuration, or what?
  • do you currently do anything to configure NIX_PATH or directories in it (e.g., nix.nixPath on NixOS, setting up local checkouts, modifying /etc/bashrc on Debian or macOS, etc.)?
  • if you use nix-channel, how do you use it?
  • do you store some or all of your Nix configurations in source control? which and how?
8 Likes

We have a good discussion here, but its kind of stalled now. @nrdxp @pxc @ryantm and others, any way we can move to possible RFC target as mentions in OP with what we already have, and then continue to hash out larger outline under it?

1 Like

Yeah, so for me, I’m still trying to understand the best way to move forward with a proposal that will actual garner approval from all (or most) stateholders. Since the original flakes RFC was closed, I’m left wondering if the RFC process is even the appropriate place to suggest changes to flakes at all, and if not, then where do we have these dicussions in a more formalized fashion?

If it is deemed appropriate to go ahead and open an RRC on this, or perhaps we could start a separate RPC process specifically targeting experimetal features, then here’s what I’m thinking it could include:

  • It appears the most controversial feature of flakes, by far, is the registry. So perhaps, in an effort to get flakes merged faster, we just drop this feature entirely. Really, it doesn’t seem like we lose all that much without it, other than the shorthand for common flakes. Perhaps to mitigate this loss, we can simply add a configuration option to specify desired shorthands, with nixpkgs included as the default entry.

  • Also, it seems the major difficultly with merging flakes as they stand right now, is that their implementation is kinda tied up with the overhaul of the nix command in general, which is a quickly moving taget. Depending on how difficult this would be to architect, perhaps we could simply leave out the major changes to the UI, and only include the nix flake subcommand, leaving all other additions to continue as experimental features until the devs are content with the redesign.

    The biggest hurdle to this might be introducing the notion of the flake uri to the existing stable nix * commands. I’m not familiar enough with the codebase to know if this would be a feasible task.

  • Other than that, perhaps we should leave out any major changes to flakes for a separate RFC, and simply begin to identify a list of bugs/todos that exist in the current implementation, and start closing them up. Once done, we can just merge as is, and iterate on the design with each point release.

Of course, since I lack experience with the core nix codebase, my suggestions may end up being less than practical, but I guess that’s what a request for comments is for anyway right :sweat_smile:

11 Likes

No! Even though I do not like or understand why or what is in the registry, please do not remove that feature. Beeing able to write nixpkgs#hello rather than having to write github:nixos/nixpkgs#hello is a time safer!

Though I’d appreciate if usage of the registry would be restricted from a flake.nix, and neither implicit inputs nor flake: URLs would be allowed.

2 Likes

As a matter of structuring the process appropriately, I think packaging ideas into flysheets (similar to the system one proposed by @domenkozar & already referenced above) and let them stick around for a while is a valid option to prepare the field for consensus. They are also easy to x-link and graduate into a full-blown RFC.


Since not everyone uses https://notion.so, here are two alternatives:

Seems like this is possible without a registry.

We just need to say that flake-url without a prefix: points to a configurable location, with default github:nixos/. We get the short invocation syntax without having to maintain a registry.

What we lose is the ability to have more “top-level names” (e.g. nix run blender-bin), but do we invoke that so often to require a registry?

2 Likes

Sounds like a local only registry. I’m totally fine with a registry that does no remote look ups, but works locally only, as long as it gives the same freedom as the current one.

And I’d be even happier if it wasn’t available from within a flake.nix but only from the CLI.

4 Likes

Looking at the linked thing, I do not see anything beyond Discourse Wiki posts, except complete lack of content accessibility without Javascript and broken PgDn. And I would expect the timescale of discussions to not need parallel real-time editing of hackmd…

3 Likes

Hedgedoc is the FOSS fork/rename of codimd/hackmd. They renamed a while ago. I’d prefer Hedgedoc from a political point of view and actually from a feature set point of view. It sees more development these days.

You can read more about that history here:

3 Likes

This was essentially the second half of my suggestion: i.e. maintain some sort of alternative way of specifying flake shorthands. Perhaps a local registry, or perhaps a config option. In any case, it seems further brainstorming is needed before an RFC is even viable.

3 Likes

Agreed on all 3 points, this could be the lowest common multiple.

Even if anything specific to flakes can be delayed, we should really hash out RFC process for experimental features, as you mention. Simply relying on Experimental tag is not sufficient in Nix, IMO. One of the tweets in reply to @domenkozar 's introspective series of tweets on this topic was that it gets confusing for the masses, as the feature is experimental, but all the cool kids are already using it. It fragments the community and ongoing efforts.

3 Likes

Just wanted to emphasize re: the registries that

  1. There are currently 3 registries on the flakes builds of Nix (user, system, and global)

  2. Registries are just shortnames for URIs pointing to Nix code

  3. Only the global registry, being both built in and unconfigurable, is really any organizationally different from channels, which are also configurable shortnames for URIs pointing to Nix code (and come with Nix if you use the tarball installers)

aside from the global registry, Nix flake registries and the whole nix registry subcommand have no more basis for controversy in them than ol’ nix-channel. So to me, everything but the global registry obviously belongs in the MVP flakes release we’re trying to imagine here

3 Likes

Thanks to @Pacman99 (“eagle eye”), who spotted it and pointed me to it, I guess this is actually less of a problem than I thought, since flake inputs seem to be lazily fetched (not only evaled)..

This is so implicit, that maybe a clarifying conceptualization would suffice.

EDIT: I suggest using an _ prefix to denote local dependencies. For ecample if any library want’s to use nixpkgs and devshell as local deps for the devshell and the checks, we could conventionally import them as inputs._nixpkgs & inputs._devshell.

sorry the interruption

3 Likes

For example, I want to set XDG_DATA_DIRS to make bash auto completion to work automatically for things in my shell; and similarly I want to have man pages work as soon as I enter the shell ( MANPATH ).

I have partially implemented this in nix#4702, but seems like upstream is not too keen on introducing such features so far.

However I’d say this is much more related to the CLI redesign and the Installables abstraction than flakes.

1 Like

Lots of great ideas here. Do we have a any plan of action or Github issue where all this is condensed?

1 Like