I'm not convinced flake inputs should exist

[nrdxp]

@L-as

I mean purity is the whole point of Nix. FWIW you don’t have to copy the world to /nix/store, and there are PRs to this effect being worked on right now (lazy trees etc.).

Sure I should have been more clear: “flakes version of purity”. I get the lazy trees PR is being worked on, but it’s been in that stage for years now, also. I also can’t help but wonder if it is actually necessary. Nix is a lazy language, there are such things as chroot’s and namespaces. Do we really want to break Nix’s laziness when it comes to what it copies to the store when we could, e.g. just isolate the evaluator itself?

Just saying, there may be simpler, more useful ways to do what flakes are trying to do in a way that is perhaps more naturally compatible with Nix’s design, and doesn’t require an entire virtual file-system layer.

[con-f-use]

It seems to me like you can do everything flake inputs can do without using flake inputs

You cannot conveniently update dependencies without copying hashes to and fro. You cannot easily override dependencies in code you fetch. The only real design mistake with flakes, from my point of view, was the inclusion of the flake registry. That’s just a re-hash of the nightmare that is channels.

[SergeK]

I think having a mechanism for declaring and defining “inputs” across projects systemically, in a pre-defined structure, is actually good, regardless of flakes. In particular I very much sympathize with the view, flake: add Nixpkgs default arguments as input by roberth · Pull Request #160061 · NixOS/nixpkgs · GitHub, that flake inputs turn out to implement a dependency injection system, like callPackage does (if even accidentally). I think this touches a problem of utmost importance: preventing the sprawl of ad hoc vendoring in Nix projects.

In Nixpkgs we mend the vendoring in upstream projects forcing everything to live in the same fixpoint and sandboxing the build. However, when using an out-of-tree Nix project we have no sandboxing mechanism to ensure this project doesn’t spawn extraneous Nixpkgs (e.g. with an extended unfree predicate) instances or pull in more projects we’re not aware of. With flakes, all of that complexity becomes visible in the lock file, and you can also attempt implementing a form of a fixpoint (inputs.${x}.follows) to reduce it. I absolutely do wish to have some form “purity” for this, the “sandboxing” for evaluation.

Ironically, while flakes happen to partially address the issue, they might have been keeping us from designing and adopting a mechanism specifically for the task…

[matklad]

@L-as

It seems to me like you can do everything
via builtins.fetch*,

One thing you can’t do is managing transitive dependencies:

You depend on A and B. A depends on C1, B depends on C2, where C1 and C2 are different versions of the same C.

There are two choices: either you duplicate C1 and C2 and use both versions at the same time, or you unify them somehow into the single version.

To do this, you need some component that looks at the full set of input constraints from all transitive dependencies, and then solves the constraint set to find the actual set of inputs to use.

To clarify, I am not saying that flaks actually solve this problem, just pointing out that the hard problem here is not “how I get my inputs” but rather “how my inputs get their inputs in a way that doesn’t lead to gratuitous duplication”.

[waffle8946]

@L-as

If you didn’t use inputs, it would be natural to make the flake a function that takes a nixpkgs as argument.

That’s a very optimistic assessment, have you seen actual nix code or how people mindlessly copypaste?
I’ll bet you 100 (insert currency) we’d have people using import <nixpkgs> { ... } instead.
If you think that’s nuts, we already have people doing exactly that in flake.nix.

@L-as

you can do everything flake inputs can do without using flake inputs,
via builtins.fetch*,

Well there’s a real funny bug with builtins.getFlake, which was discussed in nixos-option: rewrite as a nix script by FireyFly · Pull Request #313497 · NixOS/nixpkgs · GitHub. Not that it can’t be mitigated, of course it can by explicitly specifying that it’s a git-type flake, but it’s a subtle error that will just bite you later. And now, knowing that getFlake works differently than flake inputs, I can no longer consider that getFlake (or its cousin fetchTree) is an actual drop-in replacement.

@SergeK

flake inputs turn out to implement a dependency injection system … (if even accidentally )

That’s shocking that it’s considered unintentional; proliferation of nixpkgs instances is the biggest reason for slow eval, so we certainly need such an injection system (flakes or not), and simply - isn’t follows just dep injection by nature? What else was it meant to be used for?

[fzakaria]

I was a long time non-flake user specifically because I like the old CLI output.

I did eventually give in…tbh I’m not sure what I gained really aside from just better support with other flakes…
(I guess importing nixosModules is a bit easier)

There is a purity to Nix Flakes about how it includes system as an input but I wonder if that one papercut could have been fixed in the traditional way.

The boilerplate to creating a flakes is high.
I’m also trying to remember when I want to wrap things for each system or not

[cafkafk]

@L-as

There is the issue of “nixpkgs” bloat, “nixpkgs” appearing many times again and again.

If some inputs expects a specific version of nixpkgs different from your own, that bloat means it keeps working. If you have more than one input using different versions of nixpkgs, you need both their nixpkgs to avoid breakage.

[L-as]

Yes, and in those cases, I assume you would hard-code it instead of using the nixpkgs passed by the “caller flake”.

[L-as]

Honestly, I do think it could have been fixed “in the traditional way”.
If the Nix file is inside a repository, you could have prevented accessing files outside the repository,
and otherwise, you could have prevented accessing files in directories above the root file you evaluate from.
But this wouldn’t be that different from what a flake currently is, of course besides the inputs and schema stuff.

[L-as]

Flakes are still experimental, and we need to find a way forward that solves this, and I think this sort of in-between feature is more amenable to making unexperimental and stable.

[tomberek]

There is already an object we commonly use that takes a package set as an input in a very flexible way, overlays. They are already intended to be used by a consumer without regard for the Nixpkgs of the upstream flake. Notice that overlays don’t hard-code a system either. Overlays also shouldn’t depend on any inputs, just on final/prev, making them quite independent. But they are complicated.

The first few flakes we saw were awkward, and then we had a proliferation of flake libraries intended to hide the boilerplate, and “system” and then add niceties. These have had a chance to explore the design space. I think it comes down to two difficult to grok concepts, overlays and callPackage.

If everyone implemented their flakes with proper overlays, well partitioned and organzied, and if everyone including beginers understood overlays, we might be in a better position. Instead we end up with packages that are hard to re-use elsewhere, hard-coding them into modules, and so forth. People are even forced into this situation because it is easier than the alternative. It is also the case the overlays are just inherently complicated, and useful, and hard to learn, and have too many footguns. And we made the correct behavior require expertise, but the easy behavor damaging in the long run.

In some sense, the problem of too many Nixpkgs means that there was also a need, and that people are leveraging this capability, but without the tooling or the idioms to guide them.

I dont think merely having people use niv or raw getFlake is the answer. I suspect we would end up in the same position, with yet another form of having multiple nixpkgs, but now less structured. At least flake inputs are all in one place, localizing the problem and allowing for tooling to handle them. I do want to eventually get back to a proposal from last year about updating how lockfiles treat transitive locks (Reuse input lock files · Issue #7730 · NixOS/nix · GitHub).

One approach I’ve been enjoying is twofold, based around the notion of a “recipe”, a function that produces a derivation (pkg-fun, the default.nix 's you find all over Nixpkgs, a callPackage-able, etc):

1. Remove the need to understand super/self/final and callPackage in 95% of cases. Most people just want to add a leaf package or do a simple override. Not some deep compiler replacement or cross-compilation. Let people define an attribute set/tree, where each value is a recipe. There is an fairly mechanical translation of this form into overlays that adds callPackage and does the right fixed-point magic.

2. Re-use. These more limited forms are easier to handle than overlays, less footguns and less power. Attrsets of recipes can be manipulated predictably and composed together without needing to understand fixed points. They can be grabbed from a flake without reading the upstream lockfile. One can enforce this by a “flake = false” (perhaps that becomes the default?) and passing dummy values as inputs, if it errors it means someone’s overlay accessed a package value directly instead of doing proper composition.

None of this requries any technical change. There is work that can make it nicer; and tooling + templates + guides to make it understandable. Farid was making a tool to more easily manage follows statements, simply having a better interfaces to visualize and manage inputs might go a long way.

[fricklerhandwerk]

@tomberek I fully agree that the “recipe as a function from derivations to (attribute set of) derivation” pattern has the exact right properties in your framing of learnability and composability, and that we should promote it to beginners. This is pretty much what we’re doing in the tutorials on nix.dev, this is the direction pkgs/by-name is taking, and where an overhaul of the Nixpkgs manual could help to instill and solidify the right ideas in casual users.

But this doesn’t address the problem of organizing remote sources, because it’s a completely orthogonal issue. I’d even go as far as saying that the recipe pattern is at odds with the “everything is a flake” pattern, because at the point where you package everything as a function at the level of default.nix, you may as well provide default values for your inputs based on your local pins, however you manage them, and leave the rest to callPackage (one of those great misnomers) and override. Almost inevitably you’ll then have to ground everything in some version of Nixpkgs, ideally controlled by the consumer.

All that remains is a user interface for the imperative (essentially end-user) use case of the likes of nix run, which, as I argued multiple times, is probably better suited as a Nixpkgs idiom since it would much better live off derivation metadata rather than some superimposed rule for what an “installable” or “app” is. All Nix itself would need to provide is the language for composition and process isolation for realising derivations.

[amjoseph]

@tomberek

I think it comes down to two difficult to grok concepts, overlays and callPackage.

If everyone implemented their flakes with proper overlays, well partitioned and organzied, and if everyone including beginers understood overlays, we might be in a better position.

Exactly. Overlays and callPackage are the secret sauce of nixpkgs. You can try to make them easier to use, but if you replace them with something less powerful the end result will be much worse than nixpkgs.

@tomberek

Remove the need to understand super/self/final and callPackage in 95% of cases. Most people just want to add a leaf package or do a simple override. Not some deep compiler replacement or cross-compilation. Let people define an attribute set/tree, where each value is a recipe. There is an fairly mechanical translation of this form into overlays that adds callPackage and does the right fixed-point magic.

This is what infuse.nix does.

I’ve been working on it for the last nine months, and will be releasing it next week have released it (and, shortly afterward, two other projects that use it). I’ve converted my (massive) set of overlays to use infuse and can’t imagine ever going back.

Here’s what my override of pkgs.xrdp looks like today; notice that this requires a nested overrideAttrs:

infuse pkgs {
  xrdp.__input.systemd.__assign = null;
  xrdp.__output.env.NIX_CFLAGS_COMPILE.__append = " -w";
  xrdp.__output.passthru.xorgxrdp.__output.configureFlags.__append = ["--without-fuse"];
};

Here’s what it looks like without infuse:

pkgs // {
  xrdp = (pkgs.xrdp.override {
      systemd = null;
    })
    .overrideAttrs(previousAttrs: {
      env = previousAttrs.env or {} // {
        NIX_CFLAGS_COMPILE =
          (previousAttrs.env.NIX_CFLAGS_COMPILE or "")
          + " -w";
      };
      passthru = previousAttrs.passthru or {} // {
        xorgxrdp = previousAttrs.passthru.xorgxrdp
          .overrideAttrs (previousAttrs: {
            configureFlags = (previousAttrs.configureFlags or []) ++ [
              "--without-fuse"
            ];
          });
      };
    });
}

We always had the right power tools, it just took a while to figure out how to make them less awkward.

[L-as]

I have some qualms with overlays and callPackage, e.g. that input packages are derived implicitly from the argument list, which creates problems when you don’t have a single global namespace, as happens with e.g. Haskell packages. Then you have the issue that you want to depend on some system package called x, but there’s a Haskell package with the same name, so you can’t depend on it! Horrible! Of course you could have some encoding system where you can prefix the name with _root_ such that it’d _root_x to get the top-level x, and _root__haskellPackages_x to get the one in the haskellPackages namespace, but this is also horrible.
callPackage in its current form is not a viable option.

Other than that, I very much endorse the direction, but especially in this direction it does not make sense to use flake inputs as we use them now.

I feel like a good first step would be to give non-flake Nix code the same purity guarantees that flakes get, i.e. as I described above, for any code in a Nix repo, restrict access to files checked in to Git, by default. It would break very little code I imagine. Doing it for files outside of a Git repository would break much more code.

[L-as]

OTOH, perhaps the more important part,

as an end-user, you want to be able to go into a repository and just nix build.
But as a library consumer, you probably do just want to do the equivalent of flake = false, and
you’re probably right, it would probably make sense to have this be the default. This would be a big breaking change though.

[SergeK]

@L-as

want to depend on some system package called x, but there’s a Haskell package with the same name, so you can’t depend on it

IMO this is because makeScopeXXXXXX and callPackage functions aren’t powerful enough:

Frustrations about splicing

Yes I think it’s not splicing per se that is cursed, it’s the way we implement scopes and callPackage, which hinders both cross-compilation and situations like “there’s foobar in python3Packages but I need to ask for foobar from the top-level pkgs”. Then there’s also the general issue of expressing more complex graphs than makeScope newScope chains when working out of tree I wish I could do a more explicit/fine-grained dependency injection, sort of like # package.nix { dependencies.wayland-s…

EDIT: Where I was going with the linked snippet is that instead of feeding just a lambda into callPackage, we could say a “recipe” is a structure with a lambda and the metadata explaining how to fill in the lambda’s arguments. In particular, we could try and express the solution to your problem as “take X from the scope two levels up”

It is notable that we’re not discussing flakes anymore, it likely never was about flakes

[tomberek]

infuse

@amjoseph Looks interesting! I’ve been using a similar construct, using functions as the delimiter for traversal:

using pkgs {
    xrdp = { xrdp }: xrdp.override {pulseaudioSupport = true;};
    my-hello = {hello}: hello:
    pkg-from-file = ./default.nix;
    pkgA-from-dir = ./a;
    acmePackages = ./pkgs;
    my-other-hello = {hello}: hello.overrideAttrs {name = "my-name";};
    my-cmd = {runCommand}: runCommand "mine" {} "touch $out";
    python3Packages.thing = {requests}: requests.overrideAttrs {name = "my-requests";};
    lib.add1 = {}: a: a + 1;
    data.b = {lib}: lib.add1 12;
    _hiddenDep = {stdenv, lib}: stdenv.mkDerivation ....;
}

This allows for a simple lib.recursiveUpdate for merging. So I can imagine incorporating infuse as well, to simplify the cases of override/overrideAttrs. In my implementation i started adding some nice features with an “expand” step that would interpret some things to add niceties ( a = {}; roughly means a = {a}: a;, or handling path directories. Your lib is adding another set of handlers specialized to overrideAttrs and override (plus with better names!). I do like your idea of interpreting lists as pipes, so far I’ve only been map’ing through them. Mind if I incorporate some of those ideas/conventions into my lib?

random ideas

Two more random ideas to bounce around:

1. Make use of the otherwise useless <thing> syntax in flakes to access/define inputs instead of NIX_PATH. Sure, it’s an easy way to do the wrong thing, but easy to syntactically detect without eval, build tooling can manage them, less weaving of self or inputs everywhere. You can get the same behavior as Go, where importing a new module anywhere is detected and becomes part of the root, tracked and locked.

2. Subflakes don’t quite work the way people expect, but peolple keep trying to use them to fill a need. Introduce a new concept “crystals” (just a tiny snowflake!). It’s definition looks like another flake, but it can never have its own lockfile. Instead its inputs declarations bubble-up the to host flake’s lock, and all the input values flow back to the crystal during eval. This is similar to the Cargo.toml and workspaces concept. Allows to group and subdivide input definitions (PoC: init: crystal support · flox/nix@8a684bd · GitHub).

Flakes also provide standardization and a URL scheme to give things names. Inputs are also needed in some form, but i agree they should be improved.

callPackage and overlays

IMO this is because makeScopeXXXXXX and callPackage functions aren’t powerful enough:

@SergeK Can you clarify? I’ve been digging into these two and making variants that are aware of hierarchical attrsets / package sets. Or do you mean something else?

[amjoseph]

@amjoseph

This is what infuse.nix does. I’ve been working on it for the last nine months, and will be releasing it next week (and, shortly afterward, two other projects that use it). I’ve converted my (massive) set of overlays to use infuse and can’t imagine ever going back.

Better late than never, I guess. Two mirrors:

• https://codeberg.org/amjoseph/infuse.nix (prettier markdown rendering)
• https://git.sr.ht/~amjoseph/infuse.nix (send patches here)

Tutorial and the announcement will be posted tomorrow. And a few other things. And I will fix all the typos in the README.md.

I will bump the version number to 1.0 once I double-check everything.

And I will catch up on replies to comments. I’m not ignoring anybody, I’m just backlogged!

This is what my overlays look like using infuse.nix.

The hardest part to get right was how to deal with empty attrsets; this took a really really long time to figure out. It’s very counterintuitive.

[amjoseph]

@SergeK

IMO this is because makeScopeXXXXXX and callPackage functions aren’t powerful enough:

Exactly. But there wasn’t much demand to give them this power, mostly because…

@L-as

which creates problems when you don’t have a single global namespace, as happens with e.g. Haskell packages.

… until very recently – hierarchical packagesets were hellaciously painful to override (both .override and .overrideAttrs). This isn’t just about overlays; nixpkgs itself uses .override and .overrideAttrs very extensively.

Making hierarchical packagesets as easy to deal with as flat packagesets was a major motivation for infuse.nix; see this example headline example. Now I think we are ready to consider a version of (or alternative to) callPackage that allows hierarchical names. It’s a shame that nix doesn’t allow hierarchical arguments though, like

{ lib
, stdenv
, haskellPackages.compiler
}:

@tomberek

@SergeK Can you clarify? I’ve been digging into these two and making variants that are aware of hierarchical attrsets / package sets.

I’m pretty sure this doesn’t exist yet, but it should. There was a lot of pressure to not create any new hierarchical packagesets (there is no rustPackages) because of how painful overriding was.

@L-as

I feel like a good first step would be to give non-flake Nix code the same purity guarantees that flakes get, i.e. as I described above, for any code in a Nix repo, restrict access to files checked in to Git, by default.

You can do this with --option pure-eval true – no flakes or --extra-experimental-features needed. I don’t think the purity comes from flakes, it’s that flakes take away the --option pure-eval false escape hatch.

[amjoseph]

@tomberek

Make use of the otherwise useless <thing> syntax in flakes to access/define inputs instead of NIX_PATH

Is there a way to support this in flakes-compat?

Also, for a dose of crazy, check out this reuse of the <thing> syntax.