My understanding of overlays is that when I reference final.somePackage, I’ll get the one after all overlays have been applied, and if I reference prev.somePackage, then it will reference the package from before the current overlay has been applied.
When I do nix-build -A packageFromPrev on the file above, then I get the error message This is somePackage from overlay1, and when I do nix-build -A packageFromFinal, I get This is somePackage from overlay3.
That’s all well, no surprises here.
But when I try to do the same with the callPackage helper, then this doesn’t work as expected:
nix-build -A packageFromPrevCallPackage prints This is somePackage from overlay3, and I just don’t understand why.
Why isn’t prev.callPackage restricted to the packages in prev?
callPackage is essentially defined with a reference to final, though in the code in question, I believe the variable is self. That’s where it gets its packages from. You could say callPackage has a dependency on every package in nixpkgs, and those dependencies are updated by the overlay, as they should be. If you replace a library in a later overlay, a package depending on that libarary in an earlier prev will get the replacement, too.
I don’t think I understand. It sounds like this is by design, but if I use prev.callPackage, then I explicitly don’t want further overlays to change these dependencies. Granted, that’s a questionable use-case, but at least that’s how I always understood overlays to be intended to be used, and it makes sense to me because that way, callPackage and overlays are completely orthogonal.
But now I learn that the semantics of overlays and callPackage are intertwined in a really unintuitive way.
The semantics aren’t entwined. callPackage is behaving like any other element of the package set. The other elements of the package set just behave in a way you probably never realized.
Your image of overlays seems to be a layering concept, as in, each overlay generates a new pkgs value from the previous one. However it’s more accurate to think of the entire package set as a big loop of many strings, and each overlay is a knot or twist among those strings. Every dependency relationship follows the entire loop around when being resolved, except for special cases where the dependency is loaded from prev in an overlay, so later overlays normally affect earlier prev values so long as there’s a dependency in between.
It’s an extension of the same functionality that makes it possible to replace dependencies with overlays in the first place:
nix-repl> f = lib.makeExtensible (self: { foo = self.bar; })
nix-repl> g = f.extend (final: prev: { bar = "test"; })
nix-repl> g.foo
"test"
Or more clearly:
nix-repl> f = lib.makeExtensible (self: { foo = self.bar; })
nix-repl> g = f.extend (final: prev: { bar = prev.foo; })
nix-repl> g.foo
error: infinite recursion encountered
at «string»:1:30:
1| lib.makeExtensible (self: { foo = self.bar; })
| ^
Your mental model would probably expect the error here to be attribute 'bar' missing, but it’s an infinite recursion instead.
nix-build -A packageFromPrev evaluates to the hello package. But if I swap the order of overlays around, then it evaluates to error: attribute 'somePackage' missing. So prev seems to be without the “next” overlays here, not just without the current one.
It wouldn’t even make sense to me to have prev be “pkgs with all overlays except this one”, because that would mean that two overlays independently overriding the same package can result in infinite recursion, e.g., with an override like this:
You’re right in that my mental model of overlays is that it consists of multiple layers over an initial package set that depends on the “final” layer (i.e., the layer after all overlays are applied).
f = lib.makeExtensible (self: { foo = self.bar; })
In my mental model self is the final layer (because everything else wouldn’t make sense), so of course this results in an infinite recursion.
The only thing that doesn’t fit this model is callPackage, because that somehow escapes these layers and always refers to final. So i really don’t see how the semantics of overlays and callPackage aren’t closely coupled.
Everything in nixpkgs gets its dependencies from self, including callPackage. The difference between using callPackage and direct reference is simply that, directness. Whether there is or is not a node in between affects whether the dependency is resolved with another loop around the fixed point or not.
Hm? That seems… extreme. Overlays aren’t that complicated. prev is only the overlays before this one, and final is with all overlays are applied. The only quirk is that due to laziness, things in prev can actually refer to things in final; which is where this callPackage behavior comes from. But the overlay aspect isn’t bad. All the overlays are called in succession with an accumulator as the prev argument, with the quirk that their fixed point is provided to all of them as final.
As for callPackage, you can think of it like this at a high level:
callPackage = f: f (builtins.intersectAttrs (builtins.functionArgs f) final)
The fact that people usually get it from prev as in prev.callPackage doesn’t change the fact that it’s extracting arguments to give to f from final, not any version of the prev accumulator.
And in the unreflected and not quite needed approach they are often used in cause more trouble than not, as they cause unexpected recompilations or get forgotten about when debugging why an update from nixpkgs isn’t visible in someones configuration.
Of course there are valid usecases for overlays, though in at least 90% of cases I have seen them used, a straight inline package definition/override would have been more than appropriate.
Thanks everyone! The small example of how to build a smaller, controlled package set with lib.makeExtensible allowed me to experiment and figure out where my misunderstanding comes from:
prev.callPackage is the same as final.callPackage precisely because callPackage and overlays are completely orthogonal, and the final package set is simply the only one that makes sense as auto args for callPackage. Trying to implement the semantics as I originally understood them would couple callPackage more closely to overlays, because with each overlay application a new callPackage function would need to be injected into the package set automatically.
Overlays aren’t that complicated, but they also aren’t particularly easy to intuitively get right. It’s not clear when to use final and when to use prev, and my attempts to abstract from the internals (e.g., use prev when you want to access packages that aren’t overridden yet, use final for the set of final packages) never quite cover the whole story - see my confusion about callPackage in this question. I can kinda understand why one would refrain from using overlays when not absolutely necessary.