I’ve been organizing overlays for my NixOS configuration into separate files of the form
self: super: { foo = /*...*/; }
and have some code in my configuration.nix like
{
importOverlaysDir = path:
with builtins;
map (n: import (path + ("/" + n))) (filter (n:
match ".*\\.nix" n != null
|| pathExists (path + ("/" + n + "/default.nix")))
(attrNames (readDir path)));
}
(I first saw this pattern used in jwiegley’s nix-config).
I’m running into trouble when I want to compose multiple overlays in a single file. Using the composeExtensions method recommended here, I end up with something like
Unfortunately, this results in infinite recursion. I think this has to do with the fact that the result depends on self, but I don’t see a way around this, since we’ve already consumed two arguments and need to return a set (not a function).
I suppose an alternative would be to change the structure so overlay files contain lists of overlays, e.g.
[ (self: super: { foo = /* ... */ }) /* ... */ ]
and then just join the lists before passing to overlays, but I have a hunch that I might be missing an alternative method that’s closer to a minimal change.
Thanks, that is a really helpful example! (I like the let inherit syntax; this is the first I’d noticed that). I’m assuming the self = composed self {} line models how Nix computes the final package set, as the fixed point of the composition of all of the overlays?
I also tried modifying your example to have the same form as mine (i.e. a function taking self and super):
Yeah let inherit (pkgs) lib; in is nice. You can also write with { inherit (pkgs) lib; }; and it will do the same thing, but let inherit feels cleaner.
self = composed self {} is calculating the fixed point, and is pretty much what nixpkgs itself does for overlays. nixpkgs itself is actually using lib.extends instead of lib.composeExtensions; if you want to do precisely what it does it would look something like
but it’s a little less obvious how to nest this into other overlays, whereas with my approach you can ditch the self = composed self {} line and just return composed self super, like you did in your reply.
If you did want to use the lib.extends approach you’d end up writing it like
The easiest way to get infinite recursion in an overlay is to refer to self outside of the top-level attribute set that your overlay returns (or as part of one of the keys). For example, the following overlay will recurse infinitely:
self: super:
let
foo = self.callPackage ./foo.nix;
in
{
…
}
Any references to self have to be within the expression given as one of the attribute values. However references to super can go anywhere.
Thanks, this has been enlightening for me! I’m still a bit confused about a few points:
This is a useful guideline, and it sort of makes sense to me on an intuitive level. I was playing with your example to try to get a deeper understanding of how the infinite recursion happens, and ended up with the following example:
self: super:
let foo = self.callPackage ./foo.nix { };
in { inherit (foo) foo; }
# ./foo.nix
{ }: { foo = "foo"; }
But I’m not able to reproduce infinite recursion with this example. Also, this seems equivalent to just writing
which has self only in the expression for a top-level attribute value, so seems to meet the above criterion for an acceptable use of self?
I managed to pare down my initial example into a MWE reproducing the infinite recursion (which doesn’t involve lib.composeExtensions after all):
self: super:
let
path = super.writeText "my-overlay" ''
self: super: {
foo = "foo";
}
'';
in (import path) self super
I’ve also noticed that there is no infinite recursion if path just points to a local file, like
self: super:
(import ./overlay.nix) self super
# ./overlay.nix
self: super: { foo = "foo"; }
(But I don’t think this is helpful, since I eventually want to replace writeText with fetchFromGitHub as in my original example.)
I don’t think this case violates your advice above; self doesn’t appear except as an argument, and when passed to the sub-overlay (but that part is the same as the second example, which doesn’t have the issue with infinite recursion).
Sorry, you’re right. I threw my example together without testing it, and it’s actually fine because the foo binding isn’t strictly evaluated. If the evaluation happens outside of the attribute set it’s a problem, so something like the following:
if self.stdenv.hostPlatform.isDarwin then {
# attribute set for darwin
} else {
# attribute set for everyone else
}
This clearly must reach into self outside the attribute set.
Oh this one is really subtle. You wrote writeText but internally it’s got references to self. Specifically, writeText comes from pkgs/build-support/trivial-builders.nix. This file is imported as a built-in overlay in stage.nix. The file takes arguments and the import explicitly resolves those arguments via the self reference. So when you write super.writeText that ultimately ends up evaluating self.stdenv.
And since you’re importing from the resulting derivation rather than merely referencing path in your attr set, you’re strictly evaluating it.
If you need to write a file to disk from a string (rather than just copying it from a local path) you can bypass nixpkgs and use builtins.toFile name str. So here that would look like
self: super:
let
path = builtins.toFile "my-overlay" ''
self: super: {
foo = "foo";
}
'';
in (import path) self super
Not only does this bypass nixpkgs but it also doesn’t require import-from-derivation to work. However it has limitations; in particular, you can’t reference the result of a derivation in such a file.
In this case the builtins.toFile doesn’t really help. In order to do this you have to bypass fetchFromGitHub as well. You can either use builtins.fetchTarball with a manually-computed path to the GitHub archive you want (GitHub will try and push a .zip on you, but you can change the URL to .tar.gz), or you can use builtins.fetchGit if you need the full git repo for some reason.
Just wanted to say thanks @lilyball , I got it working with your help. The key as you said was switching to builtins.fetchTarball, since fetchFromGitHub internally references self, resulting in infinite recursion. I ended up with: