Infinite recursion getting started with overlays

I’m trying to get my head around using overlays to add custom packages to my config, but I’m getting infinite recursion errors in some cases, and I don’t understand why I’m only seeing them in some scenarios and not others.

Here’s a simple example case that shows the problem:

let
  testApp = {writeShellApplication}:
    writeShellApplication {
      name = "test.sh";
      text = "echo hi";
    };

  packageSet = {callPackage}: {
    test = callPackage testApp {};
  };

  pkgs = import <nixpkgs> {};

  overlaidPackage = import <nixpkgs> {
    overlays = [(final: prev: {test = prev.callPackage testApp {};})];
  };

  overlaidSet = import <nixpkgs> {
    overlays = [(final: prev: prev.callPackages packageSet {})];
  };
in {
  fromPackage = pkgs.callPackage testApp {};
  fromPackageSet = (pkgs.callPackages packageSet {}).test;
  fromOverlay = overlaidPackage.test;
  fromOverlaySet = overlaidSet.test;
}

Running nix-instantiate test.nix with -A fromPackage, -A fromPackageSet and -A fromOverlay all work just fine, but when I try with -A fromOverlaySet I get an infinite recursion error. I can’t see why the first three attributes can be built, but the fourth can’t!

callPackages is typically only used to create nested package sets whose attributes are then brought to top-level:

let
  testApp = {writeShellApplication}:
    writeShellApplication {
      name = "test.sh";
      text = "echo hi";
    };

  packageSet = {callPackage}: {
    test = callPackage testApp {};
  };

  overlaidSet = import ./. {
    overlays = [
      (final: prev: {
        inherit (prev.callPackages packageSet {}) test;
      })
    ];
  };
in {
  fromOverlaySet = overlaidSet.test;
}

Might have something to do with the fact that Nix attribute names are evaluated strictly, since toggling the crash variable in the following prevents the infinite recursion:

let
  testApp = {writeShellApplication}:
    writeShellApplication { name = "test.sh"; text = "echo hi"; };

  packageSet = {callPackage}: { test = callPackage testApp {}; };

  overlaidSet = import ./. {
    overlays = [
      (final: prev: let
        nested = prev.callPackages packageSet {};
        crash = true;
      in
      if crash then
        prev.lib.trace (builtins.attrNames nested) { test = nested.test; }
      else
        { test = prev.lib.trace (builtins.attrNames nested) nested.test; })
    ];
  };
in {
  fromOverlaySet = overlaidSet.test;
}

Or, looking at the trace, you might have encountered some cursed interaction with splicing.

The fact that if I replace builtins.attrNames nested with a simpler value like 54, it is printed multiple times in the crash branch also supports this hypothesis.

1 Like

Side note: you’ll always want to prefer final rather than prev, unless (of course) you’re using the thing that you’re overlaying (e.g. final: prev: { foo = prev.foo.override { ... }; })

1 Like

@jtojnar Thank you! I’m slightly surprised that {inherit (prev.callPackages p {}) test;} produces a different result here to prev.callPackages p {}, but it clearly does!

@waffle8946 Good spot, thank you! I think I was originally using final, but started using prev as I was experimenting with the recursion errors. I’d probably have left it in place had you not pointed it out…

The difference between the two is that the callPackages application doesn’t have to be evaluated to know whether the first one has a given attribute, but it does have to be evaluated to know if the second one does.

When asked to evaluate fromOverlaySet, the evaluator will go through these motions, approximately:

  • Does overlaidSet contain an attribute called test?
  • To answer that, do any of the overlays added in overlaidSet contain an attribute called test?
  • To answer that, does the value of prev.callPackages packageSet {} contain an attribute called test?
  • To answer that, prev.callPackages will have to resolve the parameters of packageSet and find each of them in the final Nixpkgs set.
    • (Why the final Nixpkgs set, when it’s prev.callPackages being used here? The only difference between prev.callPackages and final.callPackages is if an overlay happens to redefine callPackages. Unmodified, the definition of callPackages refers circularly to the final spliced value.)
  • So does overlaidSet contain an attribute called callPackage?
  • To answer that…

The infinite loop can be short-circuited if the third question can be answered without evaluating callPackages.

2 Likes

Ah! That makes sense now, thank you!

I’d been testing things in nix repl, and could see both forms were producing an attribute set that had a single test attribute, so I couldn’t work out why only one caused an infinite loop. I’d missed that the inherit form means the evaluator can – and does – safely assume the thing it’s being asked to inherit is available to do so without needing to evaluate what it’s inheriting from.

1 Like