Infinite recursion in module with mkMerge

Help
1

I’ve been trying to create a module that will essentially merge two sets of attributes. Here’s the module:

{ config, lib, pkgs, ... }:
with lib;
let
  cfg = config.tilde.modules.configuration;
in {
  options.tilde.modules.configuration = {
    machine = mkOption {
      type = types.attrs;
      default = { };
    };

    user = mkOption {
      type = types.attrs;
      default = { };
    };
  };

  config = mkMerge [ cfg.machine cfg.user ];
}

And the usage of the module:

{
  imports = [
    ./modules/configuration
  ];

  tilde.modules.configuration = { };
}

When I try to build as part of a nix-darwin configuration, I get an infinite recursion encountered error:

building the system configuration...
error: while evaluating 'evalModules' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:62:17, called from /nix/store/0882sb0p0lhf4b964apsbbfpijl94ngz-darwin/darwin/eval-config.nix:25:10:
while evaluating the attribute '_module.freeformType' at undefined position:
while evaluating 'g' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/attrsets.nix:276:19, called from undefined position:
while evaluating anonymous function at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:139:72, called from /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/attrsets.nix:279:20:
while evaluating the attribute 'value' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:512:9:
while evaluating the option `_module.freeformType':
while evaluating the attribute 'mergedValue' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:544:5:
while evaluating the attribute 'values' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:538:9:
while evaluating the attribute 'values' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:637:7:
while evaluating 'byName' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:362:25, called from /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:378:22:
while evaluating 'byName' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:362:25, called from /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:374:21:
while evaluating anonymous function at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:337:19, called from /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:337:8:
while evaluating 'pushDownProperties' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:576:24, called from /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:337:73:
while evaluating 'pushDownProperties' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:576:24, called from /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:578:7:
while evaluating the module argument `config' in "/Users/eturkeltaub/tilde/modules/configuration":
while evaluating the attribute 'config' at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:130:21:
infinite recursion encountered, at /nix/store/pci4ssg62nr8mg3kax4ccrpzbz4vn5lc-nixpkgs-21.03pre262944.f18ba0425d8/nixpkgs/lib/modules.nix:130:21

It seems like I can’t refer to cfg inside of mkMerge, but I have no idea why that’s the case. Can anyone help me out?

2

In theory the result of mkmerge could contain new options for tilde.modules.configuration therefore it’s recursing infinitely. You need to make sure that this overlap can not happen.

1 Like
3

Is there any way to avoid this? The result of mkMerge won’t need to access the tilde.modules.configuration options.

4

As it potentially can have those keys, it would mean that those options would need to be merged again due to how the nodule system and final merging works.

Merge them under a key that is not tilde and it should work without infinite recursion.

5

Do you mean something like this?

{ config, lib, pkgs, ... }:
with lib;
let
  cfg = config.modules.configuration;
in {
  options.modules.configuration = {
    machine = mkOption {
      type = types.attrs;
      default = { };
    };

    user = mkOption {
      type = types.attrs;
      default = { };
    };
  };

  config = mkMerge [ (cfg.machine) (cfg.user) ];
}
6

No, I meant something like config.foo = mkMerhe ...

As long as you merge directly into the top-level config, you will keep the the infinite recursion as your merged results still could contain the keys you want to merge over.

1 Like
7

Is there any way to apply this sort of pattern then, where some configurations get merged into the top level? I could use an example, I’m a bit of a Nix newcomer :slight_smile:

8

No. Not that I am aware of. Unless you have it at the top-level from the very first moment.

What do you actually want? Mirror top-level configuration in your 2 sets?

9

Basically I’d like to have definitions for machines (nix-darwin or NixOS) and definitions for users (home-manager) and then use combinations of those to create a configuration. I could do this with just imports but I like the module pattern because it allows for a set of required options and merges applies that to a default configuration (so I can pass the networking hostname to a machine definition, and it will merge that in with the rest of the default machine configuration).

10

Common approach is to provide different entrypoint for the config using nixos-config entry in the nix path. Or even provide host and user entries to import from.

{
  imports = [ <host> <user> ];
}

This at least works when not using flakes.

11

I seem to have the same Problem:

{
  options.myModule = mkOption {
    type = types.attrsOf types.anything;
    default = {};
  };
  config = mkMerge (
    attrsets.mapAttrsToList (
      name: value: {}
    )
    config.myModule
  );
}

The actual module I was trying to make was larger, but I broke it down to this.

What I don’t really understand, why would it cause an infinite recursion error, if it could contain options for itself? Shouldn’t the infinite recursion be caused by the containing itself instead of the possibility?

I mean something like this:

{
  options.myModule.enable = mkEnableOption "myModule";
  config = mkIf config.myModule.enable {};
}

would literally be the same thing, wouldn’t it?
It could contain options, that would cause infinite recursion, but doesn’t.

12

no. the infinite recursion is caused by some config attribute depending, in some way, on itself in a way the module system cannot resolve. config (the argument) is (to first approximation) a fixed point of mkMerge, so try to imagine the config (attribute) you write as though every config (argument) had been replaced with the entire value bound to config (attribute). in your first case this process goes on forever, so you get an infinite recursion. in your second example it stops after one iteration, so you don’t get an infinite recursion.

importantly note that nix is a lazy language, so only values that are actually used count towards producing an infinite recursion. that’s why you get an infinite recursion out of

{ options.myModule = mkEnableOption "";
  config = mkMerge (mapAttrsToList (_: _: {}) config); }) ]; }

but not out of

{ options.myModule = mkEnableOption "";
  config = mkMerge (mapAttrsToList (_: _: {}) { a = config; }); }) ]; }

the first case uses the attribute names (and thus the value) of config, the second one does not use config at all. if we introduce a use of config with a trace we get an infinite recursion again:

{ options.myModule = mkEnableOption "";
  config = mkMerge (mapAttrsToList (_: c: trace c {}) { a = config; }); }) ]; }

mkIf gets some additional special handling on top of all this or it would create infinite recursion pretty much every time it’s used.

2 Likes
13

Just last week I held a Nix Hour relating to recursion in the module system, and it also covered getting infinite recursion with mkMerge and how to work around it. Feel free to check it out, comes with timestamps :slight_smile: https://www.youtube.com/live/cZjOzOHb2ow

And if you want something else covered in this format, feel free to ask a question directly by joining, or writing it down in GitHub - tweag/nix-hour: Questions for the weekly Nix Hour beforehand

5 Likes
14

Here’s an approach by @udf

This takes the keys on the first level and mkMerges their values.

However, it needs to get a list of keys to recurse into, why? Is it because listing all the keys triggers the recursion alert?

2 Likes
15

@Infinisil Thanks for sharing the tips. To avoid mkMerge recursion, you recommend pushing the list one level down. In my case, since I use quite a lot top level attrs, that’s a bit cumbersome.

Do you have any idea why @udf’s approach (posted above) could work? The last step where the top-level attrs are pulled, seems to be a no-op, since the list of attrs don’t contain any recursive value at all.

16

mkMerge is a red herring here. The issue is that you’re using an attribute inside config to define the set of attribute names in config. Values of an attrset cannot be retrieved until after the set of attributes have been determined.

nix-repl> let attrs = { foo = { bar = 1; }; } // attrs.foo; in attrs
error:
       … in the right operand of the update (//) operator
         at «string»:1:37:
            1| let attrs = { foo = { bar = 1; }; } // attrs.foo; in attrs
             |                                     ^

       error: infinite recursion encountered
       at «string»:1:40:
            1| let attrs = { foo = { bar = 1; }; } // attrs.foo; in attrs
             |                                        ^
1 Like
17

Makes sense.

However @udf’s approach can break out of the recursion by accessing a specific attribute:

{
  config.systemd = (mkMergeTopLevel (...).systemd);
}

But accessing attrs.foo (or attrs.bar) still results in recursion. Is there an example that explains it more closely?

18

udf’s version works because it frontloads the information about which attrs need to get into config, allowing that to be determined without knowing the value of the last mapAttrsToList arg (which is an option, and thus needs config to have already figured out its set of attribute names in order to access it).

Note that udf couldn’t make it work until he explicitly listed the options he wanted to set, rather than trying to get them out of the mapAttrsToList result somehow.

I suppose the most direct translation of udf’s fix to my example would be:

nix-repl> :p let attrs = { foo = { bar = 1; }; } // { bar = attrs.foo.bar; }; in attrs
{
  bar = 1;
  foo = { bar = 1; };
}
1 Like
19

Thank you for explaining it so clearly.

Thinking about it more:

  1. Do you think “values of an attrset cannot be retrieved until after the set of attributes have been determined” is a limitation of the nix implementation itself, since it’s pretty clear what the result of let attrs = { foo = { bar = 1; }; } // attrs.foo; in attrs should be, and it’s not really recursive by nature.
  2. Given this limitation, do you think explicitly listing the attribute names as udf did is currently the best solution? Other than pushing the list value down, I can’t think of another one. But listing the top attribute names statically doesn’t always work, since it can change based on a config value.
20

It’s fair to call it a limitation of the nix implementation. Attrsets could have been defined with finer-grained laziness, allowing access to some attributes without having yet realized whether other attributes exist. That may have had unpleasant performance implications, with more thunks being generated and evaluated for the same code.

I think I’d prefer a solution involving defining a helper function and then using it in each top level attr, at least where you were writing the map function’s output as a literal attrset in the file anyway, as in udf’s case:

let
  merger = f: lib.mkMerge (mapAttrsToList f cfg.whatever);
in
{
  options = ...;
  config = {
    foo = merger (n: v: {
      ...
    });
    bar = merger (n: v: {
      ...
    });
  };
}
1 Like