Portable Submodules

10 Likes

The implementation for this feature can be found here:

2 Likes

Nice experiment! As a resident module system expert I do need to point out how this can be done with the standard Nixpkgs module system though :wink:

Itā€™s a bit of a mouthful, but the original problem can be fixed like this:

    b = lib.mkMerge [
      (lib.modules.mkAliasAndWrapDefsWithPriority lib.id options.a)
      {
        static = 99;
      }
    ];
Full code
(import <nixpkgs/lib>).evalModules {
  modules = [
    ({ options, lib, ... }:
    let
      type = lib.types.submodule ({ config, ... }: {
        options = {
          static = lib.mkOption {
            type = lib.types.int;
            default = 0;
          };

          dynamic = lib.mkOption {
            type = lib.types.int;
            default = config.static + 1;
          };
        };
      });
    in
    {
      options = {
        a = lib.mkOption {
          inherit type;
        };

        b = lib.mkOption {
          inherit type;
        };
      };

      config = {
        b = lib.mkMerge [
          (lib.modules.mkAliasAndWrapDefsWithPriority lib.id options.a)
          {
            static = 99;
          }
        ];
      };
    })
  ];
}

Iā€™d support having something like lib.mkAlias for this use case instead, itā€™s definitely not the first time Iā€™ve come across this, so we could have

    b = lib.mkMerge [
      (lib.mkAlias options.a)
      {
        static = 99;
      }
    ];

Whatā€™s also neat here is that config.a doesnā€™t even need to be defined: In the full code Iā€™m able to remove the default for a and the above still works.


Itā€™s also possible to get the .extend as you showed with @roberthā€™s extendModules work:

      type = lib.types.submodule ({ config, extendModules, ... }: {
        options = {
          extend = lib.mkOption {
            default = module: (extendModules {
              modules = [ module ];
            }).config;
          };
          # ...
        };
      });

# ...

      config = {
        b = config.a.extend {
          static = 99;
        };
      };

Full code
(import <nixpkgs/lib>).evalModules {
  modules = [
    ({ options, config, lib, ... }:
    let
      type = lib.types.submodule ({ config, extendModules, ... }: {
        options = {
          extend = lib.mkOption {
            default = module: (extendModules {
              modules = [ module ];
            }).config;
          };
          static = lib.mkOption {
            type = lib.types.int;
            default = 0;
          };

          dynamic = lib.mkOption {
            type = lib.types.int;
            default = config.static + 1;
          };
        };
      });
    in
    {
      options = {
        a = lib.mkOption {
          inherit type;
          default = {};
        };

        b = lib.mkOption {
          inherit type;
        };
      };

      config = {
        b = config.a.extend {
          static = 99;
        };
      };
    })
  ];
}

Arguably types.submodule should expose the .extendModules more directly, or even a simpler .extend that takes additional modules and returns the new .config, which would remove the need to declare the .extend yourself (though youā€™d have to access it through options.a.extend).

5 Likes

Also, remembering the attempted Packages Modules WG, arguably the main problem with using something like the module system for packages was performance. @DavHau experimented with this idea by making mkDerivation use the module system, and determined that evaluation was 2-3x slower. Shouldnā€™t come at a surprise though, the module system does much more with its type checking and co, but we considered that a blocker for the scale of Nixpkgs.

4 Likes

Portables would just be a convenience module in the Nixpkgs Module System.
Itā€™s an instance of an application agnostic module, so thatā€™s fun.

# extend-option.nix (or portable.nix)
{ extendModules, lib, ... }: {
  options.extend = lib.mkOption {
    description = "Extend this configuration or submodule with another module";
  };
  config.extend = module: (extendModules {
    modules = [ module ];
  }).config;
}

Usage:

# configuration.nix
{
  # EDIT: oops, posted wrong snippet 
  # services.nginx.virtualHosts."foo" = { ... }: {
    imports = [ ./extend-option.nix ];
  # };
}
$ nixos-rebuild repl
nix-repl> config.networking.hostName
"nixos"

nix-repl> c = config.extend { networking.hostName = lib.mkForce "bar"; }
nix-repl> c.networking.hostName
"bar"

I donā€™t feel like it makes things portable; just extensible, but maybe something flew over my head at 1 AM :sweat_smile:

I think it might be confusing if we start a module based Portable Service Layer implementation, because that would just use imports, and have no use for extendModules or this extend option iirc.

For context:

2 Likes

Iā€™ve also stumbled across wanting to re-use modules defined within the module system while building dream2nix which is also a packaging framework built on the module system.

The secret sauce Iā€™ve found is the deferredModule type from nixpkgs. It allows to specify, so to speak, raw-modules which are not evaluated automatically, in contrast to how it would happen with types.submodule.

deferredModule allows you to construct a pattern where the user facing options are wrapped as a deferredModule, and later the framework evaluates all these module definitions to the final result using the usual submodule type. This ensures that values are never evaluated to early and all priorities and dependencies stay in tact until the final evaluation.

Example:

{ config, lib, ... }:
let
  packageInterface = {config, ...}: {
    options = {
      static = lib.mkOption {
        type = lib.types.int;
        default = 0;
      };

      dynamic = lib.mkOption {
        type = lib.types.int;
        default = config.static + 1;
      };
    };
  };
in
{
  options = {
    package_a = lib.mkOption {
      type = lib.types.deferredModule;
      default = {config, ...}: {
        static = 0;
        dynamic = config.static + 1;
      };
    };

    package_a_final = lib.mkOption {
      type = lib.types.submoduleWith {
        modules = [
          packageInterface
          config.package_a
        ];
      };
      default = {};
    };
  };

  config = {
    package_a.static = lib.mkForce 99;
    # this is also possible
    package_a.imports = [./user-override.nix];
  };
}
5 Likes

I had originally tried implementing this using extendModules, but kept having issues where things were not propagating through multiple layers. It is possible I was holding it wrong, but I ended up making things work by tracking modules and evaluating them manually.

I think repeated, or ā€œnestedā€ use worked fine when I used it for a test framework once. Or maybe you had something different in mind for what youā€™re calling layers? Sounds like the same thing though.

Anyway, if you have a reproducer, I could take a look

2 Likes