Use lib.types system to merge attrsets without the module system?

Development
1

Hi,

I am trying to convert the current kernel config process from a string to a structured attrset (https://github.com/NixOS/nixpkgs/pull/41393), i.e., from the legacy:

linux_latest_9p = prev.pkgs.linux_latest.override({
     extraConfig = ''
       NET_9P y
      NET_9P_VIRTIO y
       NET_9P_DEBUG y
     '';
});

to the desired

linux_latest_9p = prev.pkgs.linux_latest.override({
     extraStructuredConfig = {
       NET_9P = yes;
       NET_9P_VIRTIO = yes;
       NET_9P_DEBUG = yes;
     };
});

Among the motivations is the ability to merge different configs so that programs can specify required dependencies (e.g., if you enable the openvswitch module, hopefully it should enable CONFIG_OPENVSWITCH ).

Instead of rewriting the merging logic (e.g., how to merge the 2 configs { NET_9P_DEBUG = yes; } and { NET_9P_DEBUG = no; }), I thought I could reuse the module logic mkMerge/mkIf etc. but this proves harder than I thought.
It seems the module system expects a list of paths config.foo.bar for any operation. I don’t want to rely on a single option boot.kernel.config, I want to use the lib/types.nix to “typecheck” the structured config of different kernels each with their own configuration. Each configuration entry should conform to

 kernelItem = types.submodule {
    options = {
      answer = mkOption {
        type = types.str;
        default = null;
        internal = true;
        description = ''
          For most options "y" or "m" or "n" but freeform.
        '';
      };

      optional = mkOption {
        type = types.bool;
        default = false;
        internal = true;
        description = ''
          Wether it should fail if not asked.
        '';
      };
    };
  };

Then I tried different things to merge the 2 configs { NET_9P_DEBUG = yes; } and { NET_9P_DEBUG = no; } but it is not straightforward.
Playing with the REPL I found out that I could merge two booleans via lib.types.bool.merge [] [{ value = false; file="papa"; } {value = true; file="toto";} ].

I proceeded likewise to run kernelItem.merge to merge 2 structured configs:
kernelItem.merge [ { file= “PLACEHOLDER”; value = { NET_9P_DEBUG = no; };} { file=“PLACEHOLDER”; value = { NET_9P_DEBUG = yes; optional = true; }; } ]
but I run into further trouble:

while evaluating the attribute '"9P_FSCACHE"' at /home/teto/nixpkgs3/lib/attrsets.nix:200:46:
while evaluating 'option' at /home/teto/nixpkgs3/lib/kernel.nix:53:12, called from /home/teto/nixpkgs3/pkgs/os-specific/linux/kernel/common-config.nix:473:27:
while evaluating 'traceValSeqFn' at /home/teto/nixpkgs3/lib/debug.nix:80:22, called from /home/teto/nixpkgs3/lib/kernel.nix:60:5:
while evaluating 'traceValFn' at /home/teto/nixpkgs3/lib/debug.nix:42:19, called from /home/teto/nixpkgs3/lib/debug.nix:80:25:
while evaluating 'id' at /home/teto/nixpkgs3/lib/trivial.nix:46:8, called from /home/teto/nixpkgs3/lib/debug.nix:42:29:
while evaluating the attribute '_module' at /home/teto/nixpkgs3/lib/attrsets.nix:200:46:
while evaluating the attribute 'args' at /home/teto/nixpkgs3/lib/attrsets.nix:200:46:
while evaluating anonymous function at /home/teto/nixpkgs3/lib/modules.nix:75:45, called from /home/teto/nixpkgs3/lib/attrsets.nix:200:54:
while evaluating the attribute 'value' at /home/teto/nixpkgs3/lib/modules.nix:312:9:
while evaluating the option `_module.args':
while evaluating the attribute 'mergedValue' at /home/teto/nixpkgs3/lib/modules.nix:345:5:
while evaluating anonymous function at /home/teto/nixpkgs3/lib/modules.nix:345:32, called from /home/teto/nixpkgs3/lib/modules.nix:345:19:
while evaluating 'merge' at /home/teto/nixpkgs3/lib/types.nix:274:20, called from /home/teto/nixpkgs3/lib/modules.nix:348:8:
while evaluating anonymous function at /home/teto/nixpkgs3/lib/attrsets.nix:199:9, called from /home/teto/nixpkgs3/lib/types.nix:275:9:
while evaluating 'filterAttrs' at /home/teto/nixpkgs3/lib/attrsets.nix:115:23, called from /home/teto/nixpkgs3/lib/types.nix:275:35:
while evaluating anonymous function at /home/teto/nixpkgs3/lib/lists.nix:104:41, called from /home/teto/nixpkgs3/lib/attrsets.nix:116:18:
while evaluating anonymous function at /home/teto/nixpkgs3/lib/attrsets.nix:116:29, called from undefined position:
while evaluating anonymous function at /home/teto/nixpkgs3/lib/types.nix:275:51, called from /home/teto/nixpkgs3/lib/attrsets.nix:116:62:
while evaluating the attribute 'name' at /home/teto/nixpkgs3/lib/attrsets.nix:335:7:
while evaluating anonymous function at /home/teto/nixpkgs3/lib/types.nix:275:86, called from /home/teto/nixpkgs3/lib/attrsets.nix:335:15:
while evaluating the attribute 'optionalValue' at /home/teto/nixpkgs3/lib/modules.nix:352:5:
while evaluating the attribute 'values' at /home/teto/nixpkgs3/lib/modules.nix:338:9:
while evaluating the attribute 'values' at /home/teto/nixpkgs3/lib/modules.nix:434:7:
while evaluating anonymous function at /home/teto/nixpkgs3/lib/lists.nix:104:41, called from /home/teto/nixpkgs3/lib/modules.nix:434:16:
while evaluating anonymous function at /home/teto/nixpkgs3/lib/lists.nix:104:41, called from /home/teto/nixpkgs3/lib/modules.nix:324:17:
while evaluating anonymous function at /home/teto/nixpkgs3/lib/modules.nix:324:28, called from undefined position:
while evaluating 'dischargeProperties' at /home/teto/nixpkgs3/lib/modules.nix:392:25, called from /home/teto/nixpkgs3/lib/modules.nix:325:62:
while evaluating the attribute 'value' at /home/teto/nixpkgs3/lib/types.nix:280:55:
while evaluating the attribute 'name' at /home/teto/nixpkgs3/lib/types.nix:368:13:
while evaluating 'last' at /home/teto/nixpkgs3/lib/lists.nix:512:10, called from /home/teto/nixpkgs3/lib/types.nix:368:25:
assertion failed at /home/teto/nixpkgs3/lib/lists.nix:513:5

I wonder if something similar is done anywhere in nixpkgs (i.e., bypass the whole module creation

{ config, lib, pkgs, ... }:

with lib;

let
  cfg = config.mymodule;

in {

  options.mymodule = {
  };

  config = mkIf cfg.enable ( ... )

 };

to directly call mkMerge on submodules and retrieve the merged value as in mkMerge [{ NET_9P_DEBUG = yes; } { NET_9P_DEBUG = no; }] returns sthg like NET_9P_DEBUG = yes;

NB: I already asked this on IRC without much success but I hope my request is clearer with more context.

1 Like
2

Hm, Though I understand the need for mkForce, why do you want to support mkIf?

Without mkIf and mkPrioirty/mkOverride simple attrset update (//) is enough.

3

Using // means the last attribute wins which is not what one would want. You may want to be notified if settings conflict.
Right now the whole kernel config is centralized in https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/kernel/common-config.nix#L45. mkIf is not mandatory but would be nice to have to add conditions on features/version.

4

Can you show an example of what would you like to encode with module system? I see you want some asserts/warnings to be defined.

You may want to be notified if settings conflict.

Notified? Can you show an example of such a situation?

5

@teto: is the goal to then re-inject the kernel into a NixOS setup?

It would be awesome if packages had their own toConfig function and schema attached to then but it’s not supported right now. I think it’s a great idea for various reasons, for example it would make it easier to upgrade the package and config schema at the same time.

Your best bet for now is to create your own NixOS module that looks a bit like this:

{ lib, config, pkgs, ... }:
with lib;
let
   toConfig = attrs: ""; # TODO: generate the kernel extraConfig for each key if the value is true
in;
{
  options = {
    boot.kernel.extraConfig = {
      NET_9P = mkEnableOption "NET_9P";
      # ...
    };
  };

  # ----
  boot.kernelPackages = config.kernelPackages.extend (self: super: {
    kernel = super.kernel.override {
      extraConfig = toConfig config.boot.kernel.extraConfig;
    };
   };
}

Something like that. nixos/modules/system/boot/kernel.nix also does some overriding so it might conflict.

6

is the goal to then re-inject the kernel into a NixOS setup?

Not really, I do a bit of kernel development which I test in qemu via -kernel so I don’t need to reinject the kernel into nixos (which in factwould be easy as your example show it). I would like to specify several kernels with different configs while boot.kernel.extraConfig is unique.
(Regarding your example, you can now pass extraStructuredConfig too).

I believe that the lib/types.nix should be more independant from lib/modules.nix for me to use its typechecking mechanism without the modules part. Without types, that part of the code is hard to follow so I don’t intend to touch it.

I guess for now I will just experiment with boot.kernel.extraConfig and maybe create several ones like boot.kernel2.extraConfig / boot.kernel3.extraConfig and refer to these when defining my kernels.

7

You can relatively easily write your own module evaluating function without a dependency on NixOS, fully contained in the package definition:

  { stdenv, lib }:

  stdenv.mkDerivation {
    # ...

    passthru = rec {
      module = import ./module.nix;
      configFile = cfg: (lib.evalModules {
        modules = [
          module
          cfg
        ];
      }).config.file;
    };
  }

Have a module.nix that defines options like this:

  { lib, config }:

  with lib;

  {

    options = {
      file = mkOption {
        type = types.path;
      };
      fileContents = mkOption {
        type = types.lines;
      };
      params = mkOption {
        type = types.attrsOf (types.submodule {
          # ...
        });
      };
    };

    config = {
      file = builtins.toFile config.fileContents;
      fileContents = throw "Some function to convert config.params to a string, possibly with some assertions";
    };
  }

Then the user can directly get a config file from the package definition through pkgs.foo.configFile { params.foo = true; }. It’s then even possible to have this module as a submodule in NixOS by using options.some.nixos.option = mkOption { type = types.submodule pkgs.foo.module; };. This module declaration is in fact usable by anybody that has access to the package, so even different distros can benefit from it. I’d like to have more packages use something like this in the future, it’s really useful.

7 Likes
8

Thanks for the great example. I add to change the header to remove an error { lib, config, ... }:

I’d like to have more packages use something like this in the future, it’s really useful.

Are there any example in nixpkgs ?

I managed to adapt and run a variation of your code

    textStr = with lib.kernel; passthru.configFile {
      params.NET_9P_VIRTIO = yes;
    };

What I am looking at next is - how to conduct/what is needed - to merge different pieces { params.foo = true; } of configurations that could reside in different places. In the kernel case, parts of the config are in the kernel itself (pkgs/os-specific/linux/kernel/common-config.nix) and some additionnal parts are defined in kernelPatches extraConfig field.

EDIT: hum I guess I can just pass additionnal cfg entries in the modules list

modules = [ module cfg cfg2 cfg3 ... ];

EDIT2: it works indeed ! I now have to look at how easy it is to modify the merging function of my kernelItem submodule (defined in first post).

9

Just as a heads up I managed to do what I wanted so thanks for the help !! I will send a PR once I’ve polished enough my patch .

1 Like
10

for elegance I would liketo be able to write MMC_BLOCK_MINORS = "32"; and have it converted to a kernelItem submodule automatically.

So I run a mapAttrs on my config items and when I detect a freeform answer (ie, different than y/n/m), I generate { answer = "32"; optional = false; }

This doesn’t seem to be enough though:


while evaluating anonymous function at /home/teto/nixpkgs2/lib/types.nix:275:22, called from /home/teto/nixpkgs2/lib/attrsets.nix:200:54:
while evaluating the attribute 'value' at /home/teto/nixpkgs2/lib/modules.nix:353:27:
while evaluating anonymous function at /home/teto/nixpkgs2/lib/modules.nix:345:32, called from /home/teto/nixpkgs2/lib/modules.nix:345:19:
The option value `settings.MMC_BLOCK_MINORS' in `<unknown-file>' is not of type `submodule'.

how can I get my attrset converted to a kernelItem submodule ? I’ve seen the packSubmodule but haven’t been too successful with it.

11

@Infinisil I pushed an update of my PR on [RFC] add ability to merge structured configs by teto · Pull Request #42838 · NixOS/nixpkgs · GitHub . I wonder if there is any way to improve the code ?
I don’t like that error messages contain the <unknown-file> and also that I have to encapsulate my configs in { settings = { ..} }