Best resources for learning about the NixOS module system?

Help
4

It’s implementation is basically 2 files: https://github.com/NixOS/nixpkgs/blob/master/lib/modules.nix and https://github.com/NixOS/nixpkgs/blob/master/lib/types.nix

The first one is actual module system DSL implementation, the second is a type checker library. Typechecking is optional, but it enables sophisticated and efficient submodules.

Module system was first created by Nicolas B. Pierron (@nbp in Github). Here’s introduction email: [Nix-dev] NixOS: New scheme

If you read through it, you’ll find out that Ludovic didn’t like the complicated merging algorithm (he proposed using functions to compose system configuration),

and maybe module system adoption was one of reasons to fork NixOS as GuixSD. I don’t have much historical context here though…

So, why was module system created? Previously NixOS configuration was more like JSON nested dictionary. During NixOS build code tried to extract various options from
that JSON dictionary (it was in Nix syntax actually) and convert those to actual Bash files. The problem was that extensibility wasn’t easy. @nbp proposed

to write a generic merge algorithm, which merged recursive dictionaries (see also poor-man merge algorithm https://github.com/NixOS/nixpkgs/blob/c7104d97c87316a6062c88696d80a45918fab7a6/lib/attrsets.nix#L396-L418 ).

The great insight here was to do automerge of multiline strings and lists, and setup merge priority system. It actually showed, how much useful is “merge by default”

Another great feature, which defined the way modules are written now, was the clever use of “knot-tiying”, which allowed using IF, conditions.

Here is simple module example:

{ config }:
{
    config.foo.bar = if config.baz then "Hey" else "Hai";
}

So, this module adds an attribute config.foo.bar, but itself it uses config argument to query baz attribute. But both "config"s are same! It looks like recursion (and it is), but

it works because Nix attrsets are lazy in attrset values. So “config.baz” evaluates “config”, but doesn’t evalutate “config.foo” and infinite recursion is not triggered.

(However you can’t use “if” in all cases. For those cases where Nix language wasn’t lazy enough, “mkIf” function was added. See next example:

{ config }:
{
    config = if config.foo.meh then { foo.bar = "Hey"; } else { foo.baz = "Hai"; };
}

This is real infinite recursion. But if you rewrite it like:


{ config }:
{
    config.foo.bar = delayedIf (config.foo.meh) "Hey";
    config.foo.baz = delayedIf (not config.foo.meh) "Hai";
}

then evaluating config.foo won’t cause infinite recursion, and so config.foo.meh won’t require values of config.foo.{bar,baz}

So this all is abstracted behind “mkIf”. See also Wiki https://nixos.wiki/wiki/NixOS:config_argumenthttps://nixos.wiki/wiki/NixOS:config_argument )

The design had problems though - the namespace for options (recursive attributes) was open, so each module could access each other module options.

This was exactly what Ludovic didn’t like - some module can read/adjust your module without your consent. Another consequence is that you don’t trust

modules from Internet - backdors are easy to implement, you just have to ask smbd to use your module on production system.

So, security audit for system built of modules is hard to impossible. But it is so convenient, and <nixpkgs/nixos> is trusted to not contain backdors.

Another problem, which occurred recently, was performance. Though merge algorithm is efficient and linear in module count, it still is slow when

many modules used, especially for multiple evaluation (NixOps case and declarative containers case, also there is lesser known “nesting.children” and “nesting.clone” options).

So it may be that in future revision of module system the “imports” mechanism will be lost. The nice thing about “imports” is that module order doesn’t matter

Yup. You can try reorder imported modules and it won’t matter. You can even duplicate and it won’t matter. The core thing for this to work is internal module attributes:

key and _file (https://github.com/NixOS/nixpkgs/blob/c7104d97c87316a6062c88696d80a45918fab7a6/lib/modules.nix#L128-L129). You may follow logic through the code,

and you can define your own _file and key for ad-hoc modules.

I don’t dive into typechecking, it should be pretty easy to deduce algorithm from source code. In short: module system types are first class values in Nix,

so you can create new ones, and type consists of a) merge function b) validation function. That’s all folks! (strictly speaking, submodules are much more complicated).

Does that sound OK? Is it enough “internals” for basic insights? The next things is my POV on module system as expert system database.

I’ve described it previously in <nixpkgs/nixos> is an expert system database . It wasn’t designed as an expert system

originally (as I’ve asked @nbp), but was a coincident.

So in theory, you can use module system as an alternative to CLIPS. Though I didn’t do a performance and feature comparison of those two.

30 Likes