Hi all,
I am interested in having a deeper understanding of the NixOS module system, which I have heard is a powerful tool with potentially many more uses than it currently has. Can people point me to learning resources about it? In particular, I’m interested in where it’s implemented, where it’s used, and what theoretical uses it might have. Any help is appreciated! I’ll summarize my findings once I’m done looking at it.
Hi,
Rickard Nillson is giving a talk about the NixOS module system next Friday at NixCon. It’s going to be recorded and streamed live (if all goes well). Hopefully it will be interesting to you.
You might find this thread useful, along with the reuslting implementation linked in the PR
Bottom line is: it’s possible to use the module system outside of nixos, to configure specific aspects of a package. It could for instance replace the very poor nixpkgs vimrc configure mechanism.
It’s implementation is basically 2 files: https://github.com/NixOS/nixpkgs/blob/master/lib/modules.nix and https://github.com/NixOS/nixpkgs/blob/master/lib/types.nix
The first one is actual module system DSL implementation, the second is a type checker library. Typechecking is optional, but it enables sophisticated and efficient submodules.
Module system was first created by Nicolas B. Pierron (@nbp in Github). Here’s introduction email: [Nix-dev] NixOS: New scheme
If you read through it, you’ll find out that Ludovic didn’t like the complicated merging algorithm (he proposed using functions to compose system configuration),
and maybe module system adoption was one of reasons to fork NixOS as GuixSD. I don’t have much historical context here though…
So, why was module system created? Previously NixOS configuration was more like JSON nested dictionary. During NixOS build code tried to extract various options from
that JSON dictionary (it was in Nix syntax actually) and convert those to actual Bash files. The problem was that extensibility wasn’t easy. @nbp proposed
to write a generic merge algorithm, which merged recursive dictionaries (see also poor-man merge algorithm https://github.com/NixOS/nixpkgs/blob/c7104d97c87316a6062c88696d80a45918fab7a6/lib/attrsets.nix#L396-L418 ).
The great insight here was to do automerge of multiline strings and lists, and setup merge priority system. It actually showed, how much useful is “merge by default”
Another great feature, which defined the way modules are written now, was the clever use of “knot-tiying”, which allowed using IF, conditions.
Here is simple module example:
{ config }:
{
config.foo.bar = if config.baz then "Hey" else "Hai";
}
So, this module adds an attribute config.foo.bar, but itself it uses config argument to query baz attribute. But both "config"s are same! It looks like recursion (and it is), but
it works because Nix attrsets are lazy in attrset values. So “config.baz” evaluates “config”, but doesn’t evalutate “config.foo” and infinite recursion is not triggered.
(However you can’t use “if” in all cases. For those cases where Nix language wasn’t lazy enough, “mkIf” function was added. See next example:
{ config }:
{
config = if config.foo.meh then { foo.bar = "Hey"; } else { foo.baz = "Hai"; };
}
This is real infinite recursion. But if you rewrite it like:
{ config }:
{
config.foo.bar = delayedIf (config.foo.meh) "Hey";
config.foo.baz = delayedIf (not config.foo.meh) "Hai";
}
then evaluating config.foo won’t cause infinite recursion, and so config.foo.meh won’t require values of config.foo.{bar,baz}
So this all is abstracted behind “mkIf”. See also Wiki https://nixos.wiki/wiki/NixOS:config_argumenthttps://nixos.wiki/wiki/NixOS:config_argument )
The design had problems though - the namespace for options (recursive attributes) was open, so each module could access each other module options.
This was exactly what Ludovic didn’t like - some module can read/adjust your module without your consent. Another consequence is that you don’t trust
modules from Internet - backdors are easy to implement, you just have to ask smbd to use your module on production system.
So, security audit for system built of modules is hard to impossible. But it is so convenient, and <nixpkgs/nixos> is trusted to not contain backdors.
Another problem, which occurred recently, was performance. Though merge algorithm is efficient and linear in module count, it still is slow when
many modules used, especially for multiple evaluation (NixOps case and declarative containers case, also there is lesser known “nesting.children” and “nesting.clone” options).
So it may be that in future revision of module system the “imports” mechanism will be lost. The nice thing about “imports” is that module order doesn’t matter
Yup. You can try reorder imported modules and it won’t matter. You can even duplicate and it won’t matter. The core thing for this to work is internal module attributes:
key and _file (https://github.com/NixOS/nixpkgs/blob/c7104d97c87316a6062c88696d80a45918fab7a6/lib/modules.nix#L128-L129). You may follow logic through the code,
and you can define your own _file and key for ad-hoc modules.
I don’t dive into typechecking, it should be pretty easy to deduce algorithm from source code. In short: module system types are first class values in Nix,
so you can create new ones, and type consists of a) merge function b) validation function. That’s all folks! (strictly speaking, submodules are much more complicated).
Does that sound OK? Is it enough “internals” for basic insights? The next things is my POV on module system as expert system database.
I’ve described it previously in <nixpkgs/nixos> is an expert system database . It wasn’t designed as an expert system
originally (as I’ve asked @nbp), but was a coincident.
So in theory, you can use module system as an alternative to CLIPS. Though I didn’t do a performance and feature comparison of those two.
Wow, thanks for all your technical and historical insights! Now I’m sure I’ll be well prepared to appreciate Rickard Nilsson’s talk at NixCon, which I’ll see in person. 
See you all there!