I think the nix module system needs a type called “endo”, basically it’s just a mergable a -> a. Endo in Haskell is a monoid, so it is definitely mergable.
endo a would suffer from the same flaw that listOf a does, which is that it is sensitive to the order in which definitions are processed. This can lead to surprises when moving value definitions between modules.
Usually, with lists, these effects are relatively harmless, because the result has more of a set-like behavior anyway. Usually.
With endo, I would expect more problems, because it describes changes, and the order in which changes are made often matters. We have already observed this with overlays, which are functionTo (endo a), or in Haskell types, a -> Endo a.
The nasty surprise can be prevented by requiring explicit ordering, such as with mkBefore, mkAfter, mkOrder. This prevents the problem, but also causes valid, commutative definitions to be rejected when they’re specified in separate modules. (“hooray”, we can reason locally about options, a bit)
So a refactor-proof version of this type is not all that useful.
That doesn’t mean that we shouldn’t have it. In fact by having them, we have a good place to document why they are the way they are, and that they’re a last resort.
I think we don’t need the mkBefore , mkAfter , mkOrder helpers for the endo type. Just topologically sort the modules.
The problem to determine the order of prev / super calls is similar to linearization in Scala. I think the common practices in object-oriented languages is to inference the order from the hierarchy, like a topological sort, instead of manually specifying the order. Some other languages, like Python, enforce a more restrictive C3 linearization, which is basically useless, resulting in the infamous Cannot create a consistent method resolution error.
Overlays can result in infinite loops easily because of the fixed point of the final / self reference. On the other hand, in module system, the fixed point is the config reference. Therefore, merging endo values would not result in infinite loops unless they use config in a strict way.
In object-oriented languages or overlays, when super/prev is used to call the current function’s original implementation, it’s like an endo. However endo cannot be used to make super/prev calls to functions other than the current function. Fortunately it is a rare use case. For example:
myOverlay = final: prev: {
currentPackage =
prev.currentPackage # Good
.overrideAttrs (old: {
buildInputs = old.buildInputs ++ [
final.otherPackage # Good
prev.otherPackage # Bad. Can result in surprising behavior
];
});
};
Therefore, I think not supporting prev.otherPackage is actually a good thing.
In fact, current module system do have an apply function on options. The only problem is that in the current implementation apply functions are not mergeable even when options are mergeable. I think a simple patch to actually merge apply, would make it work, similar to how we merge submoduleWith and deferredModuleWith.
That’s not how Endo composes in Haskell. It composes like super, not self.
self doesn’t even really exist during overlay composition; it only exists when the overlay is used in some context (what I’ve called finalization in my previous comment).
Other than the definition of Endo, I think we agree on this.
Correct, but in the module system this process will be even more opaque than in those languages, which is why I think we should avoid this solution.
By virtue of being implemented in the Nix language, it basically¹ can’t be non-deterministic, but from a user perspective, it would still be unpredictable.
¹: modulo exceptions (which is ok) and lambda pointer equality (unfortunate, but rare)
I created a PR to make apply mergeable, so that it can be used as an Endo to visit the previous value. modules: make `apply` mergeable by Atry · Pull Request #301472 · NixOS/nixpkgs · GitHub
Even for lists, I think mkAfter and mkBefore are not good enough, because sometimes we need to update existing elements in the list. I think we can introduce a special _module.mergeKey config for submodules in a list, then submodules whose _module.mergeKey are the same will be merged into a single submodule.
For example, suppose we want to create a module to generate kubernetes Pod YAML, where the spec.containers is a list of attrsets whose name must be unique.
options.spec.containers = lib.mkOption {
type = lib.types.listOf (lib.types.submoduleWith {
modules = [
({config, ...}: {
config._module.mergeKey = config.name
options.name = ...;
options.image = ...;
options.ports = ...;
})
];
});
};
config.spec.containers = lib.mkMerge [
[
{
name = "nginx";
image = "nginx:1.14.2";
}
{
name = "another-container-1";
}
]
[
{
name = "nginx";
ports = [ { containerPort = 80; } ];
}
{
name = "another-container-2";
}
]
];
Then spec.containers will be evaluated as:
[
{
name = "nginx";
image = "nginx:1.14.2";
ports = [ { containerPort = 80; } ];
}
{
name = "another-container-1";
}
{
name = "another-container-2";
}
]
The nginx submodules are merged because they have the same _module.mergeKey.
1 Like