Nix data-merge mini DSL (with array merge semantics)

Announcements
1

When using nix as a configuration language for your dev/staging/production environments, the current standard operating procedure that I know of has been:

  1. Implement the config interface in nixpkgs module system
  2. Configure your thing through that module system

Examples of this approach are kubenix or terranix.

However, this has it’s shortcomings:

  • Need to manually implement the config as nixpkgs module system, since to my knowledge there are no (generalized) 2nixpkgsModuleSystem code generators, that can for example consume jsonschema or openapi specs.

  • When you want to have a shared base configuration alongside dev, staging & prod, at some point you might end up using lib.mkOverride in various levels of priority to slap another layer on top of something. As a result, the provenance of the respective base values might become obscured, by virtue of the lazy merging and the fact that any value can be legally overwritten anywhere.

  • There is no real good solution / ergonomics for array merge strategies.

divnix/data-merge proposes a monotonistic, git-like alternative by introducing a special variant of lib.recursiveUpdate.

  • A simple attributeset that represents a configuration data structure on the left hand side (LHS) can be recursively merged with a RHS
  • A RHS leaf not matching the type of such existing LHS type (if it exists) is not allowed. Arrays & Maps are not leaves and part of the data spine.
  • To command the merging of array type data spines, two simple operations append [] and update [ idx ] [ value ] are provided and can be used on the RHS.
  • It is not allowed to do destructive operations on the “data spine”. For example you can’t delete a key from an attribute set nor can you override an array or drop one of its items (“monotonicity”). This ensures that in your layering of configuration, you can mentally stick with the “+” operator. This makes it much easier to reason about the configuration and also prevents potential spaghetti where several “+” & “-” operation end up cancelling out each other inadvertedly.

This model has its similarities with git, where the RHS applies a changest to the LHS, with the added restriction of no destructive operation on the “data spine”.

Hence, it is mentally very simple to reason about config management.

The full potential of this mini DSL would come to bear only if a builtins.jsonSchemaValidate attrs ./path/to/schema.json is implemented, since having an eval-time pre-flight check on the validity of your config data — although very incomplete without taking into account remote state — still is helpful on the inner-most devops loop.

I hope we can gather some consensus & momentum for a builtins.jsonSchemaValidate on this issue: `builtins.jsonSchemaValidate attrset ./path/to/schema.json` · Issue #5392 · NixOS/nix · GitHub

2 Likes