Working group member search: Module system for packages

After the RFC to simplify the package structure, the Nixpkgs Architecture Team is now assembling a working group (WG) to investigate the potential of using a module system for packages in Nixpkgs (see below sections for more details). With this post we’re searching for members of this WG.

The WG members will meet regularly to synchronize, while working asynchronously between meetings. As such, WG members are expected to dedicate some time towards this, at least some hours every week.

We have some people already interested in joining:

But we’d like more help from the wider community to reach a working group size of 4-6 people. If you’re interested in helping out, feel free to contact us in this thread or on Matrix.

The general process for the WG is:

  1. The WG works on an RFC draft in a separate repository
  2. The Nixpkgs Architecture Team reviews the RFC draft until consensus is reached that it looks good for the next step, or it’s not worth pursuing further
  3. The WG opens the RFC on the rfcs repository for the wider community to give feedback. Nixpkgs Architecture Team members will nominate themselves as shepherds for the RFC (with full disclosure, it’s up to the steering committee to pick the shepherds)
  4. The WG addresses the feedback from the community until the RFC is either accepted or rejected
  5. The WG implements the RFC

The following sections go into details about the problem to solve and the potential solution.

Problem

  • If you want to override something in Nixpkgs, it’s hard to know which override mechanism you need to use:
    • There’s too many specific mechanisms, here’s an incomplete listing:
      • Global: overlays, config, pkgs.extend, pkgs.appendOverlays
      • General package builders: .override, .overrideAttrs, overrideDerivation, extendDerivation
      • General package sets: .overrideScope', .extend, makeScope
      • Builder-specific: .overridePythonAttrs, pkgs.haskell.lib.overrideCabal
      • Package-set specific: pythonPackagesExtensions
    • Different mechanisms can interfere with each other
  • It’s hard to know how to write the override itself:
    • Options to override are not discoverable, there’s no listing of available options, while the source code is hard to find and understand
    • The available options are poorly documented if at all
    • No type safety for overrides, e.g. dontUnpack = "false" doesn’t do what you expect to and gives no error
    • No name checking, e.g. doUnpack = true; doesn’t give an error, but it should be dontUnpack = false; instead
  • Merging with previous values is done manually
    • It’s the responsibility of the override author to merge correctly, e.g. patches = (old.patches or []) ++ [ ... ]
    • Makes overrides more verbose and repetitive than they should have to be
  • Over-use of order-dependent lists, making ordering relevant for correctness
    • Flipping the order of overlays can change the result, e.g. ordering of patches, buildInputs, phases, etc.
  • Override issues are hard to debug. Existing interfaces lack useful error messages or traces.
  • Inconsistent configuration styles between NixOS (the module system) and Nixpkgs (function arguments, and an ad hoc config attribute set).
  • Package sets and packages are evaluated for each subsequent overlay.
    • Needs to be compared to module system performance though

Potential Solution

Something like the module system used by NixOS might be able to solve the above problems:

  • Module options are typed, automatically mergeable, discoverable and documentable
  • Modules may be powerful enough to serve as a general override mechanism
  • Modules have more composability and structure, they encourage less ordering and state
  • Modules allow merging overrides together before processing them

More specifically, these tasks may be involved among others:

  • Evaluate whether the module system can support all override mechanism use cases
  • Implement an API to extend Nixpkgs with modules
  • Benchmark the module system and improve its speed if necessary
  • Ensure reasonable backwards-compatibility and establish a migration plan
  • Set up rendering of override options on https://nixos.org/
  • Update reference documentation

Previous Work

Example

Here’s an example from @DavHaus’s drv-parts, showing how traditional overriding mechanisms compare to a module-system-based approach:

Changing options of packages in Nixpkgs can require chaining different override functions like this:

{
  htop-mod = let
    htop-overridden = pkgs.htop.overrideAttrs (old: {
      pname = "htop-mod";
    });
  in
    htop-overridden.override (old: {
      sensorsSupport = false;
    });
}

… while doing the same using drv-parts looks like this:

{
  htop-mod = {
    imports = [./htop.nix];
    public.name = lib.mkForce "htop-mod";
    flags.sensorsSupport = false;
  };
}

For variety, and to highlight that the interface isn’t decided yet, here’s another idea:

{
  config.packages.htop = {
    interface = "mkDerivation";
    pname     = "htop";
    version   = "4.2.0";
    src       = {...};
    # ...
  };
  config.packages.htop-mod = {
    interface = "override";
    base      = "htop";
    overrides.pname          = "htop-mod";
    overrides.sensorsSupport = false;
  };
}
32 Likes

I’m interested in joining this WG.

16 Likes

Flipping the order of overlays can change the result, e.g. ordering of patches, buildInputs, phases, etc.

I am not actually sure this is a pure negative. Overrides have a semantics closer to sequential data crunching, which is easier to analyse than global-resolution-of-whatever in the module system.

Inconsistent configuration styles between NixOS (the module system) and Nixpkgs (function arguments, and an ad hoc config attribute set).

To be completely honest, I would of course prefer unification closer to the makeExtensible than to global module system — well, why «would», after giving up on mainline NixOS I construct the system expression exactly via makeExtensible. I don’t care that much about NixOS side, but moving Nixpkgs towards module system is disappointing.

Package sets and packages are evaluated for each subsequent overlay.

Actually a good question how much is really evaluated given laziness, and how hard is it to reduce the amount (i.e. would storing what-to-reimport instead of the imported function be a benefit anywhere?)…

Also, how all this interacts with caching?

3 Likes

I’ve used both really extensively trying to create a big node package set framework.

I started with makeExtensible and simple type assertions and a bit of YANTS ( lightweight type system ) which honestly worked pretty well, but over time it became incredibly hard to maintain.
Both in terms of composing overlays and aggregating packages from multiple projects/package sets. Overlays and extensions are sensitive to ordering, but lack any kind of tracing mechanism or useful helper routines for complex merges which can become really frustrating at scale.

It took me a really long time to give the module system a fair shot, mostly because I saw it as overly complex or bloated; but when I started thinking of the system as a structured merge operator with nice debugging features rather than a system configuration tool, it was an enormous relief to use.

There’s certainly a lot of improvements that could be made to the module system, and I would advise restraint when writing the options ( keep them simple, make something useful not “interesting” ), but I think it’s a tool worth exploring for wrangling the package set.

As for caching the evaluator could care less if the contents of your .nix files or flake outputs were creating using modules or overlays/extensions ( with the exception of “trivial files” ). I will say that I found it a bit easier to avoid re-evaluation of many overlays that referenced final but that is more about how you organize your expressions than anything.

In any case, as someone who was personally really resistant to use the module system to build packages because I perceived it as overly-complex - I really do recommend giving them an honest investigation. After using them for a week I got used to the quirks and was able to be much more productive.

6 Likes

On Matrix @snowytrees also showed interest in joining

1 Like

Yep! Been meaning to post here. But I would like to join the WG.

3 Likes

I would like to throw POP in the mix as a possibility to consider when redesigning the package set. It is a proper object system with inheritance that could be ideal for building packages. I thinks it’s pretty simple to understand how to use if you’ve used any class/object system in another language like python. Its almost exaxtly like jsonnets object system where methods are functional in that they don’t modify an object instance in place but rather return a modified instance.

I think it is simpler in principle than the module system and quite extensible. I have a prototype example of building something similar to flake-parts with it here, GitHub - divnix/flops: Flake Pops and using yants to handle type checking through the interface.

cc @fare (author of POP)

4 Likes

@Pacman99 Good point, added to the previous work section!

1 Like

Update: I created a Matrix room (open for anybody to join) for this working group and scheduled a regular weekly meeting with everybody interested starting at Friday, 2023-04-14 15:00 CEST (date -d@1681477200) in this Jitsi Meet (also open for anybody to join). There is also a calendar event in the NixOS Google Calendar.

The initial working group will consist of

But if anybody else is interested in helping out, you’re more than welcome! Either join the meeting, send a message in the Matrix room or here on Discourse, or PM me :slight_smile:

3 Likes

The first meeting will be delayed by 2 weeks because some participants can’t make it this and next Friday. The new date is Friday, 2023-04-28 15:00 CEST (date -d@1682686800).

1 Like

In order to create packages in module system, we will need a way to visit the prev arguments like what we did in override / overrideAttrs. Currently we cannot patch a config value if the type is lib.types.package. I think introducing some properties like mkPackageOverride and mkPackageOverrideAttrs would solve this problem.

For example, suppose you want to create a special mongodb built from clang, and you also need its dependency curl to be built from clang, too, you can do:

options.my-mongodb = lib.mkOption {
  type = lib.types.package;
};
config.my-mongodb = pkgs.mongodb.override (prev: {
  stdenv = llvmPackages.stdenv;
  curl = prev.curl.override {
    stdenv = llvmPackages.stdenv;
  };
});

However, currently it is not possible to patch an existing package in module system.

With the help of mkPackageOverride, it would be possible to create things like this:

# In the base module
config.my-mongodb = pkgs.mongodb;
# In another module
config.my-mongodb = lib.mkPackageOverride (prev: {
  stdenv = llvmPackages.stdenv;
  curl = prev.curl.override {
    stdenv = llvmPackages.stdenv;
  };
});

mkPackageOverride is conceptually similar to mkAfter, incrementally changing the existing config.

Note that drv-parts cannot solve this problem because of the lack of the ability to visit prev.

Related idea: Adding more control over overriding default module definitions

A quick update:

5 Likes

There will be a small presentation on drv-parts by
@DavHau in todays pkg-modules meeting at 2023-06-16T13:00:00Z
The meeting is open to everyone at Jitsi Meet, and

@Infinisil scheduled a YoutTube stream at https://youtube.com/live/FmaFOlKRUKw

5 Likes

I’m interested, but unfortunately I can’t join at that time, will be the recording available to watch it later?

3 Likes

I believe the YouTube stream recording should be public, yes.

4 Likes

Thank you very much for the presentation @DavHau. I was able to join the live stream for the first half and then watched the remainder over the YouTube video. Very interesting work. I look forward to hearing more about this.

2 Likes

There’s been a technical hiccup, so we had to re-record the first part of the presentation, but both recordings are now merged and available at https://www.youtube.com/watch?v=AsCvRZukX0E

4 Likes

There will be a small presentation by @Growpotkin’s on floco next week, 2023-07-21T13:00:00Z (tracking issue)

The presentation will be held in this Jitsi meet, anybody is free to listen in and ask questions

It will also be live-streamed and recorded to YouTube: https://www.youtube.com/watch?v=rLtcCXJZuAU

5 Likes

After a couple of months we’re concluding that this working group didn’t work out entirely how we were hoping it would. You can read more about that here.

Instead this working group will be merged with the Nixpkgs Architecture Team (NAT), which should serve mostly as a platform to talk/discuss/review ideas regarding Nixpkgs Architecture. There will be a NAT meeting tomorrow where it will be discussed further.

4 Likes