After the RFC to simplify the package structure, the Nixpkgs Architecture Team is now assembling a working group (WG) to investigate the potential of using a module system for packages in Nixpkgs (see below sections for more details). With this post we’re searching for members of this WG.
The WG members will meet regularly to synchronize, while working asynchronously between meetings. As such, WG members are expected to dedicate some time towards this, at least some hours every week.
We have some people already interested in joining:
- @infinisil from the Nixpkgs Architecture Team
- @growpotkin from the Nixpkgs Architecture Team
- @DavHau, the author of drv-parts
But we’d like more help from the wider community to reach a working group size of 4-6 people. If you’re interested in helping out, feel free to contact us in this thread or on Matrix.
The general process for the WG is:
- The WG works on an RFC draft in a separate repository
- The Nixpkgs Architecture Team reviews the RFC draft until consensus is reached that it looks good for the next step, or it’s not worth pursuing further
- The WG opens the RFC on the rfcs repository for the wider community to give feedback. Nixpkgs Architecture Team members will nominate themselves as shepherds for the RFC (with full disclosure, it’s up to the steering committee to pick the shepherds)
- The WG addresses the feedback from the community until the RFC is either accepted or rejected
- The WG implements the RFC
The following sections go into details about the problem to solve and the potential solution.
Problem
- If you want to override something in Nixpkgs, it’s hard to know which override mechanism you need to use:
- There’s too many specific mechanisms, here’s an incomplete listing:
- Global:
overlays,config,pkgs.extend,pkgs.appendOverlays - General package builders:
.override,.overrideAttrs,overrideDerivation,extendDerivation - General package sets:
.overrideScope',.extend,makeScope - Builder-specific:
.overridePythonAttrs,pkgs.haskell.lib.overrideCabal - Package-set specific:
pythonPackagesExtensions
- Global:
- Different mechanisms can interfere with each other
- There’s too many specific mechanisms, here’s an incomplete listing:
- It’s hard to know how to write the override itself:
- Options to override are not discoverable, there’s no listing of available options, while the source code is hard to find and understand
- The available options are poorly documented if at all
- No type safety for overrides, e.g.
dontUnpack = "false"doesn’t do what you expect to and gives no error - No name checking, e.g.
doUnpack = true;doesn’t give an error, but it should bedontUnpack = false;instead
- Merging with previous values is done manually
- It’s the responsibility of the override author to merge correctly, e.g.
patches = (old.patches or []) ++ [ ... ] - Makes overrides more verbose and repetitive than they should have to be
- It’s the responsibility of the override author to merge correctly, e.g.
- Over-use of order-dependent lists, making ordering relevant for correctness
- Flipping the order of overlays can change the result, e.g. ordering of patches, buildInputs, phases, etc.
- Override issues are hard to debug. Existing interfaces lack useful error messages or traces.
- Inconsistent configuration styles between NixOS (the module system) and Nixpkgs (function arguments, and an ad hoc config attribute set).
- Package sets and packages are evaluated for each subsequent overlay.
- Needs to be compared to module system performance though
Potential Solution
Something like the module system used by NixOS might be able to solve the above problems:
- Module options are typed, automatically mergeable, discoverable and documentable
- Modules may be powerful enough to serve as a general override mechanism
- Modules have more composability and structure, they encourage less ordering and state
- Modules allow merging overrides together before processing them
More specifically, these tasks may be involved among others:
- Evaluate whether the module system can support all override mechanism use cases
- Implement an API to extend Nixpkgs with modules
- Benchmark the module system and improve its speed if necessary
- Ensure reasonable backwards-compatibility and establish a migration plan
- Set up rendering of override options on https://nixos.org/
- Update reference documentation
Previous Work
- @DavHau’s drv-parts, which implements part of this idea as a third-party library
- @roberth’s minimod, a stripped-down version of the module system
- @growpotkin’s floco has some modules for package specifications:
- @edolstra’s ideas for Nix modules
- @nbp’s S.O.S. proposal: [RFC 0003] SOS: Simple Override Strategy. by nbp · Pull Request #3 · NixOS/rfcs · GitHub
- @fare’s POP
- Tweag’s Nickel language comes with overridable attribute sets by default
Example
Here’s an example from @DavHaus’s drv-parts, showing how traditional overriding mechanisms compare to a module-system-based approach:
Changing options of packages in Nixpkgs can require chaining different override functions like this:
{
htop-mod = let
htop-overridden = pkgs.htop.overrideAttrs (old: {
pname = "htop-mod";
});
in
htop-overridden.override (old: {
sensorsSupport = false;
});
}
… while doing the same using drv-parts looks like this:
{
htop-mod = {
imports = [./htop.nix];
public.name = lib.mkForce "htop-mod";
flags.sensorsSupport = false;
};
}
For variety, and to highlight that the interface isn’t decided yet, here’s another idea:
{
config.packages.htop = {
interface = "mkDerivation";
pname = "htop";
version = "4.2.0";
src = {...};
# ...
};
config.packages.htop-mod = {
interface = "override";
base = "htop";
overrides.pname = "htop-mod";
overrides.sensorsSupport = false;
};
}
