Drv-parts - configure packages like NixOS systems

Announcements
#1

drv-parts

See github readme for more info: GitHub - DavHau/drv-parts: Configure packages like NixOS sytems

drv-parts replaces callPackage, override, overrideAttrs, ... as a mechanism for configuring packages.
It makes package configuration feel like nixos system configuration.

This project is suitable as a plugin for flake-parts, which is an amazing project that I can recommend to any flakes user.
drv-parts can also be used without flakes or flake-parts.

drv-parts has been kickstarted during the oceansprint.
Thanks to @zupo and @domenkozar for organizing this amazing event, where I could experiment on projects like this, sitting next to some of the best experts of the field.

Thanks especially to @roberth for answering important questions.
Thanks also to @Flakebi for taking part in developing this idea.

Please contribute if you are interested in moving this project forward.

drv-parts was funded as part of the dream2nix project through the NGI Assure Fund, a fund established by NLnet with financial support from the European Commission’s Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 957073. Applications are still open, you can apply today.

30 Likes
Noogle – like Hoogle but for Nix
#2

Already the concept of “env variables separated” alone gets an upvote from me!

1 Like
#3

Structured args when?

#4

Now the killer question:

If I combine this with CA could we archive full molecularity of all package options with sane rebuilds?

I imagine something like packages declare their features and then this does the magic and compiles one library that fits the system instead of having plenty of variants?

1 Like
#5

I wonder if the ecosystem has shot itself in the foot with the builtin __structuredAttrs. I assume the motivation behind supporting bash specifically was to make integration into nixpkgs easy. At the same time __structuredAttrs is implemented in such a backward breaking fashion that nixpkgs is still struggling to add support for it.

Maybe it would be better to just use toJSON and passAsFile instead, and have a script at build time that converts json to bash variables.

The whole thing could be done in a backward compatible manner. Just keep all env variables as they are (for now) and add the json file ontop. Then gradually migrate build logic to make use of the json/file.

It seems to me, by trying to support nixpkgs specifically, nix has layed out a path that makes it hard for nixpkgs to upgrade.

I haven’t thought about this much yet. Maybe you could provide some specific example to help me understand that problem domain better.

Right now it seems to me, that having one global configuration for each package can also have adverse effects, because changing requirements in one place can lead to mass rebuilds everywhere else. Does that change fundamentally with CA? As far as I understand, even with CA, a changing runtime dependency hash, still propagates transitively to all dependants.

Currently drv-parts evaluates your derivations as submodules. That means, by default, derivations are evaluated in isolation and cannot access each others options. I thought that this is a good thing as it makes the whole evaluation more scalable (lazy) and minimizes side effects between derivations.

A global package configuration feature could still be implemented as an additional module, that controls the options of all the submodules.

#6

no more override functions
Changing options of packages in nixpkgs can require chaining different override functions like this

How would you write an overrideAttrs that modifies one of the derivation’s fields rather than merging it with something?

Here’s a simple example found in nixpkgs:

.overrideAttrs (a: { pname = "${a.pname}-stage3"; }

A more advanced example is in pkgs/stdenv/adapters.nix, where the attributes added by the override depend on un-overridden attributes.

Would we still be able to do things like these if there were no more override functions?

1 Like
#7

I see the following options:

  • Define a new derivation, that defines some of it’s fields depending on the original one.
    {
  hello-overridden.pname = "${hello.pname}-stage3"
}
  • copy paste the previous value and extend it
  • redesign your original module to make the field properly configurable, for example by adding an option pname-suffix, and re-define pname to contain the suffix.
  • The nixos module system also provides a function called extendModules. I’m not sure if this could be utilized here. I have not experimented with it so far.

Your example highlights one of the fundamental differences between overrides and modules.
In the module system, there isn’t really a before and after to base your modification on.
This can be seen as a shortcoming, but AFAIU it leads to better scalability in terms of UX at the same time.

The interface could be redesigned to make use of the module system’s merging capabilities. For example NIX_CFLAGS_LINK could be defined as a list of strings, which would allow to append to it. Or if the order of arguments really matters, defined the option to be an attribute set of arguments:

{
  0010 = "--first-arg";
  0020 = "--second-arg";
  # ...
}

Alternatively if the override style configuration is really required, you could still implement it if you want:

{config, stdenv, ...}: {
  options.stdenvAdapters = mkOption {
    type = attrsOf functionTo attrs;
  };
  config.finalStdenv = applyAdatpers stdenv config.stdenvAdapters;
}
#8

Thanks for your reply!

To summarize, it looks like all the options except this one are some variation on “rearchitect the package that you are overriding”. In my opinion one of the great strengths of the override[Attrs] mechanism is that it is non-invasive.

Yes, I see this as a fundamental limitation of the module-based approach compared to the current approach.

#9

It could. A module may expose this function by declaring an option that exposes it, but doing so reintroduces the overrideAttrs weakness that it may require evaluating the package attrset without using it.

I like the idea that overriding is fundamentally a non-abstraction, and that in order to do it you’d need to have access to the implementation - the original modules. Providing this access is not necessarily the responsibility of the package attrset, which is the public interface of a package.

There’s a trade-off between “invasive” and coupled. However, I don’t think invasive is the right term. If all our overriding was based on well-defined interfaces rather than implementation details, we’d have fewer errors when updating, and package functionality will be properly discoverable.

It is not. One possible implementation of .overrideAttrs could give access to the previous config, giving access to before. To emulate the current after behavior, you could partially automate mkOverride to prioritize step by step, using the old options metadata, but I don’t think we’d really need this. mkForce should be sufficient. Also we shouldn’t want this because overriding is not declarative in spirit, as it describes a transformation rather than a single state.

1 Like
#10

For example rsyslog has way to many options. If I turn Krb5 in one place and systemd in another then I would like to combine that and build one rsyslog that has both options set and use that in both places.

#11

I assume, that one of your goals is to build a minimal rsyslog. Otherwise the problem could be solved by just enabling all options of rsyslog by default.

I guess it should not be a big issue to create a system like that. Every package could define which features it requires enabled for its dependencies and this can then be merged by some global package configuration module. As long as we don’t need a conflict resolution mechanism, it sounds quite doable.

1 Like
#12

Rust’s “feature” mechanism works like this – it is “convex” like modules in the sense that a crate can “turn on” feature flags in its dependencies but cannot turn them off. This is why Rust features don’t create diamond dependency problems. This works because Rust features (unlike C/C++ #defines) are always booleans.

I think the nixos module-merge would be a really good fit for managing the enableFoo/fooSupport boolean flags that many packages have.

2 Likes
#13

Please keep me up to date when something evolves.

#14

This would be “spooky action at a distance”, so it needs an overarching configuration to carry the information. NixOS could fulfill that role, but the problem is more general than that, so the solution should be general too.
We’d have to integrate the NixOS and Nixpkgs fixpoints, and use (hundreds, or thousands of) use flags instead of environment.systemPackages.
Those flags would then conditionally set environment.systemPackages, as appropriate.

This goes back to earlier discussions about “use flags” which weren’t fruitful, because we did not expect to be able to carry the extra load of having to build and troubleshoot a much larger configuration space.

#15

Hello,

drv-parts looks usable, thank you.

Is there any some prominent repository uses drv-parts?

In Composable Finance we have monorepo with everything possible to imagine based on flake parts.

Is it known that some, may be Arion, Nixops, whatever, will not easy compose with drv-parts ?

#16

Currently I don’t know of such project. My plan is to use this in dream2nix at some point. In that case I will let you know here in this thread.

1 Like
#17

I really think this, or something like it, is the way forward for a more user friendly package system. I’ve been working on something similar to your project, @DavHau. it’s similar that it’s a package system built around modules, but there’s a few differences I’m excited about that evolve on the idea. I’d love to share with you when I get something more presentable and see what you think :slight_smile:

1 Like
#18

I think much of the current dream2nix can be simplified and improved. I plan to re-design its API based on drv-parts.

This is relevant for drv-parts because it makes dream2nix become the main driver behind it until further adoption occurs.

I sketched out usage examples here: v1 API - dream2nix documentation

I’m very much interested in your feedback. You can use this PR for commenting: feat(v1 API): document v1 API examples by DavHau · Pull Request #468 · nix-community/dream2nix · GitHub

Most notable changes that dream2nix’ new API will bring:

  • inspect options of any package via dream2nix man
  • override packages safely and conveniently using modules
  • no dependency on nix’ flakes feature
  • better composition of outputs
  • vastly simplify integration of existing lang2nix tools into dream2nix (see examples in docs)
1 Like
Hosted by Flying Circus.