drv-parts replaces callPackage, override, overrideAttrs, ... as a mechanism for configuring packages.
It makes package configuration feel like nixos system configuration.
This project is suitable as a plugin for flake-parts, which is an amazing project that I can recommend to any flakes user. drv-parts can also be used without flakes or flake-parts.
drv-parts has been kickstarted during the oceansprint.
Thanks to @zupo and @domenkozar for organizing this amazing event, where I could experiment on projects like this, sitting next to some of the best experts of the field.
Thanks especially to @roberth for answering important questions.
Thanks also to @Flakebi for taking part in developing this idea.
Please contribute if you are interested in moving this project forward.
drv-parts was funded as part of the dream2nix project through the NGI Assure Fund, a fund established by NLnet with financial support from the European Commission’s Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 957073. Applications are still open, you can apply today.
If I combine this with CA could we archive full molecularity of all package options with sane rebuilds?
I imagine something like packages declare their features and then this does the magic and compiles one library that fits the system instead of having plenty of variants?
I wonder if the ecosystem has shot itself in the foot with the builtin __structuredAttrs. I assume the motivation behind supporting bash specifically was to make integration into nixpkgs easy. At the same time __structuredAttrs is implemented in such a backward breaking fashion that nixpkgs is still struggling to add support for it.
Maybe it would be better to just use toJSON and passAsFile instead, and have a script at build time that converts json to bash variables.
The whole thing could be done in a backward compatible manner. Just keep all env variables as they are (for now) and add the json file ontop. Then gradually migrate build logic to make use of the json/file.
It seems to me, by trying to support nixpkgs specifically, nix has layed out a path that makes it hard for nixpkgs to upgrade.
I haven’t thought about this much yet. Maybe you could provide some specific example to help me understand that problem domain better.
Right now it seems to me, that having one global configuration for each package can also have adverse effects, because changing requirements in one place can lead to mass rebuilds everywhere else. Does that change fundamentally with CA? As far as I understand, even with CA, a changing runtime dependency hash, still propagates transitively to all dependants.
Currently drv-parts evaluates your derivations as submodules. That means, by default, derivations are evaluated in isolation and cannot access each others options. I thought that this is a good thing as it makes the whole evaluation more scalable (lazy) and minimizes side effects between derivations.
A global package configuration feature could still be implemented as an additional module, that controls the options of all the submodules.
redesign your original module to make the field properly configurable, for example by adding an option pname-suffix, and re-define pname to contain the suffix.
The nixos module system also provides a function called extendModules. I’m not sure if this could be utilized here. I have not experimented with it so far.
Your example highlights one of the fundamental differences between overrides and modules.
In the module system, there isn’t really a before and after to base your modification on.
This can be seen as a shortcoming, but AFAIU it leads to better scalability in terms of UX at the same time.
The interface could be redesigned to make use of the module system’s merging capabilities. For example NIX_CFLAGS_LINK could be defined as a list of strings, which would allow to append to it. Or if the order of arguments really matters, defined the option to be an attribute set of arguments:
To summarize, it looks like all the options except this one are some variation on “rearchitect the package that you are overriding”. In my opinion one of the great strengths of the override[Attrs] mechanism is that it is non-invasive.
Yes, I see this as a fundamental limitation of the module-based approach compared to the current approach.
It could. A module may expose this function by declaring an option that exposes it, but doing so reintroduces the overrideAttrs weakness that it may require evaluating the package attrset without using it.
I like the idea that overriding is fundamentally a non-abstraction, and that in order to do it you’d need to have access to the implementation - the original modules. Providing this access is not necessarily the responsibility of the package attrset, which is the public interface of a package.
There’s a trade-off between “invasive” and coupled. However, I don’t think invasive is the right term. If all our overriding was based on well-defined interfaces rather than implementation details, we’d have fewer errors when updating, and package functionality will be properly discoverable.
It is not. One possible implementation of .overrideAttrs could give access to the previous config, giving access to before. To emulate the current after behavior, you could partially automate mkOverride to prioritize step by step, using the old options metadata, but I don’t think we’d really need this. mkForce should be sufficient. Also we shouldn’t want this because overriding is not declarative in spirit, as it describes a transformation rather than a single state.
For example rsyslog has way to many options. If I turn Krb5 in one place and systemd in another then I would like to combine that and build one rsyslog that has both options set and use that in both places.
I assume, that one of your goals is to build a minimal rsyslog. Otherwise the problem could be solved by just enabling all options of rsyslog by default.
I guess it should not be a big issue to create a system like that. Every package could define which features it requires enabled for its dependencies and this can then be merged by some global package configuration module. As long as we don’t need a conflict resolution mechanism, it sounds quite doable.
Rust’s “feature” mechanism works like this – it is “convex” like modules in the sense that a crate can “turn on” feature flags in its dependencies but cannot turn them off. This is why Rust features don’t create diamond dependency problems. This works because Rust features (unlike C/C++ #defines) are always booleans.
I think the nixos module-merge would be a really good fit for managing the enableFoo/fooSupport boolean flags that many packages have.
This would be “spooky action at a distance”, so it needs an overarching configuration to carry the information. NixOS could fulfill that role, but the problem is more general than that, so the solution should be general too.
We’d have to integrate the NixOS and Nixpkgs fixpoints, and use (hundreds, or thousands of) use flags instead of environment.systemPackages.
Those flags would then conditionally set environment.systemPackages, as appropriate.
This goes back to earlier discussions about “use flags” which weren’t fruitful, because we did not expect to be able to carry the extra load of having to build and troubleshoot a much larger configuration space.
I really think this, or something like it, is the way forward for a more user friendly package system. I’ve been working on something similar to your project, @DavHau. it’s similar that it’s a package system built around modules, but there’s a few differences I’m excited about that evolve on the idea. I’d love to share with you when I get something more presentable and see what you think
I watched this presentation on drv-parts. I wonder, what would be the pros and cons of having a derivation for each build step? (Especially the cons are not clear to me)
I guess one of the cons would be (1) the evaluation and build time overhead. Every derivation is expensive. In particular mkDerivation is more expensive than builtins.derivation. But i have rarely seen people using derivation directly. (2) Also the “type safety” comes at evaluation costs because (almost) every value must be typechecked. But overall i think the pros are currently much stronger.
