Idea: mkDerivation: auto generate unwrapped package right after build

Development Nixpkgs Architecture
1

This is a naive idea: can we automatically generate an intermediate, unwrapped package right after package build (before postBuild)? This could potentially be realized in the mkDerivation internals without any user facing API changes.

Rationale: Since the actual package build is usually the most resource intensive / time-consuming part of writing / tweaking / overriding a package, it would be quite nice to reuse the build results as long as everything before preBuild has not changed.

Also, in nixpkgs there are a lot of unwrapped packages + wrappers constructed with symlinkJoin and the likes, precisely to avoid this problem of rebuilding the binaries. That’s a lot of boilerplate! Many of them can be simplified if things after postBuild are low-cost operations: one can just .overrideAttrs without rebuilding the whole package.

Implementations: I have no idea about the mkDerivation internals, so this is just a sketch of a possible way of implementing this idea:

  • When mkDerivation is called on some package, generate a package-vanilla derivation with everything after (and including) postBuild stripped, and generate the usual package derivation by wrapping package-vanilla.
  • When building the package, package-vanilla is first built, copied into a new store for the eventual package (do not symlink to avoid possible issues), and the postBuild commands are applied to generate the final package.
  • This new behavior can be protected behind a mkDerivation flag such as vanillaPackage = true;

What do you think? Again, I am relatively new to nix, and this might be an unrealistic idea; if this is the case, please let me know!

Update: as pointed out by @shamrocklee something similar is introduced very recently:
https://github.com/NixOS/nixpkgs/pull/167670
Documentation: https://github.com/NixOS/nixpkgs/blob/master/doc/build-helpers/special/checkpoint-build.section.md

3 Likes
2

IIRC, a tool for incremental build got merged into Nixpkgs recently, but I cannot recall its name.

@roberth reviewed the PR then.

2 Likes
3

Oh, wow! I didn’t know that! After some searches with the keyword “incremental build” I found:

Naturally people have long thought about this before haha! I am hoping that mechanisms of this sort can be easily turned on by e.g. flipping an option of mkDerivation. That would be ideal!

Update: is this the PR you are referring to?

https://github.com/NixOS/nixpkgs/pull/167670

2 Likes
4

I would like to have such a thing also, e.g. for the Ceph build, see here the point

Split up the build into 2 parts: The C++ part (e.g. ceph-unwrapped), and a symlinked derivation on top of it that has all the Python wrapping.
This is so that one can very quickly iterate on Python wrapping, e.g. changing Python dependencies, without a multi-hour-rebuild.

2 Likes
5

I looked at #167670 linked above (don’t want to link again to avoid triggering the github discord bot) and it’s quite amazing! It is even more powerful than what I have imagined, since it can deal with source patches as well.

However, at the moment, I am not sure if it is robust enough to be used in nixpkgs. There are also minor issues about the docs and the interface, which I would like to address in a following PR (see my reply at the end of the #167670 thread).

With that in mind, I think my original proposal of tweaking the mkDerivation internals is still worth investigating. It would be a less versatile solution than #167670, but hopefully simpler thus more robust to be used in nixpkgs, such that it can replace most of the trivial wrappers.

1 Like
6

Update: is this the PR you are referring to?

Yes! Thank you for finding it out!

2 Likes