Python propagatedBuildInputs: could we do better?

Development Nixpkgs Architecture
1

Recently I talked with Stéphane Bidoul in person. He is a core pip developer. I asked him about one problem I see with Python packaging when using nix: if I use a package whose run-time dependency has changed, I have to rebuild it always.

Turns out this only happens for nix. In the normal python world, you only need build-time dependencies to build a python package. Often, these are just flit, poetry-core, setuptools, wheel or similar. The package output should always be the same. Later, when you actually want to use it, is when you really need the run-time dependencies.

I can see that buildPythonPackage can make use of those in buildInputs, whereas it still requires us to add runtime dependencies as propagatedBuildInputs. Those make sense as the build will perform some tests such as python-importing the module, or even running pytests or unittests of the package. However, even if those tests are useful, shouldn’t the build output still be the same?

So, could we do better? If the only things that impact in the build output are the buildInputs, couldn’t we split the inner mechanism of buildPythonPackage into 2 derivations? One that builds the package, and another one that just tests it and returns the previous one on success.

Couldn’t we replace the propagatedBuildInputs list of derivations by a list of strings that, when evaluated, are obtained from python3Packages or similar, and then produce the dependency chain depending on that input?

If we manage to do something like this, the amount of rebuilds across nixpkgs python packages will be reduced dramatically (I guess).

4 Likes
2

couldn’t we split the inner mechanism of buildPythonPackage into 2 derivations

Iirc the longer term plan is to do exactly that, and it’s the reason pythonPackages were being migrated so as to produce a wheel first, and only then to install the outputs from that: two steps disentangled, they could be split into separate derivations. Of course, not all back-ends/not all python packages support wheel generation.

1 Like
3

FWIW today I found this feature:

buildPythonPackage = rec {
    ...
    nativeBuildInputs = [
      pythonRelaxDepsHook
    ];
    pythonRemoveDeps = [
      "some-dependency-rebuilt-very-often"
    ];
}

It can serve as an ugly workaround for downstream packaging. Edit: it doesn’t work for me.

However, what you said is very interesting @SergeK. Could we track that progress somewhere? Or be involved in the discussion?

1 Like
4

Could we track that progress somewhere? Or be involved in the discussion?

I don’t know of a place to track this issue, but it is my assumption that one can always engage in the discussion at https://matrix.to/#/#python:nixos.org

5

a place to track this issue

https://github.com/NixOS/nixpkgs/issues/170577

2 Likes