Python runtime dependencies

Hi all,

I have a question regarding the Python build infrastructure and runtime dependencies.

To denote a run-time dependency for a python package, what one does is to put it in propagatedBuildInputs. This has always struck me as somewhat odd in terms of how things are named. That is: if it’s really just a run-time dependency, why put it in the build inputs? However, I do somewhat understand that this works in practice: the inputs are propagated to the closure of the python-with-pacakges that includes the python module, and in that way all packages have each other available at run-time.

However, I wonder whether this approach is not unnecessarily wasteful / overly conservative when actually building the packages: if a package is expensive to build (e.g. pandas) it will still be fully rebuilt whenever one of its dependencies is updated: even if those dependencies don’t actually come into play until runtime. I.e. even if those dependencies do not actually affect the build.

(I realize that one could argue that, when the process of building the package includes running the tests, the runtime dependencies actually become build-time dependencies… let’s ignore that for the moment, or imagine that we might change our infrastructure such that building (i.e. compiling) the package is separated out from the testing phase).

Is my thinking correct here? And is there such a thing in Nix as a runtime-only dependency, which could be used to mitigated the abovementioned wastefulness?

Klaas

For the sage package, I actually build the “library” (just build time dependencies) and the tests in separate derivations. I then have a wrapper that adds runtime dependencies.

The overhead is almost certainly not worth it for all packages though. Maybe the test/build separation would be nice to generalize.

That sounds like an interesting approach… could you share your expressions with me?

With Python one does not actually denote run-time dependencies but dependencies that need to be available when installing, which is what we do when we install the built wheel. This in effect requires them to be available during run-time. The reason we use propagatedBuildInputs is historic; all run-time dependencies are needed when building a reverse dependency, so propagatedBuildInputs seemed to make sense there.

Separating building and installing from testing has been considered as well. There are pros and cons to both. For the typical package it is just not worth it.

sagelib builds the library, sage-with-env builds the runtime, sagedoc builds the docs and default.nix links it all together.

Edit: and sage-tests tests it

I’ve been thinking that it might actually be easy for python packages in particular, since the tests are usually run after installation. Am I missing something?

I’ve been thinking that it might actually be easy for python packages in particular, since the tests are usually run after installation. Am I missing something?

I don’t think you are.

I’d say it should be possible in principle to have the output of the phases up unto the installPhase end up in the nix store separately, in such a way that [a] retrying the build after altering only phases which come after these steps does not trigger a full rebuild, but instead reuses the build from the store [b] an environment can be activated with the build from the store; dropping into a shell of that environment would then enable to debug the tests without incurring the cost of a full build.

I’d say it should be possible in principle: e.g. the fetched sources are generally available in the store, even when the rest of the build fails. However, I’m not proficient enough in nix (yet) to be able to cook this up myself. If anyone could point the way: much appreciated!