Inputs in Python derivations

Help
1

Background

“What inputs do setup_requires , install_requires and tests_require map to?” says

In a setup.py or setup.cfg it is common to declare dependencies:

  • setup_requires corresponds to nativeBuildInputs
  • install_requires corresponds to propagatedBuildInputs
  • tests_require corresponds to checkInputs

That covers two of the build input lists, but there are two others:

  • buildInputs
  • propagatedNativeBuildInputs

Both of these are used in Nixpkgs, as seen by

% rg buildInputs pkgs/development/python-modules | wc -l
586
% rg propagatedNativeBuildInputs pkgs/development/python-modules | wc -l
3

Question

What is the flowchart or 2x2 table that I can go through to determine which of the following locations is the right place for a dependency in a Python derivation?

  • buildInputs
  • nativeBuildInputs
  • propagatedBuildInputs
  • propagatedNativeBuildInputs
2

There is no flowchart, but you can find more on the details of python packaging in the nixpkgs manual:
NixOS - Nixpkgs 21.05 manual (See: “15.19.2.2.1.1. buildPythonPackage parameters”)

The stdenv.mkDerivation function accepts various parameters for describing build inputs (see “Specifying dependencies”). The following are of special interest for Python packages, either because these are primarily used, or because their behaviour is different:

  • nativeBuildInputs ? [] : Build-time only dependencies. Typically executables as well as the items listed in setup_requires .
  • buildInputs ? [] : Build and/or run-time dependencies that need to be be compiled for the host machine. Typically non-Python libraries which are being linked.
  • checkInputs ? [] : Dependencies needed for running the checkPhase . These are added to nativeBuildInputs when doCheck = true . Items listed in tests_require go here.
  • propagatedBuildInputs ? [] : Aside from propagating dependencies, buildPythonPackage also injects code into and wraps executables with the paths included in this list. Items listed in install_requires go here.
3

Thanks for the reference. That section doesn’t mention propagatedNativeBuildInputs, so I’ll take a guess.

Depending on whether a dependency is needed at runtime and needs to be compiled, this table says where it should be placed.

needs to be compiled doesn’t need to be compiled
needed at runtime propagatedBuildInputs propagatedNativeBuildInputs
not needed at runtime buildInputs nativeBuildInputs

Is this right? Or should propagatedNativeBuildInputs just never be used? – In which case, what explains the exceptions? The only examples of it in nixpkgs python-modules are:

% rg propagatedNativeBuildInputs pkgs/development/python-modules    
pkgs/development/python-modules/bootstrapped-pip/default.nix
18:  # Should be propagatedNativeBuildInputs

pkgs/development/python-modules/mesonpep517/default.nix
34:  propagatedNativeBuildInputs = [ meson ninja ];

pkgs/development/python-modules/protobuf/default.nix
21:  propagatedNativeBuildInputs = [ buildPackages.protobuf ];  # For protoc.
4

You can refer to the standard section Specifying Dependencies in the nixpkgs manual. The semantics of dependencies and dependency propagation don’t differ from normal derivations and python packages.

propagatedNativeBuildInputs are probably not documented specially for Python since they are rarely used there (and generally as far as I am aware).

Your chart is a bit inaccurate:

buildInputs may be used at runtime and are. From the manual:

A list of dependencies whose host platform and target platform match the new derivation’s. […] These are often programs and libraries used by the new derivation at run -time, but that isn’t always the case. For example, the machine code in a statically-linked library is only used at run-time, but the derivation containing the library is only needed at build-time. Even in the dynamic case, the library may also be needed at build-time to appease the linker.

The key difference between native and normal buildInputs is their platform: native variant dependencies need to run on the build machine while normal variant dependencies (buildInputs, propagatedBuildInputs) need to be built for the target platform the derivation is built for.

Thus build tools (meson, setuptools_scm, …) are usually nativeBuildInputs because they are only used on the build machine and the built derivation will absolutely never need them (I think it is impossible for a built derivation to reference a nativeBuildInput but I am not 100% sure). While libraries the application depends on for example are buildInputs.

What now about propagation? In pythonPackages almost all python library dependencies are specified as propagatedBuildInputs. Propagation means that if package A depends on package B it will also depend on the propagatedBuildInputs of package B (as if they were buildInputs). In pythonPackages this is necessary because if package B imports package C, package A will also need to have package C in its PYTHONPATH although it does not directly depend on it.

propagatedNativeBuildInputs is the propagated variant of nativeBuildInputs: If package A depends on package B then propagatedNativeBuildInputs of package B will be added to package A’s nativeBuildInputs. A situation were this may be necessary is if package B cannot be linked against without a specific build tool or something similar. I’ve never encountered this before though, maybe the packages you found via grepping have a note somewhere why specifically this was deemed necessary.

1 Like