I want the python multi-version packaging system to be formally standardized by an RFC

Development
1

Currently, there’s PRs that create packages named somepackage_VERSION (ex. werkzeug_1) and there’s others that take the latest module and do overrides on it. And then there’s the way in the middle where the different version is stored in python-modules/MODULE/VERSION.nix to be reusable but unavailable in top-level/python-modules.nix

Maintainers always decide willy-nilly: I’ve now had people tell me that having another version in top-level is ok (https://github.com/NixOS/nixpkgs/pull/157689#issuecomment-1026960097), in other PRs overrides had been suggested (https://github.com/NixOS/nixpkgs/pull/128145#discussion_r659191551)

That’s clearly messed up and that’s not how it should be

I tend to be split in which method is the best, prob not going to be the one writing the RFC. I suppose this thread could serve as a discussion for the topic.

1 Like
2

Very often I find myself resurrecting older versions of packages (Python packages included) from nixpkgs history because the newer versions are either broken (in a non-nixpgks-related way) or not compatible with some other packages. I usually put those resurrected nix files to overlays. For instance, I want to have multiple versions of R available to me at the same time - the same way multiple versions of Python are available. On the other hand it is clear that building all the older versions of all packages is not sustainable and practically impossible from the compatibility standpoint, so I think a good compromise would be to keep building/testing just the latest versions but also provide an easy access (via maybe standardized functions, etc.) to the older versions. So my suggestion is not to replace the default.nix file every time there is new version of a package but keep adding them and use some kind of discoverable file names.

3

The main problem is that when python goes to do a import <module>. It will only find one.

So if we have many versions, then it becomes non-deterministic whether a collection of packages will work together.

If a package is very fragile (in terms of dependencies), then something like poetry2nix would probably be a better route to go.

Here’s my saved reply on the matter:

Pinning within python-modules is highly discouraged. This will potentially introduce incompatible versions in a user’s environment as either the new pinned version or the original version will be imported at runtime, breaking one of the packages.

The preference to handling this is to relax version bounds in the “install_requires” field. (could be in setup.py, pyproject.toml, requirements or others). In most cases, packages are still compatible with small API changes which may warrant a major version bump. We use test suites to verify that the package still works correctly.

If the package is still incompatible with the latest major version, then the most proper way to handle this is make an issue with the upstream package to adopt the latest major version. Or if upstream is not very responsive, you are free patch the source to make it compatible.

In very few circumstances, two versions of the same package are allowed to exist if the packages are extremely difficult to package. Some examples of this are tensorflow, which has huge ecosystems built around it and is hard to package. Another is django, which has 2 actively developed versions, and large ecosystems built around each.

One exception to this is applications, due to the way buildPythonApplication and toPythonApplication functions work, the related derivations will not bleed dependencies between packages. If the package doesn’t need to be imported by other python modules, then this package would be a good candidate to convert into application. You can look at https://github.com/NixOS/nixpkgs/blob/2211a7cf7498ec326fa66cb81e4cdb2c5a07b10c/pkgs/tools/admin/awscli/default.nix as an example of using an overlay within a python application.

Another valid exception is pinning within the same interpreter version. If a package no longer supports older-but-still-relevant interpreter versions, then you’re free to pin a package for an interpreter. This was very common with python2, but less common amongst python3 interpreters. (e.g. if (pythonOlder "3.8" then <older pkg> else <latest package> ;)

Info on buildPythonApplication can be found here.
Info on toPythonApplication can be found here.

4

A mechanism for this is implemented at: https://github.com/on-nix/python
and works very well

5

I packaged packages with very big closures (alone) at: https://github.com/on-nix/python
and maintenance time has been almost zero once packaged for the first time. Maintainers time is precious and we should optimize for that, so we can scale and suffer less

I find Nixpkgs pythonPackages very time consuming to maintain due to its fragility.
This fragility originates from the version constraints (bumping one package breaks others), and python.on-nix.com proves that not having this constraint is possible at no correctness cost. Correctness is not afected because the python packages set offers many versions, but only one is allowed to be present on each user-environment. Packages and versions are decoupled but coherent at the user-environment level.

I think we can incorporate many ideas from https://github.com/on-nix/python into Nixpkgs to make everyone’s life easier, particularly maintainers life easier. These ideas can be extended to other package sets like ruby as well

Questioning the status quo is important sometimes for being able to drive progress

6

Agreed, it is quite the burden. And that we are carrying over a “treat python packages like c libraries” model.

Would something like that work on nixpkgs? with restrict-eval ?

7

Yes, it uses fixed output derivations to fetch package sources, then feed that information to a stdenv.mkDerivation that builds the thing, then there is some logic to create the user-environment (setting the PATH, PYTHONPATH, etc, basically)

Everything builds with sandbox enabled, and without --impure from the flake

2 Likes