This is something that happens in python land: python cannot deal with
multiple packages that have the same name, so trying to use two versions
of a single python package in a single interpreter will lead to one of
the two versions being just ignored.
I tried once to look in the CPython code to see how hard that’d be to
change (by using stuff like importlib for actually doing it after that),
but it looked like it is very, very deeply ingrained in CPython.
Now, maybe an importlib-based solution would work for a significant
number of packages even without CPython fixes, in which case that’d be
interesting to try and maybe use in some circumstances, but either way
nixpkgs probably won’t be able to assume that for libraries at least,
due to things like dynamic imports that the executable could be doing.
I’ll post my github template on the topic since it comes up usually 1-2 times a week:
Pinning within python-modules is highly discouraged. This will potentially introduce incompatible versions in a user’s environment as either the new pinned version or the original version will be imported at runtime, breaking one of the packages.
The preference to handling this is to relax version bounds in the “install_requires” field. (could be in setup.py, pyproject.toml, requirements or others). In most cases, packages are still compatible with small API changes which may warrant a major version bump. We use test suites to verify that the package still works correctly.
If the package is still incompatible with the latest major version, then the most proper way to handle this is make an issue with the upstream package to adopt the latest major version. Or if upstream is not very responsive, you are free patch the source to make it compatible.
In very few circumstances, two versions of the same package are allowed to exist if the packages are extremely difficult to package. Some examples of this are tensorflow, which has huge ecosystems built around it and is hard to package. Another is django, which has 2 actively developed versions, and large ecosystems built around each.
One exception to this is applications, due to the way buildPythonApplication and toPythonApplication functions work, the related derivations will not bleed dependencies between packages. If the package doesn’t need to be imported by other python modules, then this package would be a good candidate to convert into application. You can look at https://github.com/NixOS/nixpkgs/blob/eb6e2ac2c033a86afa6a606753aab0dbde8bddda/pkgs/tools/admin/awscli/default.nix as an example of using an overlay within a python application.
Info on buildPythonApplication can be found here.
Info on toPythonApplication can be found here.
)