TLDR; I wanted to get feedback on a potential feature that allows multiple versions of the same python package to be installed in the same
PYTHONPATH . This is a general approach that is not specific to nixpkgs and could be used in other package managers. The only nix specific part is the tooling to allow for the building of these specialized packages. All of the materials/demo is in this repo https://github.com/costrouc/python-multiple-versions. (Also posted in discuss.python.org https://discuss.python.org/t/allowing-multiple-versions-of-same-python-package-in-pythonpath/2219).
Demo of Multiple Python Versions
This is a self contained demo of having multiple versions of a python
package in the same
PYTHONPATH. It requires
nix (sorry no windows support in nix). This
idea is not nix specific but would rely on package managers/builds to
allow for multiple versions.
$ nix-shell ... [nix-shell:~/p/python-multiple-versions]$ python Python 3.7.4 (default, Jul 8 2019, 18:31:06) [GCC 7.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import foobar; foobar.foobar() I am using flask version 1.0.3 >>> import bizbaz; bizbaz.bizbaz() I am using flask version 0.12.4 >>> quit() $ echo $PYTHONPATH ...:/nix/store/f3j11lk2m8ddw2j2axvcdfc2al2bk98c-flask-0.12.4/lib/python3.7/site-packages:.../nix/store/wv42si07c8wd64ravd4va4kh4j7prwlk-python3.7-Flask-1.0.3/lib/python3.7/site-packages:...
In nixpkgs we like to have a
single version of each package (preferably latest) with all packages
compatible with one another. Often times it is true that two packages
may be incompatible with one another but if it is a compiled
library/binary we have luxury of rewriting the shared library path
allowing two packages that use different versions of a package to
coexist. In python this philosophy breaks down because all packages
are specified in the global
PYTHONPATH. This means that if a package
import flask it searches the path for flask and uses the
one that it finds.
For nixpkgs this is troublesome because it prevents all packages from
being compatible with one another.
Examples of Issue
Some packages fix the version of a package such that other packages
in the same PYTHONPATH cannot depend on the latest version. For
apache-airflowfixes pendulum ==
pendulum release is over 1.5 years old and
libraries.io reports that
400+ packages depend on pendulum. We cannot let a single package
restrict the version of other packages.
How does this work?
I wrote a tool python-rewrite-imports that helps to make multiple versions possible. Lets say that package
bizbaz wants an old version of
flask==0.12.4 but we have another package
foobar that requires the latest version of
flask>=1.0. Normally these two packages would be incompatible. In order to do this we:
- Create a build of
flaskfor 0.12.4 and install
- Use Rope to rewrite all the imports of flask of itself to
flask_0_12_4_1pamldmw2y7gand rename the package to
- Rename the
distin site-packages and move the package to
- Rewrite all imports of
Rewriting all imports is done with
Rope a robust python
- Wanting several versions of a package that builds c-extensions
looks a little hard than rewriting the imports?
- Suppose package A requires
C==1.0.0and B requires
C>=1.1. Let’s say that package B calls a method in A with a
structure built from
C>=1.1and then A proceeds to call its
package C with that data. This will probably not happen often.
- Rope does not handle all rewrites currently in
python 3. Expressions within
fstringsare the only example that I
- It is impossible for Rope to handle all import rewrites. For
import flask; globals()[chr(102) + 'lask'].__version__
I believe for the vast majority of packages that require multiple
versions these issues will be rare.