On the fly python imports using nixpkgs

Help
1

edit thanks @adisbladis

Check out this project if you want to import arbitrary python packages on the fly GitHub - t184256/nixpkgs-python-importer: Violate Nix philosophy, install Python packages mid-session with `from nixpkgs.scipy import scipy`.

usage for now until in unstable

nix-shell -I 'nixpkgs=https://github.com/NixOS/nixpkgs/archive/master.tar.gz' -p 'python3.withPackages(ps: with ps; [ ps.nixpkgs ps.ipython ])' --run 'ipython'

if you are looking at this much later use

nix-shell -p 'python3.withPackages(ps: with ps; [ ps.nixpkgs ps.ipython ])' --run 'ipython'

My Question

So crazy idea here of something I have wanted in python (and want to implement) that I feel that nixpkgs may be able to make possible. Suppose we have a python with no packages installed only stdlib. I would like to write the following and override the import statement such that it calls nixpkgs to download the package if it does not exist. Completely fine with partial answers and solutions that are hacks.

import numpy as np

np.array([1, 2, 3, 4])

In python it is possible to add import hooks to the “import” statement. It would try to import numpy and return an import error which would be caught and it would be fed off to call nixpkgs to add numpy to the environment via PYTHONPATH python37Packages.numpy for example.

Is something like this possible? I realize that in order for this to work the PYTHONPATH and PATH environment variables but not sure if others as well.

2

This is a neat idea, but rather than hooking into nix-env to install missing packages automatically, I think it might be better to arrange for the import hooks to dump a list of the missing Python modules to stdout, or even auto-generate an apprpirate shell.nix file. I say this because using something like a Nix shell is the canonical way to work with Python, and asking the script in question to help generate the list of Python dependencies would be a pretty neat trick.

1 Like
3

Not only is this possible, it already exists and is in nixpkgs!
https://github.com/t184256/nixpkgs-python-importer

1 Like
4

It turns out nixpkgs-python-importer was broken on master (and 19.03).
I’ve fixed and backported the changes required.

In case anyone wants to try this out here is a nice one-liner: nix-shell -I 'nixpkgs=https://github.com/NixOS/nixpkgs/archive/master.tar.gz' -p 'python3.withPackages(ps: with ps; [ ps.nixpkgs ps.ipython ])' --run 'ipython'

You can then try to import a module like so: from nixpkgs.numpy import numpy as np

4 Likes
5

Thank you for fixing this. Just want to say this is probably one of the most underrated packages! Wow it just works I am going to have to share this with others

6

At some point I considered mentioning it in the Nixpkgs manual, but it is so much against the Nix spirit that I decided against that. Not sure what would be the proper way to promote it.