Install python withPackages along with "python setup.py develop"-package for org-babel accessibility

Help
1

I am using NixOS having EXWM as my window manager and home-manager for my home configuration, as well as for my python environment (I use python3.withPackages).

I am commonly making use of regular programming via org-babel/ob-ipython. I have a single python package that I use to collect common functions for import in my org files.

This worked amazingly well under Arch Linux, where I just “installed” that python package using “python setup.py develop” and could access it from anywhere. Now with NixOS, I feel a couple of hurdles to maintain this convenient workflow and am looking either for a solution to it, or to alternative workflows with similar benefits (i.e. access to up-to-date python package code from within org-mode within the need to reinstall my python package after each change).

The documentation here, states that nix is capable of installing a package in develop mode under these given circumstances:

If we create a shell.nix file which calls buildPythonPackage , and if src is a local source, and if the local source has a setup.py , then development mode is activated.

So this is what I tried in my home-manager’s home.nix, however, since it is not a shell.nix, my package is installed in a static fashion, and changes to it are only noticed from within ipython/org-babel after switching my environment using home-manager switch.

Can anyone think of a solution on how to get my workflow (EXWM + org-babel/ob-ipython + development package) running or do have suggestions for a similar working workflow?

Thank you for any help!

2

Generally, I don’t try to have a global python install. Instead I have per directory nix-shell’s which allow for me to switch contexts quickly (this can be paired with direnv to make this fairly transparent).

You will probably want to use venv with python, example can be found here: https://github.com/NixOS/nixpkgs/blob/346de1fb8c3273ea74a0d15ff0580b0d6610a476/doc/languages-frameworks/python.section.md#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems

Another option would be to use something like poetry2nix, but I haven’t tried it personally.

1 Like
3

I was thinking about doing something like this, but so far struggled to use the correct python environment (i.e the nix-shell I would create) from within org-babel/ob-ipython.

I’ll dig some more in that direction thoug!

Thank you for your response @jonringer.

Edit:

I tried now the more nixy way as described here (same link as before) under the Section " Develop local package". Also the Packaging/Python wiki page recommends this way.

I hence wrote a shell.nix with the following content:

with import <nixpkgs> {};

let mypkg = python37Packages.buildPythonPackage rec {
  name = "mypkg";
  src = ./.;
  propagatedBuildInputs = with python37Packages; [
      pandas];
} in (mypkg)

When running nix-shell, the output looks promising (notice the Running setup.py develop):

moritz@moxps ~/P/mypkg (master)> nix-shell
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing setuptools-build-hook
Using setuptoolsBuildPhase
Using setuptoolsShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing setuptools-check-hook
Using setuptoolsCheckPhase
Executing setuptoolsShellHook
wObtaining file:///home/moritz/Projects/mypkg
Installing build dependencies … done
Getting requirements to build wheel … done
Installing backend dependencies … done
Preparing wheel metadata … done
Installing collected packages: mypkg
Running setup.py develop for mypkg
Successfully installed mypkg
WARNING: You are using pip version 20.0.2; however, version 20.1 is available.
You should consider upgrading via the ‘/nix/store/m2ghxhsifxf0rqyhglyrqjcm7xfbvws1-python3-3.7.7/bin/python3.7 -m pip install --upgrade pip’ command.
Finished executing setuptoolsShellHook

However, inside the shell I am not able to import mypkg at all (of note, which python shows a different path inside the shell compared to outside).

Am I overseeing something?

4

Actually, i just made a video that overlaps with this a lot for nixpkgs, you should take a look https://www.youtube.com/watch?v=jXd-hkP4xnU

5

Thanks for the video. It indeed exactly contains what I am interested in. In fact, the code that I used above is analog to one of your examples.

The pythonImportsCheck = ["mypkg"]; check is very useful and does NOT fail for me neither during nix-build nor when running nix-shell.
Also, when looking at my PYTHON_PATH variable, my package is indeed its first element and contains an egg-link to the correct directory:

[nix-shell:~/Projects/mypkg]$ ls /run/user/1000/tmp.UkbEtlgEE0/lib/python3.7/site-packages  # got this directory from PYTHON_PATH
mypkg.egg-link

[nix-shell:~/Projects/mypkg]$ cat /run/user/1000/tmp.UkbEtlgEE0/lib/python3.7/site-packages/mypkg.egg-link
/home/moritz/Projects/mypkg/src
../

/home/moritz/Projects/mypkg/src contains the main-package mypkg, which I want to be able to import.

Unfortunately, although the paths look good, when I spin up a python shell and try to import mypkg it does not find it. It’s especially weird, since the pythonImportsCheck passed…

For now, I’ll try to nix-build an environment that contains a python virtualenv (I only saw a way to do it with nix-shell so far…), although, I’d really like to understand what is going wrong.

In any case, thank you for your time and support.

6

I am running into the same problem. I think the issue is that the generated path only contains an an .egg-link file and not site.py and/or easy-install.pth.

Interestingly, if I remove pyproject.toml in the main project directory, these missing files are generated and I can import the module fine.

7

I have found the culprit and submitted a PR:

https://github.com/NixOS/nixpkgs/pull/89607

1 Like