How to build Python virtualenv with packages provided by 'python3.withPackages'?

Help
1

Hi,

I want to build virtual Python environment with some base packages provided by nixpkg and the rest installed by Pip (my real use case is more complicated, based on Flakes + Poetry + poetry2nix, but it behaves the same).

I can successfully create Python environment with numpy:

nix-shell -p 'python310.withPackages (p: [ p.numpy ])'

[nix-shell:~]$ which python
/nix/store/xv1vlbkpn26bg09yx5sm1ls9nkn3s8v4-python3-3.10.9-env/bin/python

[nix-shell:~]$ 
python -c "import numpy; print(numpy.version.full_version)"
1.23.3

but when I create Python virtualenv with --system-site-packages enabled

python -m venv --system-site-packages venv

and activate it, I can’t access numpy anymore

[nix-shell:~]$ source venv/bin/activate

[nix-shell:~]$ python -c "import numpy; print(numpy.version.full_version)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'

The problem is that virtualenv is using original Python python310 environment , not the one build with python310.withPackages, so even with --system-site-packages enabled, it wouldn’t find numpy, because it is not there.

  • Python environment used by virtualenv
cat venv/pyvenv.cfg
home = /nix/store/65cp4izx3bllnwqn7c7dhrq9h9gmjkal-python3-3.10.9/bin
include-system-site-packages = true
version = 3.10.9

I guess the virtualenv is using sys.base_exec_prefix to set the Python.

/nix/store/xv1vlbkpn26bg09yx5sm1ls9nkn3s8v4-python3-3.10.9-env/bin/python -c "import sys; print(sys.base_exec_prefix)"
/nix/store/65cp4izx3bllnwqn7c7dhrq9h9gmjkal-python3-3.10.9

Is there any way how to force virtualenv to use Python environment created by python310.withPackages (python3-3.10.9-env) ?

sys.exec_prefix points to Python environment which I want

/nix/store/xv1vlbkpn26bg09yx5sm1ls9nkn3s8v4-python3-3.10.9-env/bin/python -c "import sys; print(sys.exec_prefix)"
/nix/store/xv1vlbkpn26bg09yx5sm1ls9nkn3s8v4-python3-3.10.9-env

Thank you

1 Like
2

Setting PYTHONHOME to Python environment created by python310.withPackages helps.

For example, following works now:

[nix-shell:~]$ source venv/bin/activate

[nix-shell:~]$ export PYTHONHOME=/nix/store/xv1vlbkpn26bg09yx5sm1ls9nkn3s8v4-python3-3.10.9-env

[nix-shell:~]$ python -c "import numpy; print(numpy.version.full_version)"
1.23.3
3

I am not entirely sure if this fits your use case perfectly, but I’ll let you know how I set up things for my smaller python projects. I am using poetry to manage packages, but I see no reason why it shouldn’t work with other solutions.

I will use a flake.nix which looks something like this:

{
  description = "Nix Development Flake for your package";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/master";

  outputs =
    { self, nixpkgs, flake-utils }:

    flake-utils.lib.eachDefaultSystem
      (system:
      let
        pkgs = import nixpkgs { inherit system; };
        python = pkgs.python310;
        pythonPackages = python.pkgs;
      in
      {
        devShells.default = pkgs.mkShell {
          name = "your_package";
          nativeBuildInputs = [ pkgs.bashInteractive ];
          buildInputs = with pythonPackages; [
            pkgs.nodePackages.pyright
            pkgs.poetry
            setuptools
            wheel
            venvShellHook
          ];
          venvDir = ".venv";
          src = null;
          postVenv = ''
            unset SOURCE_DATE_EPOCH
          '';
          postShellHook = ''
            unset SOURCE_DATE_EPOCH
            unset LD_PRELOAD

            PYTHONPATH=$PWD/$venvDir/${python.sitePackages}:$PYTHONPATH
          '';
        };
      });
}

You can then run nix develop and get a development shell. After that you can run poetry install and it will pull in your dependencies into the .venv folder.

I am not quite sure if you really need all the unset things in postShellHook and postVenv, as I copypasted some of this from other people. I might try out to skim it down a bit, but in this way it kinda works for now :slight_smile:

4

Today we’ve just merged virtualenv support for https://devenv.sh that should simplify your setup and make this possible:

To get started:

$ nix-env -iA cachix -f https://cachix.org/api/v1/install
$ cachix use devenv
$ nix-env -if https://github.com/cachix/devenv/tarball/v0.5

Create a file devenv.nix:

{ pkgs, ... }: {
  languages.python.enable = true;
  languages.python.venv.enable = true;

  packages = [ pkgs.zlib ];
}

$ devenv shell
Building shell ...
Entering shell ...

$ pip install numpy
$ python -c "import numpy"
5

@APCodes ,

thank you very much for nice Flake example.

What I am trying to solve is the problem when you want to supply some Python dependencies from Nixpkgs (for example numpy) and the rest by Poetry.

The problem is, that if you add numpy from nixpkgs in to your Flake shell environment, and then add other packages depending on numpy or it’s dependencies via Poetry (poetry add <PACKAGE>), Poetry wouldn’t see numpy or it’s dependencies and will install them in virtualenv as well. Then you end up with duplicated packages in your final Python environment. This is quite messy and can have very unpredictable consequences.

I was trying to solve it by building Python environment using python.withPackages and then building Poetry virtualenv with system-site-packages enabled in hope that in this case Poetry will see my Python packages in python.withPackages environment but I can’t get it working.

1 Like
6

Thank you very much @domenkozar , I follow you devenv development with excitement !

7

Yeah I see what you are saying. I guess for me that has never been an issue because I chose to limit the dependencies I manage with nix anyways and let the programming language package manager do that for me. Also because this means people can install these dependencies independently of having nix installed on their systems.

However I just tested this out of curiosity with my flake adding numpy to buildInputs … and when I activate the flake, I can import numpy in python and use it. And when I try installing it from pip it tells me the requirement is already satisfied. So pip appears to pick up the numpy I added via nix.

So isn’t this what you wanted?

1 Like
8

Yes, you are right. pip can see those Nix packages, but unfortunately according my testing Poetry can’t and will install them again.

9

What’s the reason why you’d like to combine the two? Can’t everything be built using Nix or Poetry?

10

Yeah you might be right about this one. However that might be more an issue with poetry than with nix. I Personally always reinstall everything in venvs with poetry so I never had this issue before. And I use nix primarily as a replacement for something like pyenv.

But check this out:

https://github.com/python-poetry/poetry/issues/6035

11

The reason is that sometimes it is easier/faster/more convenient to install Python packages with C extensions from nixpkgs then from PyPi. And also, I am producing some custom Python packages with heavy C dependencies from my other Flake, so I want to install them from there.

12

@APCodes , thank you very much for pointing this out ! I am quite surprised that such a problem wasn’t fixed since then.

13

Hi, do you have plan for do the same thing with poetry like python.venv.enable, there are many problems when trying to install packages with poetry, like this and this, and I have to switch to use packages from ‘python3.withPackages’

14

I ran into this too. The problem is in the way the python interpreter packages are built. This PR fixes it.

6 Likes
15

@cwp 's PR does The Right Thing here.

16

For reference, 297628 is the PR from @cwp that was ultimately merged.

Update: see below for the link to the reversion of 297628 in 302385. Many thanks to @ruro and @chrism for identifying this.

1 Like
17

@cameronraysmith AFAIK, the PR you linked got reverted almost immediately, because there were some unexpected issues with it and there wasn’t enough time to fix it before ZHF. I am hopeful that it’ll get fixed eventually since the main idea of the fix was fine, the problem was mostly in the implementation details.

18

Oh ok thanks for clarifying. I updated my comment above to point to @chrism’s identification of the reversion below to minimize the potential for my original comment alone to create “mid-thread confusion” regarding the status of this work.

19

Ugh. This fixed so many problems. I’d be happy to help fix whatever is holding this back, if someone points me in the right direction. Or even a link to its reversion.

20

@chrism here is my current understanding.

Currently (without the PR applied), nix creates python package environments in such a way that they aren’t automatically picked up by the python interpreter. Instead, you have to do some ugly hacks using sitepackages.py and wrapper scripts that modify sys.argv[0] and put individual paths into site in order to ensure that the required packages are picked up at runtime.

PR #297628 basically tries to remove these hacks and instead make python just detect all the required packages like it would in a normal global python install. Unfortunately, the code in the PR tried to “unwrap” the already wrapped python scripts by just assuming that the third line in the wrapper would always be the line that modifies sys.argv[0] and site.

In reality, however, some wrappers may include extra lines (like a # -*- coding: utf-8 -*- comment, for example), which would make the “unwrapper” delete the wrong line.

A super simple, but dirty fix would be to match the lines that we want to delete by their content instead of by line number. So instead of

sed -e '1d' -e '3d' ".$prg-wrapped" >> "$out/bin/$prg"

we could do this

sed \
    -e '/^#!\/nix\/store\//d' \
    -e '/^import sys;import site;import functools;sys\.argv\[0\] = /d' \
    ".$prg-wrapped" >> "$out/bin/$prg"

This should be more robust to weird wrapper variations by only deleting the lines that start with #!/nix/store/ and import sys;import site;import functools;sys.argv[0] = instead of blindly expecting them to be the first and third line. (note: I haven’t tested this thoroughly, so there might still be some edge cases that this fix misses)

A more “proper” solution would probably be to avoid wrapping and unwrapping these scripts in the first place. In fact, in the original PR, @cwp mentioned that they had an idea regarding that. But “when the world needed him most, he vanished” and he hasn’t responded to my comments, so I am honestly not sure what was his idea.

4 Likes