How to build Python virtualenv with packages provided by 'python3.withPackages'?

Help
1

Hi,

I want to build virtual Python environment with some base packages provided by nixpkg and the rest installed by Pip (my real use case is more complicated, based on Flakes + Poetry + poetry2nix, but it behaves the same).

I can successfully create Python environment with numpy:

nix-shell -p 'python310.withPackages (p: [ p.numpy ])'

[nix-shell:~]$ which python
/nix/store/xv1vlbkpn26bg09yx5sm1ls9nkn3s8v4-python3-3.10.9-env/bin/python

[nix-shell:~]$ 
python -c "import numpy; print(numpy.version.full_version)"
1.23.3

but when I create Python virtualenv with --system-site-packages enabled

python -m venv --system-site-packages venv

and activate it, I can’t access numpy anymore

[nix-shell:~]$ source venv/bin/activate

[nix-shell:~]$ python -c "import numpy; print(numpy.version.full_version)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'

The problem is that virtualenv is using original Python python310 environment , not the one build with python310.withPackages, so even with --system-site-packages enabled, it wouldn’t find numpy, because it is not there.

  • Python environment used by virtualenv
cat venv/pyvenv.cfg
home = /nix/store/65cp4izx3bllnwqn7c7dhrq9h9gmjkal-python3-3.10.9/bin
include-system-site-packages = true
version = 3.10.9

I guess the virtualenv is using sys.base_exec_prefix to set the Python.

/nix/store/xv1vlbkpn26bg09yx5sm1ls9nkn3s8v4-python3-3.10.9-env/bin/python -c "import sys; print(sys.base_exec_prefix)"
/nix/store/65cp4izx3bllnwqn7c7dhrq9h9gmjkal-python3-3.10.9

Is there any way how to force virtualenv to use Python environment created by python310.withPackages (python3-3.10.9-env) ?

sys.exec_prefix points to Python environment which I want

/nix/store/xv1vlbkpn26bg09yx5sm1ls9nkn3s8v4-python3-3.10.9-env/bin/python -c "import sys; print(sys.exec_prefix)"
/nix/store/xv1vlbkpn26bg09yx5sm1ls9nkn3s8v4-python3-3.10.9-env

Thank you

1 Like
2

Setting PYTHONHOME to Python environment created by python310.withPackages helps.

For example, following works now:

[nix-shell:~]$ source venv/bin/activate

[nix-shell:~]$ export PYTHONHOME=/nix/store/xv1vlbkpn26bg09yx5sm1ls9nkn3s8v4-python3-3.10.9-env

[nix-shell:~]$ python -c "import numpy; print(numpy.version.full_version)"
1.23.3
3

I am not entirely sure if this fits your use case perfectly, but I’ll let you know how I set up things for my smaller python projects. I am using poetry to manage packages, but I see no reason why it shouldn’t work with other solutions.

I will use a flake.nix which looks something like this:

{
  description = "Nix Development Flake for your package";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/master";

  outputs =
    { self, nixpkgs, flake-utils }:

    flake-utils.lib.eachDefaultSystem
      (system:
      let
        pkgs = import nixpkgs { inherit system; };
        python = pkgs.python310;
        pythonPackages = python.pkgs;
      in
      {
        devShells.default = pkgs.mkShell {
          name = "your_package";
          nativeBuildInputs = [ pkgs.bashInteractive ];
          buildInputs = with pythonPackages; [
            pkgs.nodePackages.pyright
            pkgs.poetry
            setuptools
            wheel
            venvShellHook
          ];
          venvDir = ".venv";
          src = null;
          postVenv = ''
            unset SOURCE_DATE_EPOCH
          '';
          postShellHook = ''
            unset SOURCE_DATE_EPOCH
            unset LD_PRELOAD

            PYTHONPATH=$PWD/$venvDir/${python.sitePackages}:$PYTHONPATH
          '';
        };
      });
}

You can then run nix develop and get a development shell. After that you can run poetry install and it will pull in your dependencies into the .venv folder.

I am not quite sure if you really need all the unset things in postShellHook and postVenv, as I copypasted some of this from other people. I might try out to skim it down a bit, but in this way it kinda works for now :slight_smile:

4

Today we’ve just merged virtualenv support for https://devenv.sh that should simplify your setup and make this possible:

To get started:

$ nix-env -iA cachix -f https://cachix.org/api/v1/install
$ cachix use devenv
$ nix-env -if https://github.com/cachix/devenv/tarball/v0.5

Create a file devenv.nix:

{ pkgs, ... }: {
  languages.python.enable = true;
  languages.python.venv.enable = true;

  packages = [ pkgs.zlib ];
}

$ devenv shell
Building shell ...
Entering shell ...

$ pip install numpy
$ python -c "import numpy"
5

@APCodes ,

thank you very much for nice Flake example.

What I am trying to solve is the problem when you want to supply some Python dependencies from Nixpkgs (for example numpy) and the rest by Poetry.

The problem is, that if you add numpy from nixpkgs in to your Flake shell environment, and then add other packages depending on numpy or it’s dependencies via Poetry (poetry add <PACKAGE>), Poetry wouldn’t see numpy or it’s dependencies and will install them in virtualenv as well. Then you end up with duplicated packages in your final Python environment. This is quite messy and can have very unpredictable consequences.

I was trying to solve it by building Python environment using python.withPackages and then building Poetry virtualenv with system-site-packages enabled in hope that in this case Poetry will see my Python packages in python.withPackages environment but I can’t get it working.

1 Like
6

Thank you very much @domenkozar , I follow you devenv development with excitement !

7

Yeah I see what you are saying. I guess for me that has never been an issue because I chose to limit the dependencies I manage with nix anyways and let the programming language package manager do that for me. Also because this means people can install these dependencies independently of having nix installed on their systems.

However I just tested this out of curiosity with my flake adding numpy to buildInputs … and when I activate the flake, I can import numpy in python and use it. And when I try installing it from pip it tells me the requirement is already satisfied. So pip appears to pick up the numpy I added via nix.

So isn’t this what you wanted?

1 Like
8

Yes, you are right. pip can see those Nix packages, but unfortunately according my testing Poetry can’t and will install them again.

9

What’s the reason why you’d like to combine the two? Can’t everything be built using Nix or Poetry?

10

Yeah you might be right about this one. However that might be more an issue with poetry than with nix. I Personally always reinstall everything in venvs with poetry so I never had this issue before. And I use nix primarily as a replacement for something like pyenv.

But check this out:

https://github.com/python-poetry/poetry/issues/6035

11

The reason is that sometimes it is easier/faster/more convenient to install Python packages with C extensions from nixpkgs then from PyPi. And also, I am producing some custom Python packages with heavy C dependencies from my other Flake, so I want to install them from there.

12

@APCodes , thank you very much for pointing this out ! I am quite surprised that such a problem wasn’t fixed since then.

13

Hi, do you have plan for do the same thing with poetry like python.venv.enable, there are many problems when trying to install packages with poetry, like this and this, and I have to switch to use packages from ‘python3.withPackages’

14

I ran into this too. The problem is in the way the python interpreter packages are built. This PR fixes it.

5 Likes
15

@cwp 's PR does The Right Thing here.