Python, devenv, poetry and libraries

Help
1

Hi

I switched from Debian to NixOS recently, and I still wasn’t able to setup a working Python development environment.

TL;DR; How can I use python libs using C libraries (like wand or lxml) in a poetry venv or in a pipx devenv?

Some projects I develop rely on python dependencies like lxml (https://lxml.de/) or Wand (Wand — Wand 0.6.11).
When I develop these projects, I use poetry (https://python-poetry.org/) to setup my virtualenv.

Also, as a user, I use pipx (GitHub - pypa/pipx: Install and Run Python Applications in Isolated Environments) to install python apps in virtualenv in my userland (when apps are not packaged by NixOS).

I tried a lot of configurations, and I ended up with using Home-manager like

...
  home.packages = with pkgs; [
    # Python dev
    (python3.withPackages (p: with p; [ 
      poetry pipx yt-dlp
      lxml wand magic
    ]))
  ];
...

This seems to do the job for the default python3 environment, the next command does not fail:

$ realpath $(which python3)
/nix/store/0hmch7jfgyvhgvc9lqb56kc4lrl0jivy-python3-3.10.11-env/bin/python3
$ echo "from wand.image import Image" | python3

But then I try to use wand python library in a venv, I get an error

$ cd $(mktemp -d)
                     
$ poetry init --quiet

$ poetry add wand
Creating virtualenv tmp-mubmvzuo5u-4wZocLvi-py3.10 in /home/seb/.cache/pypoetry/virtualenvs
Using version ^0.6.11 for wand

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing wand (0.6.11)
  
$ poetry run sh -c 'echo "from wand.image import Image" | python3'
Traceback (most recent call last):
  File "/home/seb/.cache/pypoetry/virtualenvs/tmp-mubmvzuo5u-4wZocLvi-py3.10/lib/python3.10/site-packages/wand/api.py", line 154, in <module>
    libraries = load_library()
  File "/home/seb/.cache/pypoetry/virtualenvs/tmp-mubmvzuo5u-4wZocLvi-py3.10/lib/python3.10/site-packages/wand/api.py", line 143, in load_library
    raise IOError('cannot find library; tried paths: ' + repr(tried_paths))
OSError: cannot find library; tried paths: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/seb/.cache/pypoetry/virtualenvs/tmp-mubmvzuo5u-4wZocLvi-py3.10/lib/python3.10/site-packages/wand/image.py", line 18, in <module>
    from . import assertions
  File "/home/seb/.cache/pypoetry/virtualenvs/tmp-mubmvzuo5u-4wZocLvi-py3.10/lib/python3.10/site-packages/wand/assertions.py", line 155, in <module>
    from .color import Color  # noqa: E402
  File "/home/seb/.cache/pypoetry/virtualenvs/tmp-mubmvzuo5u-4wZocLvi-py3.10/lib/python3.10/site-packages/wand/color.py", line 10, in <module>
    from .api import library
  File "/home/seb/.cache/pypoetry/virtualenvs/tmp-mubmvzuo5u-4wZocLvi-py3.10/lib/python3.10/site-packages/wand/api.py", line 178, in <module>
    raise ImportError('MagickWand shared library not found.\n'
ImportError: MagickWand shared library not found.
You probably had not installed ImageMagick library.
Try to install:
  https://docs.wand-py.org/en/latest/guide/install.html

$ poetry run sh -c 'realpath $(which python3)'

/nix/store/rpri9nb8xpwwqakyrqbg8zdslkjs2hd3-python3-3.10.11/bin/python3.10

I cannot understand how poetry installed a virtualenv cannot access to C library whereas the default python3 on my system can access to these libraries.

I noticed the the realpath of the python executable are not the same in my virtualenv and by default for my user.
The python not in the virtualenv is in a folder named xxxx-python3-3.10.11-env/, I suppose because it actually contains the dependencies I specified in my home-manager config. But poetry seems to ignore this python executable and use the one from the python3 package, without dependencies.
I have the same behaviour when I use pipx, it seems to use the python executable without packages/libraries.

The only way I found, was to run poetry in a nix-shell with shell.nix containing

{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
  nativeBuildInputs = with pkgs; [
    poetry
    python3
    python3Packages.wand
  ];
}

But I would like to understand and find a way to declare my python dependencies properly, any idea?

2

I’m not a nix pro, so what I’ll say below might not be quite accurate. However, here’s what I learned from my experience.

This is quite strange, I’m not sure how python venvs actually work under the hood and how does it find the python executable. Maybe it’s the way specifically the poetry package on nixos works.

It’s generally expected that “unpatched” solutions don’t work on nixos. The deal here is that whatever python is executed, it most probably looks for C libraries with the ld loader. Run ldconfig -p on any regular distribution and you’ll get a list of system libraries, on nixos this command will fail. Any “unpatched” program doesn’t have access to system libraries.

However, it’s not the time to give up! The preferred solution should be using poetry2nix. It’s quite easy and it worked for me with minimum amount of issues. I don’t know about wand, but lxml ran just fine for me. You just generate your venv not directly through poetry, but with the use of nix, which takes care of dealing with C libraries and many other pain points.

The other question is how to use this tool. It depends on your workflow, but there’s a great chance you need to revise some part of it. It feels like the approach with nixos is to make as much things as possible “containerized”, e.g. writing shell.nix for every project you engage in. Well, it’s my personal perspective and you don’t have to follow it at all, but that’s what I do. I have systemwide python with scipy, sympy and ipython, but not much more. And when I need some specific library for my project I’ll just write it down inside a flake.

With all the above taken into consideration, I use docker for e.g. jupyter, it’s just waaaay easier this way. I don’t care about reproducibility inside notebooks, I just want to make things done as quickly as possible and without any distractions by “how do I use this specific tool in nix???”. So docker and maybe distrobox are your friends for life :slight_smile:

1 Like
3

Thanks you @heinwol for you complete answer.

I don’t think the problem is directly lonked to poetry as I found the exact same issue with pipx.
I tend (at least before switching to nixos) to use poetry to setup my venv for development, and use pipx to install pypi application inside venv too.

I’ll have a look to poetry2nix, and see if it fits my needs for my development workflow.

As a fallback, I’ll use docker … but I don’t like docker for that purpose.
As allowing docker for a user is equivalent to give root permissions … I try to avoid using docker for users, only for services.

Thanks again