Best way to find necessary system libraries for Python projects

Help
1

I am trying to continue development on a Python project which I started building when I was using Debian. I just switched to NixOS, and I am really loving it. However I am really struggling to set up a working python environment with this project.

I want to use pip to install my project and so I have been following some instructions from the NixOS wiki, specifically on using virtual environments:.

I’m creating a pip-shell.nix file with the following:

{ pkgs ? import <nixpkgs> {} }:
(pkgs.buildFHSUserEnv {
  name = "pipzone";
  targetPkgs = pkgs: (with pkgs; [
    python39
    python39Packages.pip
    python39Packages.virtualenv
    libGL
    libGLU
    xorg.libX11
  ]);
  runScript = "bash";
}).env

When I try to import a library called pyglet I was initially getting two errors:
ImportError: Library "GL" not found, ImportError: Library "GLU" not found, ImportError: Library "X11" not found By searching the Nix packages I was able to find libGL, libGLU and xorg.libX11 which I just had to guess were the required libraries.

These were easy enough to figure out with some guesswork, but my question is: going forward, how on earth is one meant to know all these libraries and which ones to add to their shell.nix files?

For example. In my new shell I want to load a program called napari. Upon trying I get the following traceback

Traceback (most recent call last):
  File "/home/georgeos/SyMBac/lib/python3.9/site-packages/numpy/core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/home/georgeos/SyMBac/lib/python3.9/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/home/georgeos/SyMBac/lib/python3.9/site-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: libz.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/georgeos/SyMBac/bin/napari", line 8, in <module>
    sys.exit(main())
  File "/home/georgeos/SyMBac/lib/python3.9/site-packages/napari/__main__.py", line 431, in main
    _run()
  File "/home/georgeos/SyMBac/lib/python3.9/site-packages/napari/__main__.py", line 230, in _run
    from napari import run, view_path
  File "/home/georgeos/SyMBac/lib/python3.9/site-packages/napari/_lazy.py", line 42, in __getattr__
    from scipy import stats  # noqa: F401
  File "/home/georgeos/SyMBac/lib/python3.9/site-packages/scipy/__init__.py", line 67, in <module>
    from numpy import show_config as show_numpy_config
  File "/home/georgeos/SyMBac/lib/python3.9/site-packages/numpy/__init__.py", line 144, in <module>
    from . import core
  File "/home/georgeos/SyMBac/lib/python3.9/site-packages/numpy/core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/home/georgeos/SyMBac/bin/python"
  * The NumPy version is: "1.22.0"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libz.so.1: cannot open shared object file: No such file or directory

How should I go about figuring out how to get libz.so.1 imported, along with the other NumPy C extensions (other than posting on here)?

Thank you

Edit: I was able to find out through some Googling that libz.so.1 is from nixos.zlib But now I get a new import error: ImportError: fontconfig not found. Now I’ve added fontconfig to my pip-shell.nix, but this time it hasn’t solved the issue.

2

Yeah I have no idea. I’ve been able to get some pretty complicated envs setup, but its been like going to war with the os battling through cuda support, glibc issues, injecting them into pytorch, fixing ssl problems, numpy mismatches, ncurses problems, openmp, vulkan drivers, pyglet, along with the x11/libGLU/libGL problems you’re facing.

I regularly need to both read the nix source code and guess-and-check misc options to get things working. It seems that currently using nix is limited to unhealthily determined people, and outstandingly lucky, who either already have or need to acquire an in-depth understanding of unix, C++ linking/compilation, build systems, shell, drivers, and nix itself.

I’m sorry to say the million-and-one missing .so errors is normal. Sometimes they’re fixed by using a different version from nixpkg, from Nix Package Versions

3

For nixpkgs, we patch the interpreter so that it doesn’t try to “find libraries” (example patch).

Within nixpkgs, we hardcode the paths for libraries and commands by using substitution

For numpy, we also do substitution, but items such as libz should be able to be found at build time from stdenv. The issue you’re having is that you’re getting numpy installed through pip, but the numpy installed through nixpkgs should work.

I have a video on using nixpkgs with venv for a development environment.

2 Likes
4

Give this a try:

1 Like
5

Some years later, I have the same problems and the same question as you… It seems there is no other way than adding more and more stuff to your nix-shell until it works. In my case it was y ImportError: libexpat.so.1: cannot open shared object file error, followed by the same ImportError: libz.so.1 error and so i found this post. I just want to share my FHSUserEnv which should work for a lot of python packages:

{ pkgs ? import <nixpkgs> {} }:
# https://ryantm.github.io/nixpkgs/builders/special/fhs-environments/
(pkgs.buildFHSUserEnv {
  name = "choose your own"; # <-- set name here
  # Packages to be installed for the main host's architecture (i.e. x86_64). Along with libraries binaries are also installed.
  targetPkgs = pkgs: (with pkgs; [
    # *** Python***
    python311Full
    python311Packages.pip
    # *** OS ***
    libgcc 
    binutils
    coreutils
    expat
    libz
  ]);
 # Packages to be installed for all architectures supported by a host (i.e. i686 and x86_64). Only libraries are installed by default.
  multiPkgs = pkgs: (with pkgs; []);
  runScript = "bash";
}).env

Furthermore, sometimes you have to set LIBRARY_PATH or LD_LIBRARY_PATH env and maybe add it to multiPkgs instead of targetPkgs as described here

I hope this gives some ideas for people facing related problems…