Flake: Trouble installing python/pip dependency that depends on libxml2

blissdev · February 5, 2023, 7:25pm

I’m trying to follow " 17.27.3.6. How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems?" from the manual, but it doesn’t seem to work, it’s not able to see the dependencies for lxml.

Can anyone tell me what I’m doing wrong?

Flake: nix-python-venv-lxml-example/flake.nix at f2921fad79c39ae3ad351b2fd146276fbbf943ab · blissdev/nix-python-venv-lxml-example · GitHub

{
  description = "Example Python development environment";

  # Flake inputs
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs"; # also valid: "nixpkgs"
  };

  # Flake outputs
  outputs = { self, nixpkgs }:
    let
      # Systems supported
      allSystems = [
        "x86_64-linux" # 64-bit Intel/ARM Linux
        "aarch64-linux" # 64-bit AMD Linux
        "x86_64-darwin" # 64-bit Intel/ARM macOS
        "aarch64-darwin" # 64-bit Apple Silicon
      ];

      # Helper to provide system-specific attributes
      nameValuePair = name: value: { inherit name value; };
      genAttrs = names: f: builtins.listToAttrs (map (n: nameValuePair n (f n)) names);
      forAllSystems = f: genAttrs allSystems (system: f {
        pkgs = import nixpkgs { inherit system; };
      });
    in
    {
      # Development environment output
      devShells = forAllSystems ({ pkgs }: {
        default =
          let
            # Use Python 3.11
            python = pkgs.python311;
            pythonPackages = pkgs.python311Packages;
          in
          pkgs.mkShell {
            venvDir = "./.venv";
            # The Nix packages provided in the environment
            buildInputs = [
              python
              pythonPackages.venvShellHook

              pkgs.libxml2
              pkgs.libxslt
              pkgs.curl
            ];

              # Run this command, only after creating the virtual environment
              postVenvCreation = ''
                unset SOURCE_DATE_EPOCH
              '';

              # Now we can execute any commands within the virtual environment.
              # This is optional and can be left out to run pip manually.
              postShellHook = ''
                # allow pip to install wheels
                unset SOURCE_DATE_EPOCH
              '';
          };
      });
    };
}

And then I try to run the following:

    nix develop
    ./venv/bin/pip install -r bootstrap-requirements.txt
    ./venv/bin/pip-sync bootstrap-requirements.txt requirements.txt

blissdev · February 6, 2023, 4:16pm

I attempted to use propagatedBuildInputs to install libxml2/libxslt without success.

blissdev · February 7, 2023, 4:53pm

How do I find out where libxml2 installed its C headers?

Sandro · February 7, 2023, 5:24pm

Evaluate lib.getDev pkgs.libxml2

No idea on the other things.

nurelin · February 7, 2023, 5:35pm

I tried your flake in your link and it worked on my system, I have access to libxml inside a python repl.

What is your error exactly ?

blissdev · February 7, 2023, 7:48pm

@nurelin
Did you follow the README.md? If I do that I get…

      creating build/temp.macosx-11.0-arm64-cpython-311/src/lxml
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/nl6w48s4xv3p62ysl080jg38hxkxrqsi-libxcrypt-4.4.33/include -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/Users/blissdev/Projects/nix-python-venv-lxml-example/.venv/include -I/nix/store/f8fdbfylqzwmil8yl1kwkbw620p49il8-python3-3.11.1/include/python3.11 -c src/lxml/etree.c -o build/temp.macosx-11.0-arm64-cpython-311/src/lxml/etree.o -w -flat_namespace
      In file included from src/lxml/etree.c:858:
      src/lxml/includes/etree_defs.h:14:10: fatal error: 'libxml/xmlversion.h' file not found
      #include "libxml/xmlversion.h"
               ^~~~~~~~~~~~~~~~~~~~~
      1 error generated.
      Compile failed: command '/nix/store/fb4rdxr69sy1vsi98z3njxjzdq3rj3ah-clang-wrapper-11.1.0/bin/clang' failed with exit code 1
      creating tmp
      creating tmp/nix-shell.mN8kJI
      cc -I/usr/include/libxml2 -c /tmp/nix-shell.mN8kJI/xmlXPathInitir6ekzl8.c -o tmp/nix-shell.mN8kJI/xmlXPathInitir6ekzl8.o
      /tmp/nix-shell.mN8kJI/xmlXPathInitir6ekzl8.c:1:10: fatal error: 'libxml/xpath.h' file not found
      #include "libxml/xpath.h"
               ^~~~~~~~~~~~~~~~
      1 error generated.
      *********************************************************************************
      Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
      Perhaps try: xcode-select --install
      *********************************************************************************
      error: command '/nix/store/fb4rdxr69sy1vsi98z3njxjzdq3rj3ah-clang-wrapper-11.1.0/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lxml
  Running setup.py clean for lxml
Failed to build lxml

gist.github.com

https://gist.github.com/blissdev/15caeb58a22b7dd53a50047585a1a2d5

pip-failure

(.venv) m1mbp:nix-python-venv-lxml-example blissdev$ ./.venv/bin/pip-sync bootstrap-requirements.txt requirements.txt
Collecting lxml==4.9.2
  Using cached lxml-4.9.2.tar.gz (3.7 MB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: lxml
  Building wheel for lxml (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1

This file has been truncated. show original

I’m wondering if this then is specific to macos/clang?

Sandro · February 7, 2023, 9:59pm

I think you are using the wrong libxml package

 ➜ nix-locate libxml/xmlversion.h
python310Packages.vtk.out                         8,036 r /nix/store/fby1w910r0166sbv4ykqwj5n99qg0n9s-vtk-9.2.2/include/vtk/vtklibxml2/include/libxml/xmlversion.h
vtkWithQt5.out                                    8,036 r /nix/store/vx536zn5zrc887vsp0c4q6yfijhb2r06-vtk-qvtk-9.2.2/include/vtk/vtklibxml2/include/libxml/xmlversion.h
vtk_8.out                                         6,445 r /nix/store/rsgh69hknvxsddwh9qh8smslgyclrs33-vtk-8.2.0/include/vtk-8.2/vtklibxml2/libxml/xmlversion.h
vtk_8_withQt5.out                                 6,445 r /nix/store/rdnr3lp0wrjji9y3ab0zww1zf92d85ih-vtk-qvtk-8.2.0/include/vtk-8.2/vtklibxml2/libxml/xmlversion.h
vtk.out                                           8,036 r /nix/store/v1d41sphhs7xldr96ijyvpxshxwd928s-vtk-9.2.2/include/vtk/vtklibxml2/include/libxml/xmlversion.h
python311Packages.vtk.out                         8,036 r /nix/store/yb5ifvk09m9bh06i9byp9szcd1plhpf2-vtk-9.2.2/include/vtk/vtklibxml2/include/libxml/xmlversion.h
python310Packages.libxml2.dev                     8,420 r /nix/store/aa3yil0m5qpvb39n287agyzqdva0np83-libxml2-2.10.3-dev/include/libxml2/libxml/xmlversion.h
python311Packages.libxml2.dev                     8,420 r /nix/store/yj32g007bz23fsmhq4ccbrx81bn26f70-libxml2-2.10.3-dev/include/libxml2/libxml/xmlversion.h

nurelin · February 8, 2023, 5:45pm

Yes I followed the Readme but I am on nixOS. Here is my build log for lxml. The interesting part seems to be:

  Building lxml version 4.9.2.
  Building without Cython.
  Building against libxml2 2.10.3 and libxslt 1.1.37
  Building against libxml2/libxslt in one of the following directories:
    /nix/store/j5w6b3v198q4zy7pwqzyg4vjcfpcgasx-libxml2-2.10.3/lib
    /nix/store/zaflwh2nwzj1f0wngd7hqm3nvlf3yhsx-zlib-1.2.13/lib
    /nix/store/zaz9l706731ni018hiwlz5is8vf8xsiz-libxslt-1.1.37/lib

which you do not seems to have in your log.

nurelin · February 8, 2023, 5:58pm

I found this and this in the libxml repository. I suppose the detection of libxml2 on MacOS but inside a nix shell break something.

Nixpkgs seems able to build lxml on MacOS, so you may want to try to add the listed inputs in your flake.

blissdev · February 9, 2023, 5:41pm

Sorry, which inputs are you referring to specifically?

nurelin · February 9, 2023, 6:02pm

This part:

  nativeBuildInputs = [ libxml2.dev libxslt.dev cython ] ++ lib.optionals stdenv.isDarwin [ xcodebuild ];
  buildInputs = [ libxml2 libxslt zlib ];

blissdev · February 9, 2023, 6:07pm

Thanks! I tried this, but unfortunately get the same error:

gist.github.com

https://gist.github.com/blissdev/c641afd741988c7a6083146f612f0e06

flake.nix

{
  description = "Example Python development environment";

  # Flake inputs
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs"; # also valid: "nixpkgs"
  };

  # Flake outputs
  outputs = { self, nixpkgs }:

This file has been truncated. show original

And if it’s any help, nix-locate gives me:

(.venv) m1mbp:nix-python-venv-lxml-example blissdev$ nix-locate libxml/xmlversion.h
(mesa.dev)                                        8,093 r /nix/store/mx69drv2c409ccpif92fhnlvgwbiqqvj-MacOSX-SDK-11.0.0/usr/include/libxml/xmlversion.h
(libxml2Python.out)                               8,420 r /nix/store/fai9gax32k7r2mb4a0801wjs31ladpr6-libxml2-2.10.0-dev/include/libxml2/libxml/xmlversion.h
python310Packages.libxml2.dev                     8,420 r /nix/store/dqydmgbf2xnpx7nrjj9958ybs7bh4amj-libxml2-2.10.0-dev/include/libxml2/libxml/xmlversion.h
python39Packages.libxml2.dev                      8,420 r /nix/store/qpnwgisbagw6dil3zjz04nji7mc1x5ab-libxml2-2.10.0-dev/include/libxml2/libxml/xmlversion.h

blissdev · February 9, 2023, 6:37pm

Even adding the directory to the header in $CPPFLAGS didn’t fix it.

      creating build/temp.macosx-11.0-arm64-cpython-310/src/lxml
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/nl6w48s4xv3p62ysl080jg38hxkxrqsi-libxcrypt-4.4.33/include -I/nix/store/dqydmgbf2xnpx7nrjj9958ybs7bh4amj-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/Users/blissdev/Projects/nix-python-venv-lxml-example/.venv/include -I/nix/store/3sixc33xj6wbjd3zqd5y1m1cbfhj2j2s-python3-3.10.9/include/python3.10 -c src/lxml/etree.c -o build/temp.macosx-11.0-arm64-cpython-310/src/lxml/etree.o -w -flat_namespace
      In file included from src/lxml/etree.c:858:
      src/lxml/includes/etree_defs.h:14:10: fatal error: 'libxml/xmlversion.h' file not found
      #include "libxml/xmlversion.h"
               ^~~~~~~~~~~~~~~~~~~~~
      1 error generated.
      Compile failed: command '/nix/store/jd9bkyx2bxz8py33c6gfr59ccjikv1cf-Toolchains/XcodeDefault.xctoolchain/bin/clang' failed with exit code 1
      cc -I/usr/include/libxml2 -c /tmp/nix-shell.Y0LxgC/xmlXPathInitnu_bnmvi.c -o tmp/nix-shell.Y0LxgC/xmlXPathInitnu_bnmvi.o
      /tmp/nix-shell.Y0LxgC/xmlXPathInitnu_bnmvi.c:1:10: fatal error: 'libxml/xpath.h' file not found
      #include "libxml/xpath.h"
               ^~~~~~~~~~~~~~~~
      1 error generated.
      *********************************************************************************
      Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
      Perhaps try: xcode-select --install
      *********************************************************************************
      error: command '/nix/store/jd9bkyx2bxz8py33c6gfr59ccjikv1cf-Toolchains/XcodeDefault.xctoolchain/bin/clang' failed with exit code 1

blissdev · February 9, 2023, 6:40pm

I realized that I can’t access those files from nix-locate. Sorry I’m out of my knowledge here, any ideas why?

(.venv) m1mbp:nix-python-venv-lxml-example blissdev$ nix-locate libxml/xpath.h
vtk_9.out                                        16,559 r /nix/store/yw6m17svyik5ksjr51avbma747drdldp-vtk-9.1.0/include/vtk/vtklibxml2/include/libxml/xpath.h
(mesa.dev)                                       16,398 r /nix/store/mx69drv2c409ccpif92fhnlvgwbiqqvj-MacOSX-SDK-11.0.0/usr/include/libxml/xpath.h
(libxml2Python.out)                              16,763 r /nix/store/fai9gax32k7r2mb4a0801wjs31ladpr6-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h
python310Packages.libxml2.dev                    16,763 r /nix/store/dqydmgbf2xnpx7nrjj9958ybs7bh4amj-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h
python39Packages.libxml2.dev                     16,763 r /nix/store/qpnwgisbagw6dil3zjz04nji7mc1x5ab-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h
(.venv) m1mbp:nix-python-venv-lxml-example blissdev$ ls /nix/store/dqydmgbf2xnpx7nrjj9958ybs7bh4amj-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h
ls: cannot access '/nix/store/dqydmgbf2xnpx7nrjj9958ybs7bh4amj-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h': No such file or directory

nurelin · February 9, 2023, 6:51pm

That’s probably because nix-locate use the database of path <-> package built for the system, not for your flake.

But I see that lxml seems to try to use pkg-config. You should try to add pkgs.pkg-config to nativeBuildInputs.

blissdev · February 9, 2023, 7:31pm

It worked! Thank you so much!

Edit: How could you tell it was trying to use pkg-config?

nurelin · February 9, 2023, 7:38pm

I saw a reference to pkg-config here. And then i straced the pip install to check

❯ strace -ff -e execve pip install lxml==4.9.2 --no-binary lxml
[...]
strace: Process 151777 attached
[pid 151777] execve("/nix/store/4xw8n979xpivdc46a9ndcvyhwgif00hz-bash-5.1-p16/bin/sh", ["/nix/store/4xw8n979xpivdc46a9ndc"..., "-c", "pkg-config --cflags libxml-2.0"], 0x7ffd1993a720 /* 168 vars */) = 0
[pid 151777] execve("/nix/store/k2vd7w2ixbqbgd9v88am8780111bng7j-pkg-config-wrapper-0.29.2/bin/pkg-config", ["pkg-config", "--cflags", "libxml-2.0"], 0x628010 /* 166 vars */) = 0
[pid 151777] execve("/nix/store/67x3spywsndigkqi7979fzzqili0g4dz-pkg-config-0.29.2/bin/pkg-config", ["/nix/store/67x3spywsndigkqi7979f"..., "--cflags", "libxml-2.0"], 0xa11010 /* 167 vars */) = 0
[pid 151777] +++ exited with 0 +++
[...]