Flake: Trouble installing python/pip dependency that depends on libxml2

Help
1

I’m trying to follow " 17.27.3.6. How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems?" from the manual, but it doesn’t seem to work, it’s not able to see the dependencies for lxml.

Can anyone tell me what I’m doing wrong?

Flake: https://github.com/blissdev/nix-python-venv-lxml-example/blob/f2921fad79c39ae3ad351b2fd146276fbbf943ab/flake.nix

{
  description = "Example Python development environment";

  # Flake inputs
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs"; # also valid: "nixpkgs"
  };

  # Flake outputs
  outputs = { self, nixpkgs }:
    let
      # Systems supported
      allSystems = [
        "x86_64-linux" # 64-bit Intel/ARM Linux
        "aarch64-linux" # 64-bit AMD Linux
        "x86_64-darwin" # 64-bit Intel/ARM macOS
        "aarch64-darwin" # 64-bit Apple Silicon
      ];

      # Helper to provide system-specific attributes
      nameValuePair = name: value: { inherit name value; };
      genAttrs = names: f: builtins.listToAttrs (map (n: nameValuePair n (f n)) names);
      forAllSystems = f: genAttrs allSystems (system: f {
        pkgs = import nixpkgs { inherit system; };
      });
    in
    {
      # Development environment output
      devShells = forAllSystems ({ pkgs }: {
        default =
          let
            # Use Python 3.11
            python = pkgs.python311;
            pythonPackages = pkgs.python311Packages;
          in
          pkgs.mkShell {
            venvDir = "./.venv";
            # The Nix packages provided in the environment
            buildInputs = [
              python
              pythonPackages.venvShellHook

              pkgs.libxml2
              pkgs.libxslt
              pkgs.curl
            ];

              # Run this command, only after creating the virtual environment
              postVenvCreation = ''
                unset SOURCE_DATE_EPOCH
              '';

              # Now we can execute any commands within the virtual environment.
              # This is optional and can be left out to run pip manually.
              postShellHook = ''
                # allow pip to install wheels
                unset SOURCE_DATE_EPOCH
              '';
          };
      });
    };
}

And then I try to run the following:

    nix develop
    ./venv/bin/pip install -r bootstrap-requirements.txt
    ./venv/bin/pip-sync bootstrap-requirements.txt requirements.txt
2

I attempted to use propagatedBuildInputs to install libxml2/libxslt without success.

3

How do I find out where libxml2 installed its C headers?

4

Evaluate lib.getDev pkgs.libxml2

No idea on the other things.

5

I tried your flake in your link and it worked on my system, I have access to libxml inside a python repl.

What is your error exactly ?

6

@nurelin
Did you follow the README.md? If I do that I get…

      creating build/temp.macosx-11.0-arm64-cpython-311/src/lxml
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/nl6w48s4xv3p62ysl080jg38hxkxrqsi-libxcrypt-4.4.33/include -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/Users/blissdev/Projects/nix-python-venv-lxml-example/.venv/include -I/nix/store/f8fdbfylqzwmil8yl1kwkbw620p49il8-python3-3.11.1/include/python3.11 -c src/lxml/etree.c -o build/temp.macosx-11.0-arm64-cpython-311/src/lxml/etree.o -w -flat_namespace
      In file included from src/lxml/etree.c:858:
      src/lxml/includes/etree_defs.h:14:10: fatal error: 'libxml/xmlversion.h' file not found
      #include "libxml/xmlversion.h"
               ^~~~~~~~~~~~~~~~~~~~~
      1 error generated.
      Compile failed: command '/nix/store/fb4rdxr69sy1vsi98z3njxjzdq3rj3ah-clang-wrapper-11.1.0/bin/clang' failed with exit code 1
      creating tmp
      creating tmp/nix-shell.mN8kJI
      cc -I/usr/include/libxml2 -c /tmp/nix-shell.mN8kJI/xmlXPathInitir6ekzl8.c -o tmp/nix-shell.mN8kJI/xmlXPathInitir6ekzl8.o
      /tmp/nix-shell.mN8kJI/xmlXPathInitir6ekzl8.c:1:10: fatal error: 'libxml/xpath.h' file not found
      #include "libxml/xpath.h"
               ^~~~~~~~~~~~~~~~
      1 error generated.
      *********************************************************************************
      Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
      Perhaps try: xcode-select --install
      *********************************************************************************
      error: command '/nix/store/fb4rdxr69sy1vsi98z3njxjzdq3rj3ah-clang-wrapper-11.1.0/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lxml
  Running setup.py clean for lxml
Failed to build lxml

I’m wondering if this then is specific to macos/clang?

7

I think you are using the wrong libxml package

 ➜ nix-locate libxml/xmlversion.h
python310Packages.vtk.out                         8,036 r /nix/store/fby1w910r0166sbv4ykqwj5n99qg0n9s-vtk-9.2.2/include/vtk/vtklibxml2/include/libxml/xmlversion.h
vtkWithQt5.out                                    8,036 r /nix/store/vx536zn5zrc887vsp0c4q6yfijhb2r06-vtk-qvtk-9.2.2/include/vtk/vtklibxml2/include/libxml/xmlversion.h
vtk_8.out                                         6,445 r /nix/store/rsgh69hknvxsddwh9qh8smslgyclrs33-vtk-8.2.0/include/vtk-8.2/vtklibxml2/libxml/xmlversion.h
vtk_8_withQt5.out                                 6,445 r /nix/store/rdnr3lp0wrjji9y3ab0zww1zf92d85ih-vtk-qvtk-8.2.0/include/vtk-8.2/vtklibxml2/libxml/xmlversion.h
vtk.out                                           8,036 r /nix/store/v1d41sphhs7xldr96ijyvpxshxwd928s-vtk-9.2.2/include/vtk/vtklibxml2/include/libxml/xmlversion.h
python311Packages.vtk.out                         8,036 r /nix/store/yb5ifvk09m9bh06i9byp9szcd1plhpf2-vtk-9.2.2/include/vtk/vtklibxml2/include/libxml/xmlversion.h
python310Packages.libxml2.dev                     8,420 r /nix/store/aa3yil0m5qpvb39n287agyzqdva0np83-libxml2-2.10.3-dev/include/libxml2/libxml/xmlversion.h
python311Packages.libxml2.dev                     8,420 r /nix/store/yj32g007bz23fsmhq4ccbrx81bn26f70-libxml2-2.10.3-dev/include/libxml2/libxml/xmlversion.h
8

Yes I followed the Readme but I am on nixOS. Here is my build log for lxml. The interesting part seems to be:

  Building lxml version 4.9.2.
  Building without Cython.
  Building against libxml2 2.10.3 and libxslt 1.1.37
  Building against libxml2/libxslt in one of the following directories:
    /nix/store/j5w6b3v198q4zy7pwqzyg4vjcfpcgasx-libxml2-2.10.3/lib
    /nix/store/zaflwh2nwzj1f0wngd7hqm3nvlf3yhsx-zlib-1.2.13/lib
    /nix/store/zaz9l706731ni018hiwlz5is8vf8xsiz-libxslt-1.1.37/lib

which you do not seems to have in your log.

9

I found this and this in the libxml repository. I suppose the detection of libxml2 on MacOS but inside a nix shell break something.

Nixpkgs seems able to build lxml on MacOS, so you may want to try to add the listed inputs in your flake.

10

Sorry, which inputs are you referring to specifically?

11

This part:

  nativeBuildInputs = [ libxml2.dev libxslt.dev cython ] ++ lib.optionals stdenv.isDarwin [ xcodebuild ];
  buildInputs = [ libxml2 libxslt zlib ];
12

Thanks! I tried this, but unfortunately get the same error:

And if it’s any help, nix-locate gives me:

(.venv) m1mbp:nix-python-venv-lxml-example blissdev$ nix-locate libxml/xmlversion.h
(mesa.dev)                                        8,093 r /nix/store/mx69drv2c409ccpif92fhnlvgwbiqqvj-MacOSX-SDK-11.0.0/usr/include/libxml/xmlversion.h
(libxml2Python.out)                               8,420 r /nix/store/fai9gax32k7r2mb4a0801wjs31ladpr6-libxml2-2.10.0-dev/include/libxml2/libxml/xmlversion.h
python310Packages.libxml2.dev                     8,420 r /nix/store/dqydmgbf2xnpx7nrjj9958ybs7bh4amj-libxml2-2.10.0-dev/include/libxml2/libxml/xmlversion.h
python39Packages.libxml2.dev                      8,420 r /nix/store/qpnwgisbagw6dil3zjz04nji7mc1x5ab-libxml2-2.10.0-dev/include/libxml2/libxml/xmlversion.h
13

Even adding the directory to the header in $CPPFLAGS didn’t fix it. :frowning:

      creating build/temp.macosx-11.0-arm64-cpython-310/src/lxml
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/nl6w48s4xv3p62ysl080jg38hxkxrqsi-libxcrypt-4.4.33/include -I/nix/store/dqydmgbf2xnpx7nrjj9958ybs7bh4amj-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/Users/blissdev/Projects/nix-python-venv-lxml-example/.venv/include -I/nix/store/3sixc33xj6wbjd3zqd5y1m1cbfhj2j2s-python3-3.10.9/include/python3.10 -c src/lxml/etree.c -o build/temp.macosx-11.0-arm64-cpython-310/src/lxml/etree.o -w -flat_namespace
      In file included from src/lxml/etree.c:858:
      src/lxml/includes/etree_defs.h:14:10: fatal error: 'libxml/xmlversion.h' file not found
      #include "libxml/xmlversion.h"
               ^~~~~~~~~~~~~~~~~~~~~
      1 error generated.
      Compile failed: command '/nix/store/jd9bkyx2bxz8py33c6gfr59ccjikv1cf-Toolchains/XcodeDefault.xctoolchain/bin/clang' failed with exit code 1
      cc -I/usr/include/libxml2 -c /tmp/nix-shell.Y0LxgC/xmlXPathInitnu_bnmvi.c -o tmp/nix-shell.Y0LxgC/xmlXPathInitnu_bnmvi.o
      /tmp/nix-shell.Y0LxgC/xmlXPathInitnu_bnmvi.c:1:10: fatal error: 'libxml/xpath.h' file not found
      #include "libxml/xpath.h"
               ^~~~~~~~~~~~~~~~
      1 error generated.
      *********************************************************************************
      Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
      Perhaps try: xcode-select --install
      *********************************************************************************
      error: command '/nix/store/jd9bkyx2bxz8py33c6gfr59ccjikv1cf-Toolchains/XcodeDefault.xctoolchain/bin/clang' failed with exit code 1
14

I realized that I can’t access those files from nix-locate. Sorry I’m out of my knowledge here, any ideas why?

(.venv) m1mbp:nix-python-venv-lxml-example blissdev$ nix-locate libxml/xpath.h
vtk_9.out                                        16,559 r /nix/store/yw6m17svyik5ksjr51avbma747drdldp-vtk-9.1.0/include/vtk/vtklibxml2/include/libxml/xpath.h
(mesa.dev)                                       16,398 r /nix/store/mx69drv2c409ccpif92fhnlvgwbiqqvj-MacOSX-SDK-11.0.0/usr/include/libxml/xpath.h
(libxml2Python.out)                              16,763 r /nix/store/fai9gax32k7r2mb4a0801wjs31ladpr6-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h
python310Packages.libxml2.dev                    16,763 r /nix/store/dqydmgbf2xnpx7nrjj9958ybs7bh4amj-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h
python39Packages.libxml2.dev                     16,763 r /nix/store/qpnwgisbagw6dil3zjz04nji7mc1x5ab-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h
(.venv) m1mbp:nix-python-venv-lxml-example blissdev$ ls /nix/store/dqydmgbf2xnpx7nrjj9958ybs7bh4amj-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h
ls: cannot access '/nix/store/dqydmgbf2xnpx7nrjj9958ybs7bh4amj-libxml2-2.10.0-dev/include/libxml2/libxml/xpath.h': No such file or directory
15

That’s probably because nix-locate use the database of path ↔ package built for the system, not for your flake.

But I see that lxml seems to try to use pkg-config. You should try to add pkgs.pkg-config to nativeBuildInputs.

2 Likes
16

It worked! Thank you so much!

Edit: How could you tell it was trying to use pkg-config?

17

I saw a reference to pkg-config here. And then i straced the pip install to check

❯ strace -ff -e execve pip install lxml==4.9.2 --no-binary lxml
[...]
strace: Process 151777 attached
[pid 151777] execve("/nix/store/4xw8n979xpivdc46a9ndcvyhwgif00hz-bash-5.1-p16/bin/sh", ["/nix/store/4xw8n979xpivdc46a9ndc"..., "-c", "pkg-config --cflags libxml-2.0"], 0x7ffd1993a720 /* 168 vars */) = 0
[pid 151777] execve("/nix/store/k2vd7w2ixbqbgd9v88am8780111bng7j-pkg-config-wrapper-0.29.2/bin/pkg-config", ["pkg-config", "--cflags", "libxml-2.0"], 0x628010 /* 166 vars */) = 0
[pid 151777] execve("/nix/store/67x3spywsndigkqi7979fzzqili0g4dz-pkg-config-0.29.2/bin/pkg-config", ["/nix/store/67x3spywsndigkqi7979f"..., "--cflags", "libxml-2.0"], 0xa11010 /* 167 vars */) = 0
[pid 151777] +++ exited with 0 +++
[...]
2 Likes