Any ideas to marry libxml2, pip and nix-shell?

Help
1

Hi. I’m looking for canonical solution to following problem (like, I know the solution, but maybe there’s something better)

with import <nixpkgs> {};
mkShell rec {
    reqs = writeText "requirements.txt" ''
        lxml
    '';
    shellHook = ''
        alias pip="PIP_PREFIX='$(pwd)/build/pip_packages' \pip"
        export PYTHONPATH="$(pwd)/build/pip_packages/lib/python2.7/site-packages:$PYTHONPATH"
        unset SOURCE_DATE_EPOCH
        pip install -r ${reqs}
    '';
}

This code tries to install lxml Python library via pip. It fails with

cc -I/usr/include/libxml2 -c /run/user/1000/xmlXPathInitcQjVdW.c -o run/user/1000/xmlXPathInitcQjVdW.o
/run/user/1000/xmlXPathInitcQjVdW.c:1:10: fatal error: libxml/xpath.h: No such file or directory
 #include "libxml/xpath.h"
          ^~~~~~~~~~~~~~~~
compilation terminated.
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command 'gcc' failed with exit status 1

----------------------------------------
Command "/nix/store/5xl4sc0fv2gcj0cbnvxja14yl89nk5c6-python-2.7.14/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/run/user/1000/pip-build-iOkjh0/lxml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /run/user/1000/pip-775R70-record/install-record.txt --single-version-externally-managed --prefix /home/danbst/dev/build/pip_packages --compile" failed with error code 1 in /run/user/1000/pip-build-iOkjh0/lxml/

Which is understandable - there are no includes for libxml2. But when I add

buildInputs = [ libxml2 ];

It still doesn’t work. Why?

What works is the following:

buildInputs = [
    libxml2
    libxslt
];
C_INCLUDE_PATH =
    "${libxml2.dev}/include/libxml2:" +
    lib.makeSearchPathOutput "dev" "include" buildInputs;

But I don’t understand why should I specify C_INCLUDE_PATH - it’s mkDerivation job AFAIK.

2

In case anybody interested why do I want to install lxml using pip, not Nix: I’m trying to run Airflow. Given there are problems with nixifying it, I’ve made a nix-shell + pip replacement:

with import <nixpkgs> { };
mkShell rec {
    nativeBuildInputs = [
        pythonPackages.ipython
        pythonPackages.pip
    ];
    buildInputs = [
        libxml2
        libxslt
    ];

    C_INCLUDE_PATH =
        "${libxml2.dev}/include/libxml2/:" +
        lib.makeSearchPathOutput "dev" "include" buildInputs;
    LIBRARY_PATH = lib.makeLibraryPath buildInputs;

    shellHook = ''
        set -v

        alias pip="PIP_PREFIX='$(pwd)/build/pip_packages' \pip"
        export PYTHONPATH="$(pwd)/build/pip_packages/lib/python2.7/site-packages:$PYTHONPATH"
        export PATH="$(pwd)/build/pip_packages/bin:$PATH"
        unset SOURCE_DATE_EPOCH

        export AIRFLOW_HOME=$(pwd)
        set +v

        pip install apache-airflow
    '';
}
1 Like
3

The following works for me:

with import <nixpkgs> {};
stdenv.mkDerivation {
  name = "env";
  buildInputs = [
    libxml2
    libxslt
    python3
    python3.pkgs.pip
  ];
}

$ nix-shell --command "pip install --user lxml"
4

Thanks! I figured out, that error message misled me

Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?

The libxml2 library in nixpkgs sets up include files in non-canonical location (like, not in $out/include, but $out/include/libxml2)

so gcc can’t find those includes.

But libxslt propagates libxml2 includes (copies) to correct location. So

buildInputs = [ libxslt ];

would be sufficient, except zlib missing, which was propagated from libxml2. So your buildInputs set

and the following:

buildInputs = [ libxslt zlib ];

both work for me!

PS. I hate sometimes propagated stuff…