Python package install using poetry + nix

#1

I am trying to install cartopy in a poetry virtual environment with system packages provided via nix. Here is my shell.nix file:

{ 
  pkgs ? import <nixpkgs> {} 
}:

pkgs.stdenv.mkDerivation rec {
  name = "poetry-hvplot";
  buildInputs = [ 
    pkgs.glibc
    pkgs.libcxx
    pkgs.gcc
    # pkgs.qt5.full
    pkgs.poetry
    pkgs.geos
    pkgs.snappy
    pkgs.graphviz
    pkgs.pkgconf
    pkgs.geckodriver
    pkgs.firefox
  ];

  shellHook = ''
    export ENVNAME="${name}";

    export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:${pkgs.graphviz}/lib/pkgconfig

    # gcc, snappy, graphviz
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${pkgs.stdenv.cc.cc.lib}/lib
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${pkgs.snappy}/lib
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${pkgs.graphviz}/lib

    #export CPATH=$CPATH:${pkgs.stdenv.cc.cc}/include/c++/9.3.0:${pkgs.glibc.dev}/include
    #export C_INCLUDE_PATH=$C_INCLUDE_PATH:${pkgs.stdenv.cc.cc}/include/c++/9.3.0:${pkgs.glibc.dev}/include
    #export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:${pkgs.stdenv.cc.cc}/include/c++/9.3.0:${pkgs.glibc.dev}/include
    #export CFLAGS="-I${pkgs.stdenv.cc.cc}/include/c++/9.3.0 -I${pkgs.glibc.dev}/include"
    #export CPPFLAGS="-I${pkgs.stdenv.cc.cc}/include/c++/9.3.0 -I${pkgs.glibc.dev}/include"
  '';
}

Once I enter the nix-shell and try to run “poetry add cartopy” which tries to install the package in a virtual environment but finds that gcc is not able to find stdlib.h. As you may have noticed in the shell.nix above I have tried to provide the include paths in multiple ways but of them work and I get the following build error. Any reason this may be happening:

    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DACCEPT_USE_OF_DEPRECATED_PROJ_API_H=1 -I/nix/store/bs03sg8b0gq2zr4v252hh9psp780qj5q-python3-3
.8.5/include -I./lib/cartopy -I/home/rowhit/.cache/pypoetry/virtualenvs/hvplot-test-5ZyshoS1-py3.8/lib/python3.8/site-packages/numpy/core/include -I/nix/store/kqq4b77277zjy7
jrdqz9c7cxik32maq7-geos-3.8.1/include -I/home/rowhit/.cache/pypoetry/virtualenvs/hvplot-test-5ZyshoS1-py3.8/include -I/nix/store/bs03sg8b0gq2zr4v252hh9psp780qj5q-python3-3.8
.5/include/python3.8 -c lib/cartopy/trace.cpp -o build/temp.linux-x86_64-3.8/lib/cartopy/trace.o
    In file included from /nix/store/rclksjxdjgp6y6qkxyl9m4dx4b9d45zk-gcc-9.3.0/include/c++/9.3.0/ext/string_conversions.h:41,
                     from /nix/store/rclksjxdjgp6y6qkxyl9m4dx4b9d45zk-gcc-9.3.0/include/c++/9.3.0/bits/basic_string.h:6493,
                     from /nix/store/rclksjxdjgp6y6qkxyl9m4dx4b9d45zk-gcc-9.3.0/include/c++/9.3.0/string:55,
                     from /nix/store/rclksjxdjgp6y6qkxyl9m4dx4b9d45zk-gcc-9.3.0/include/c++/9.3.0/bits/locale_classes.h:40,
                     from /nix/store/rclksjxdjgp6y6qkxyl9m4dx4b9d45zk-gcc-9.3.0/include/c++/9.3.0/bits/ios_base.h:41,                                  
                     from /nix/store/rclksjxdjgp6y6qkxyl9m4dx4b9d45zk-gcc-9.3.0/include/c++/9.3.0/ios:42,
                     from lib/cartopy/trace.cpp:659:
    /nix/store/rclksjxdjgp6y6qkxyl9m4dx4b9d45zk-gcc-9.3.0/include/c++/9.3.0/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
       75 | #include_next <stdlib.h>
          |               ^~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
#2

I’m not sure how to properly include build time dependencies into poetry envs.
But the newest version of cartopy is available in nixpkgs, so there is probably no need to.
You could use mach-nix to mange your python deps, which will automatically pick such dependency from nixpkgs.

#3

@DavHau: Thanks for the response. I have tried mach-nix, it’s a nice tool. However, when you are in development mode adding new python packages and re-building and relaunching Jupyter notebook everytime you need a library makes the workflow tedious. It is in this particular case where I found poetry to work really well.

Btw the python38Packages.cartopy installation fails for me.

-- Docs: https://docs.pytest.org/en/latest/warnings.html
= 10 failed, 336 passed, 1 skipped, 109 deselected, 1 xfailed, 14 warnings in 17.70 seconds =
builder for '/nix/store/vckngakxbix90iar6l0d6dgr8y1949s8-python3.8-cartopy-0.17.0.drv' failed with exit code 1
error: build of '/nix/store/vckngakxbix90iar6l0d6dgr8y1949s8-python3.8-cartopy-0.17.0.drv' failed
#4

You don’t need to “relaunch” Jupyter

(in your nix file you have shellHook=’’ jupyter lab ‘’, right? )

  1. make .nix modifications
  2. save your Notebook (optional -> because jupyter save it automatically -> depending on the settings)
  3. cancel Jupyter server/ exit nix-env
  4. re-execute .nix file / refresh the “old session” (still opened) in your browser

-> Nothing to relaunch

ps: which extensions do you use in jupyter with poetry?

“Jupyter lab command that needs to be re-run” -> not if you use the shellHook=’’ jupyter lab … ‘’

#5

Hi @igel,

Thanks a lot for the response. Yes, the above flow you mentioned is exactly what I have. What I meant from “relaunch” is the Jupyter lab command that needs to be re-run. The opened Jupyter notebooks kernel needs to be restarted to use the latest environment. I am using the following extension:

jupyter labextension install @pyviz/jupyterlab_pyviz

I am slowly realizing that nixpkgs is great for supplying system-level packages. However, for development purposes, it’s better to rely on the individual programming package manager (not my first preference but the only practical way at the moment). But here too I am encountering the issue mentioned above where I am unable to pass the Flags and environment variables inside poetry (python package manager) required to build some specific packages.

Once you know the packages that you would need for your work, then it totally makes sense to pin them down with nixpkgs (using overlays or yyyy2nix). I am looking at lorri, niv and direnv to see if it partially resolves the incremental build and hotloads.

Regards,
Rohit

#6

What do you mean exactly?

let
  pyEnv = mach-nix.mkPython rec {

    requirements =  ''
        jupyterlab
        geopandas
        pyproj
        pygeos
        shapely>=1.7.0
      '';

    providers.shapely = "sdist,nixpkgs";
  };
in
mkShell rec {

  buildInputs = [
    pyEnv
  ] ;

  envVar="pw";


  shellHook = ''
    jupyter lab --notebook-dir=~/
  '';
}

no need for pinning if you don’t like that

#7

Hi @igel,

Here is my exact use case which is what I posted in the first message of this thread. I am trying to install cartopy using the poetry2nix method (it can very well be machnix too). However, when getting into the nix-shell it seems GCC is not able to find “stdlib.h” file which exists and provided by GCC and Glibc (both). But ignored and thus cartopy package build failed. This is where I wanted to pass C_INCLUDE_PATH etc as an environment variable (CFLAGS etc method seemed to not work either). But for some reason, these are being ignored. I am curious as to why these passed variables are being ignored and thus GCC fails to build. It may very well be an issue with poetry2nix, I do not know at the moment.

Regards,
Rohit

#8

Niv is (in the process of being) superseded by nix flakes.

direnv + nix-direnv or lorri solve the hot load problem only for your shell AFAIK. All applications would still need to be re-started.

It would be nice to come up with a solution to also make this work for running applications.
I imagine one could make jupyter look for its dependencies in a certain path and have a shell hook, that always symlinks all dependencies inside that path. Therefore, whenever direnv/lorri update your environment and execute the hook, jupyter would not need to be restarted.

It may very well be an issue with poetry2nix, I do not know at the moment.

Maybe open an issue in their github and see.

#9

This works for me:

{
  pkgs ? import <nixpkgs> {}
}:

pkgs.stdenv.mkDerivation rec {
  name = "poetry-hvplot";
  buildInputs = [
    pkgs.poetry
    pkgs.geos
    pkgs.proj_5
    pkgs.snappy
    pkgs.graphviz
    pkgs.pkgconf
    pkgs.geckodriver
    pkgs.firefox
  ];
  nativeBuildInputs = [
    pkgs.python38Packages.cython
    pkgs.geos
    pkgs.proj_5
  ];
  propagatedBuildInputs = [
    pkgs.python38Packages.six pkgs.python38Packages.pyshp pkgs.python38Packages.shapely pkgs.python38Packages.numpy
  ];
}

I just looked at the what the existing Python package uses and pulled that in but also got rid of the extra GCC and such. With this as shell.nix I did poetry init and poetry add cartopy and it installed without issue. If I used your shell.nix as it was I got the same missing header file error.

#10

If a package exists in the nixpkgs repository then I have no issues.

#11

Sorry, I don’t understand what you mean. In my example cartopy was installed with Poetry, which I thought was the main issue you are trying to work around.

#12

@austin: I apologize for delayed response. I understand that in some cases some packages may go through depending on the combination of python (or system) packages in a nix environment. What I am curious is to learn how to navigate throught that where I cannot build certain packages. Here cartopy for me was complaining about a missing stdlib.h file. In those cases how do I provide the environment variable to the poetry build environment the header and object files path?

#13

When I used your shell.nix I had the same issue you reported. I made some changes to your shell.nix in my previous post and that ended up solving the missing header issue. Have you tried with the shell.nix I posted?

#14

@austin: Yes, I did and it seems to work. Thanks!