Packaging python packages which depends on libstdc++.so.6

Help
1

Hi all, I am running into a problem when packaging a specific python package called blspy that I am not sure how to solve and would like consult experienced developers here.

Some context here:

  1. The python library itself is developed in c++ and wrapped into python library with pybind.
  2. Instead of build the package from source, I tried to build it based on the wheel file provided on pypi.

The first version of packaging it looks like this:

{ lib
, pkgs
, stdenv  
, buildPythonPackage
, isPy37
, isPy38
, isPy39
, fetchPypi
, setuptools
}:

let url = if isPy37 then 
      "https://files.pythonhosted.org/packages/6f/9d/bf6ddb26bea4e04f1ba09725ce4a1eb92883b037e2fa8613ccfb00442b3e/blspy-1.0.1-cp37-cp37m-manylinux2010_x86_64.whl"
    else if isPy38 then
      "https://files.pythonhosted.org/packages/4b/99/74154d22dcb42faff37ca86154254542532e249acc40b2cd1025071f06d7/blspy-1.0.1-cp38-cp38-manylinux2010_x86_64.whl"
    else if isPy39 then
      "https://files.pythonhosted.org/packages/3b/52/89047e3fba2a55290bcc37b69a55b307779f81afdaece9829bae3faa0d2f/blspy-1.0.1-cp39-cp39-manylinux2010_x86_64.whl"
    else "WRONG!";

in buildPythonPackage rec {
  pname = "blspy";
  version = "1.0.1";

  format = "wheel";

  src = builtins.fetchurl {
    inherit url;
    sha256 = "1mms0by14v7lxcskm0x5r3gyfw1ixyaf00h6l1ld65zsp1pp0ys9";
  };

  propagatedBuildInputs = [ setuptools ];

  meta = with lib; {
    description = "BLS Signatures implementation";
    homepage = "https://github.com/Chia-Network/bls-signatures";
    license = licenses.asl20;
    maintainers = with maintainers; [ breakds ];
  };
}

I was able to build it. However, when I am trying to test it with a devShell (or a nix-shell), it fails on import blspy.

Python 3.8.8 (default, Feb 19 2021, 11:04:50)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import blspy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

Just FYI, the flake that has the dev shelled defined can be found here in case this helps reproducing the problem.

I understand that for the nix-shell or devShell this can be addressed by manually overriding the LD_LIBRARY_PATH. However in this case the main goal is about packaging the python library, is there any way I can add stdenv.cc.cc.lib as a runtime dependency for the pythonPackage that I am building, so that when the user add this package to their environment (nixos or nix-shell), such dependency will be implicitly installed and found at run time?

Thanks in advance!

1 Like
2

Try to add autoPatchelfHook to the nativeBuildInputs:

nativeBuildInputs = [ autoPatchelfHook ];

But it seems to me that is simpler to package it from the source, so you don’t need to verify the Python version. Use fetchPypi to fetch the source code and add cmake to nativeBuildInputs.

But if you just want this package to use in your project, maybe it’s better to something like Mach-Nix or Poetry2nix.

1 Like
3

Worked like a charm, thanks @brogos

I ended up with this derivation definition:

{ lib
, pkgs
, stdenv  
, buildPythonPackage
, isPy37
, isPy38
, isPy39
, fetchPypi
, setuptools
}:

let url = if isPy37 then 
      "https://files.pythonhosted.org/packages/6f/9d/bf6ddb26bea4e04f1ba09725ce4a1eb92883b037e2fa8613ccfb00442b3e/blspy-1.0.1-cp37-cp37m-manylinux2010_x86_64.whl"
    else if isPy38 then
      "https://files.pythonhosted.org/packages/4b/99/74154d22dcb42faff37ca86154254542532e249acc40b2cd1025071f06d7/blspy-1.0.1-cp38-cp38-manylinux2010_x86_64.whl"
    else if isPy39 then
      "https://files.pythonhosted.org/packages/3b/52/89047e3fba2a55290bcc37b69a55b307779f81afdaece9829bae3faa0d2f/blspy-1.0.1-cp39-cp39-manylinux2010_x86_64.whl"
    else "WRONG!";

in buildPythonPackage rec {
  pname = "blspy";
  version = "1.0.1";

  format = "wheel";

  src = builtins.fetchurl {
    inherit url;
    sha256 = "1mms0by14v7lxcskm0x5r3gyfw1ixyaf00h6l1ld65zsp1pp0ys9";
  };

  buildInputs = [ stdenv.cc.cc.lib ];

  propagatedBuildInputs = [ setuptools ];

  nativeBuildInputs = [ pkgs.autoPatchelfHook ];  

  meta = with lib; {
    description = "BLS Signatures implementation";
    homepage = "https://github.com/Chia-Network/bls-signatures";
    license = licenses.asl20;
    maintainers = with maintainers; [ breakds ];
  };
}

I might have misunderstanding about Mach-Nix though, but I would like to add an overlay that contains those packages so that other people can use it in their NixOS configuration. I think that is not the purpose of Mach-Nix?

And I do agree it might be good to package it from the source, which is something I plan to do after I get the whole packages working.

Appreciate the help!

1 Like
4

The way I see Mach-Nix and Poetry2nix they are a replacement to virtualenvs. If you have a Python project you can use them to create a nix-shell with all the python packages in needed versions installed.

2 Likes
5

Yep that aligns with my understanding too. In this case what we are looking for is a deployment environment instead of a development environment. Thanks!