Build Python package with .so files that depend on libstdc++

lriesebos · December 5, 2023, 7:36pm

My problem seems to relate to Packaging python packages which depends on libstdc++.so.6, but I haven’t been able to get my build to work.

I have a Python package which includes vendor libraries as .so files. These files are added in the MANIFEST.in to include them. These .so files depend on libstdc++, and I cannot get this package to build in the way I want. See the snippet of my flake:

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};

        LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [ pkgs.stdenv.cc.cc ];

        my-package = (with pkgs;
          python3Packages.buildPythonPackage {
            pname = "my-package";
            version = "0.1.0";
            src = nix-gitignore.gitignoreSource [ "*.nix" ] ./.;
            # inherit LD_LIBRARY_PATH;

            buildInputs = [ stdenv.cc.cc.lib ];
            nativeBuildInputs = [ autoPatchelfHook ];

            propagatedBuildInputs = with python3Packages; [
              pyvisa
              pyvisa-py
            ];

            checkInputs = with python3Packages; [ pytestCheckHook ];
          }
        );

      in
      ...

If I try to build the package, I get the following error during the pytest check hook: OSError: libstdc++.so.6: cannot open shared object file: No such file or directory. If I uncomment inherit LD_LIBRARY_PATH, the build works fine, but now I need to add that LD_LIBRARY_PATH modification to any devshell that uses the package. That doesn’t seem very nixy. The buildInputs and nativeBuildInputs are an attempt to get this fixed, but it does not seem to make a difference.

Does anyone have an idea what I might be doing wrong, or is the LD_LIBRARY_PATH modification the only way?

SergeK · December 5, 2023, 8:43pm

The package might be using dlopen() instead of declaring libstdc++.so as a dependency in a DT_NEEDED entry of ELF. You can forcefully patchelf $out/${python.sitePackages}/something/whatever.so --add-needed libstdc++.so to load the library before dlopen happens. Or you can tell autoPatchelfHook to forcefully extend the runpaths by setting something like appendRunpaths = [ "${stdenv.cc.cc.lib}/lib" ] in your derivation

lriesebos · December 5, 2023, 9:51pm

Thanks for your response! Your proposed ideas did not solve my problem immediately, so I checked the log to see if the RPATH is actually modified. The result for boththe derivation I already had and with adding the changes you proposed seems to be the same. See a small part of the log below.

...
searching for dependencies of /nix/store/2biywjqcqarl2jhlxvh14wb59fg06js5-python3.9-...
    libstdc++.so.6 -> found: /nix/store/sqhyhvf3qpnnj6xnb55kv46ckfjx2na8-gcc-11.3.0-lib/lib
setting RPATH to: /nix/store/sqhyhvf3qpnnj6xnb55kv46ckfjx2na8-gcc-11.3.0-lib/lib
auto-patchelf: 0 dependencies could not be satisfied
...

so I have the impression the RPATH is set correctly for the .so files, but the problem persists. any other ideas?

SergeK · December 5, 2023, 10:02pm

Could try adding export LD_DEBUG=libs to preInstallCheck (I think) and grep the build logs for libstdc++ to see which search paths are being used and where they come from
What exactly is inside src = nix-gitignore.gitignoreSource [ "*.nix" ] ./.;?

lriesebos · December 5, 2023, 10:42pm

ok I got some more logs. it seems that the RPATH that is set is not the one that is used for the search later. very confusing. do you think that could be the problem or what to do about it?

...
searching for dependencies of /nix/store/rdqjha80m8yg4zldkv9wswanmr5d8gax-python3.9-afrl-s3-paladin-0.1.0/lib/python3.9/site-packages/afrl_s3_paladin/vendor/smaract/lib64/libsmaractio.so
    libstdc++.so.6 -> found: /nix/store/sqhyhvf3qpnnj6xnb55kv46ckfjx2na8-gcc-11.3.0-lib/lib
setting RPATH to: /nix/store/sqhyhvf3qpnnj6xnb55kv46ckfjx2na8-gcc-11.3.0-lib/lib
...
/nix/store/3yn3fb99d9d5br9ghrviwzcvkdxpkmvx-python3-3.9.16/lib/python3.9/ctypes/__init__.py:374: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: libstdc++.so.6: cannot open shared object file: No such file or directory
------------------------------- Captured stderr --------------------------------
       216:     find library=librt.so.1 [0]; searching
       216:      search cache=/nix/store/b2hc0i92l22ir2kavnjn3z5z6mzabbvm-glibc-2.34-210/etc/ld.so.cache
       216:      search path=/nix/store/b2hc0i92l22ir2kavnjn3z5z6mzabbvm-glibc-2.34-210/lib             (system search path)
       216:       trying file=/nix/store/b2hc0i92l22ir2kavnjn3z5z6mzabbvm-glibc-2.34-210/lib/librt.so.1
       216:     
       216:     find library=libstdc++.so.6 [0]; searching
       216:      search cache=/nix/store/b2hc0i92l22ir2kavnjn3z5z6mzabbvm-glibc-2.34-210/etc/ld.so.cache
       216:      search path=/nix/store/b2hc0i92l22ir2kavnjn3z5z6mzabbvm-glibc-2.34-210/lib             (system search path)
       216:       trying file=/nix/store/b2hc0i92l22ir2kavnjn3z5z6mzabbvm-glibc-2.34-210/lib/libstdc++.so.6
       216:     
=========================== short test summary info ============================
ERROR tests/..._test.py - OSError: libstdc++.so.6: cannot open shared...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.12s ===============================

SergeK · December 5, 2023, 11:55pm

Yes I’m pretty sure that is the problem, but I’m still uncertain which process is trying to load the library. I see ctypes mentioned but I don’t expect anyone to CDLL("libstdc++"), idk what could be the purpose of that (also we’d probably see more search paths in the list than just the glibc). Could you upload the full logs somewhere? (like gist)

lriesebos · December 6, 2023, 12:48am

Yes, that sounds fair. I am working with some proprietary vendor software, so I have tried to take those names out, just so you know. nix log python depends on libstdc++ · GitHub

In Python, I am using ctypes.CDLL to load the vendor .so file libactio.so. There are more .so files that I will load later, but libactio.so is the one that is currently giving me issues. That .so file seems to only depend on libstdc++. Hope that gives some more insight!

SergeK · December 6, 2023, 1:27am

Sigh… unfortunately, we’re still missing the relevant bit, because pytest only shows a portion of the captured stderr. Could you try passing some of these flags to pytest? How to capture stdout/stderr output — pytest documentation

E.g. pytestFlagsArray = [ "-s" ]

so I have tried to take those names out, just so you kno

Sure and feel free to nuke the gist afterwards

lriesebos · December 6, 2023, 1:59am

fair point, just updated the gist!

SergeK · December 6, 2023, 10:16am

Yea, sorry, at this point I’m really not sure what’s going on

It seems that the libactio.so in site-packages is being patchelfed correctly: searching for dependencies of /nix/store/1kcf13sjc4byawbbgr6sy3hmff0ckqgy-python3.9-my-package-0.1.0/lib/python3.9/site-packages/my_package/vendor/act/lib64/libactio.so: libstdc++.so.6 -> found: /nix/store/sqhyhvf3qpnnj6xnb55kv46ckfjx2na8-gcc-11.3.0-lib/lib
I never see find library=libactio.so in the logs (expected because of CDLL("libactio.so") and LD_DEBUG=libs) so we don’t know if pytest is loading libaction from site-packages or, perhaps, from a build directory. For all we know, it might not have started loading it at all because we haven’t seen it in the logs (but they could be just truncated)
The last line in the logs before the failing libstdc++ look-up is calling init: /nix/store/s4sf0s0kbjhimkviq7ln68xbpxih2grf-python3.9-numpy-1.21.5/lib/python3.9/site-packages/numpy/random/_generator.cpython-39-x86_64-linux-gnu.so and that’s a pure C library that shouldn’t need libstdc++ (it’s not in its DT_NEEDED either)

lriesebos · December 8, 2023, 8:37pm

ok I think I have an idea what might be going on and why this initially did not work. so just posting that back here for future reference. I might be wrong, but these are my current thoughts.

first, I think there might be funny things going on with the path the tests are in and the path the build output is in. tests seem to run from /nix/store/s8s1yzl7nrmr9h6m0hizlyrlsyz40yj0-rh3jhvbm8dhg06va9d15p6bs123vp7sp-source where the .so files are not patched. the patched files are in /nix/store/1kcf13sjc4byawbbgr6sy3hmff0ckqgy-python3.9-my-package-0.1.0, and I am not sure if the tests load the unpatched .so files.

second, the .so files I am using have incomplete dependency requirements and do not list all .so files they need correctly. so that can be solved with autoPatchElfhook and appendRunpaths, but I am/was using nixpkgs-22.05 (for reasons I am not going to explain here). nixpkgs>=23.05 supports the appendRunpaths option.

eventually I got it to work on nixpkgs-22.05 by using a custom post-fixup phase as shown below. here I effectively set rpath manually due to the lack of appendRunpaths. for nixpkgs>=23.05 this custom postFixup was not needed and appendRunpaths does the job.

            # Manually adjust RPATH of .so files (autoPatchElfHook in nixpkgs<23.05 is too old to achieve this)
            postFixup =
              let
                rpath = "$out/${python3.sitePackages}/my_package/vendor/libactio/lib64/:${stdenv.cc.cc.lib}/lib";
                findCmd = "find $out -type f -name '*.so*'";
              in
              ''
                echo find .so files for RPATH patching
                ${findCmd}
                echo "setting RPATH of libraries to ${rpath}"
                ${findCmd} -exec patchelf --set-rpath ${rpath} {} \;
              '';

so I now got that to work!

finally, let me digress a bit. I also tried an approach where I just put the .so files in their own derivation to try and patch them before the python package is built. unfortunately, I do not know the path to the .so files anymore at that moment, so those files now have to be found automatically through system ld library paths. I did not manage to get that to work. in the derivation of my package I could add a patch to pass the path of the .so files to the python code, but I did not see that as a more elegant solution than the one I have now. maybe my derivation for the .so files was not correct and the .so files couldn’t be found for that reason. not sure. if it is possible to store the .so files in a separate derivation and then have python find them automatically through the system search path of ld (like how glibc is found), then I would be happy to learn that!