Load .so file from other package in Python package

Help
1

Hi,

I have a derivation with lib/*.so files, let’s call that libabc for now. Then I have another derivation with a Python package, let’s call that pyabc, that uses ctypes.CDLL to load an .so file from libabc. How do I get pyabc to find .so files in libabc?

  1. I could take the location of libabc and in the derivation of pyabc add a file or symlink in pyabc to reference libabc. That would probably work, but modifies the pyabc package in a way that feels suboptimal to me.
  2. Set LD_LIBRARY_PATH to include a reference to libabc. That works, but I do not think changing LD_LIBRARY_PATH is very nixy and it also does not propagate well into devshells as it needs to be added manually.
  3. Merge the .so files in pyabc. That works, but feels less elegant and does not allow other packages to just load the .so files from libabc.

Does anyone have another idea on how to do this in a nice way?

2

ok I think I have found a way that is satisfying enough. let me report that here.

I decided to go with makeWrapper to amend the environment observed by certain binaries in pyabc. buildPythonPackage already provides makeWrapperArgs to provide additional arguments to the makeWrapper command. see my derivation below:

        libabc = pkgs.callPackage (import ./abc/default.nix) { inherit pkgs; };
        LIBABC_PATH = pkgs.lib.makeLibraryPath [ libabc ];

        pyabc = (with pkgs;
          python3Packages.buildPythonPackage {
            pname = "pyabc";
            version = "0.1.0";
            src = nix-gitignore.gitignoreSource [ "*.nix" "/*.lock" ] ./.;
            format = "pyproject";

            buildInputs = [ libabc ];

            propagatedBuildInputs = with python3Packages; [
              numpy
            ];

            # We do not provide access to libabc during the pytest check hook
            checkInputs = with python3Packages; [ pytestCheckHook ];

            # Wrap generated binaries to include libabc path
            makeWrapperArgs = [ "--set LIBABC_PATH ${LIBABC_PATH}" ];
          }
        );

now the python package uses the environment variable to load the .so file:

def _load_libabc() -> ctypes.CDLL:
    # Load driver
    driver = pathlib.Path(os.getenv("LIBABC_PATH", default="")) / "libabc.so"
    return ctypes.CDLL(str(driver))

now my python entry points/scripts defined in `pyproject.toml, which are installed by the derivation of the python package, have access to this environment variable and load the .so file correctly. the environment variable is not available from a development shell you might use for local development. you could still achieve that to support local testing:

        devShells = {
          # Default devshell
          default = pkgs.mkShell {
            # Environment variables
            inherit LIBABC_PATH;
            # Packages
            packages = (with pkgs; [
              (pkgs.python3.withPackages (ps: [
                ps.autopep8 # for autoformatting
                ps.flake8 # for testing
                ps.pytest # for testing
                ps.mypy # for testing
              ] ++ pyabc.propagatedBuildInputs))
            ]);
          };

any python packages that use pyabc as an input would not be able to find the .so files without inheriting the LIBABC_PATH variable. that is unfortunate, but not a problem for my use case. so if someone knows how to do that, I am happy to hear that. for now, I’ll consider this solved!

3

The common pattern is patching the path directly. That way the module will work even in python3.withPackages, without having to muck around with environment variables:

1 Like
4

thank you so much @jtojnar , that was exactly the solution I was looking for!