Python library with run-time lib dpendency?

1

I have a python environment which includes camelot:

devEnv = python.withPackages ( ps: [
  ps.jupyterlab
  ps.camelot
  ps.ghostscript
  pkgs.ghostscript
]);

during runtime, camelot requires ghostscript
camelot/backends/ghostscript_backend.py (abbreviated for clarity)


import sys
import ctypes
from ctypes.util import find_library

def installed_posix():
    library = find_library("gs")
    if library is not None:
        return library
    else:
            raise OSError(
                "Ghostscript is not installed. You can install it using the instructions"
                " here: https://camelot-py.readthedocs.io/en/master/user/install-deps.html"
            )

As you can see, including it in withPackages doesn’t allow Python to find it. I also tried overriding camelot as follows:

    camelotWithExtras = pyPkgs.camelot.overridePythonAttrs ( old: rec {
      propagatedBuildInputs = old.propagatedBuildInputs ++ [pkgs.ghostscript];
      buildInputs = [] ++ [pkgs.ghostscript];
      }
    );

That didn’t work either. I’m sure I’m just missing something simple.

3

Unfortunately, when I tried this, it didn’t work because I’m already using pkgs.python3Packages.ghostscript. In my script, I set python = pkgs.python313, and withPackages provides the package set: In the withPackages block, ps is set to pkgs.python313Packages.

4

Okay, so I found the solution. The trick here is that find_library simply does not and will not work on Nix.

The source reveals that the tricks it uses to find a lib are just not aplicable in Nix. I thought about adding a clause that would work on Nix to the find_library declaration, but decided against it. Because Nix pulls its libs as dependent derivations, with a known path, if you can’t provide a absolute path at build-time, you’re doing something wrong.

Still, if you want to explore patching instead of replacing, here are my notes:

The ctypes.util.load_library function used is just straight up not looking for Nix paths. It has some variables I can pass as overrides, but the guys in the discord say that won’t work right.

  • source code for load_library

  • find_library is defined in a large if statement, that declares a version appropriate for the host OS. The linux definition has its own sub-block, which starts on util.py:101. There is a large ammount of platform specific code that doesn’t run for my machine from lines 162-280. the actual declaration for my platform is at util.py:339 The definition refers to issue #9998 and just runs _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name)) Short-circuiting means this just kicks back the first to return something not none.

    • _findSoname_ldconfig is defined on util.py:282 in the same block
    • _get_soname
    • _findLib_gcc is defined elsewhere util.py:114 near the top of the posix block. It literally runs gcc or cc if it can, and greps for expr = os.fsencode(r'[^\(\)\s]*lib%s\.[^\(\)\s]*' % re.escape(name)) in the output.
    • _findLib_ldis defined on util.py:312 also in the same block.

Instead, I replace a find_library with a direct call to ctypes.cdll.LoadLibrary, which takes a direct path. I use one of the terminal utilities provided in Nix’s standard environment. Every derivation should have access to the patching utilities detailed here, and I use them as follows:


    camelotWithGS = pyPkgs.camelot.overridePythonAttrs ( old: rec {
      postPatch = (old.postPatch or "") + 
      (
        if old.version == "0.11.0" then ''
      substituteInPlace camelot/backends/ghostscript_backend.py \
        --replace-fail 'find_library("gs")' 'ctypes.cdll.LoadLibrary("${pkgs.ghostscript}/lib/libgs.so")'
      '' else ""
      );
    });

You can mostly check the docs for substituteInPlace from above for how this works, but one thing they don’t mention is that the --replace flag is now depreciated. As per issue #356002, I’ve swapped it out for --replace-fail.

Another improvement would be to use final.version instead of old.version. As is, the version-handling is broken. Using final.version should let it use what it needs, instead of ignoring any later modifications. The new override would look like: overridePythonAttrs ( final: old: rec {

I didn’t check what phases this fix works in since I don’t understand phases. But if you search github for substituteInPlace find_library you find pretty quick that everyone seems to be using it. They probably know something I don’t.