I’m trying to create a poetry based build environment.
I seem to run into issues with python-pam not being able to find the shared library. Something like this:
>>> import pam
>>> p = pam.pam()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/nix/store/2lziz36b63gp78jg93q0x54rz6waa3wb-python3-3.10.11-env/lib/python3.10/site-packages/pam/__internals.py", line 189, in __init__
libpam = CDLL(find_library("pam"))
File "/nix/store/rpri9nb8xpwwqakyrqbg8zdslkjs2hd3-python3-3.10.11/lib/python3.10/ctypes/__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libpam.so.0: cannot open shared object file: No such file or directory
I noticed that the nix version of python-pam, has a patch for the library path:
Initially I was not using poetry2nix, so I thought that poetry2nix might help. This is my simple test case
So the reason you’re running into it is that the find_library function in setuptools (or wherever the python ecosystem has shuffled it to nowadays) won’t work on nix, I recall reading the implementation a few years ago and realizing that it simply cannot find libraries in /nix/store.
Hence, any python package that links to native libraries and is installed using setuptools will always fail to build on nix.
The patch you’re taking from nixpkgs simply hard-codes those paths, replacing the find_library calls. This is enough to fix the problem and allows the build to work. That is probably the best approach - having to patch stuff to work with nix is quite common (I wrote a long-form explanation as to why here if this confuses you).
I remember trying to patch the find_library function once but I don’t recall why I gave up on it. Either way, nix packages are intended to have library paths hard-coded (so that the output stays pure and always uses the same input packages you defined). Using whatever ldd finds at runtime is not desirable, so find_library can’t really be implemented reasonably I think.
I guess this could be contributed to nixpks?
The reason it’s broken for you is because you’re using poetry, which takes the normal python package and tries to link it against your nix libraries. You can’t make this change to the pip package, because it won’t work on other distros.
Given the package on nixpkgs already has those overrides, it will already work if you use nix instead of poetry to install it. What exactly do you intend to contribute to nixpkgs?
You might be able to contribute it to poetry2nix’ built-in override list, I’m not sure exactly what they accept.
I don’t quite understand how the project workflow works, I guess that’s why I said nixpkgs.
I’m not using the poetry2nix repository directly, so poetry2nix is just the toplevel object in nixpkgs.
I assume the contents of the poetry2nix project is periodically pulled into nixpkgs?
Ah, yes, pretty much. The author writes a PR to update it every once in a while. How often it’s updated also depends a bit on whether you’re on stable or unstable, because for the most part there’s a “no breaking changes on stable” rule in place.
This is how it works for pretty much all packages, by the way, only unusual thing is that the poetry2nix source is actually copied into the nixpkgs tree. nixpkgs itself contains only the NixOS modules, package definitions (i.e. build scripts which pull their source code from elsewhere) and some library functions.