Poetry python-pam package not usable

Help
1

Hi,

I’m trying to create a poetry based build environment.
I seem to run into issues with python-pam not being able to find the shared library. Something like this:

>>> import pam
>>> p = pam.pam()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/2lziz36b63gp78jg93q0x54rz6waa3wb-python3-3.10.11-env/lib/python3.10/site-packages/pam/__internals.py", line 189, in __init__
    libpam = CDLL(find_library("pam"))
  File "/nix/store/rpri9nb8xpwwqakyrqbg8zdslkjs2hd3-python3-3.10.11/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libpam.so.0: cannot open shared object file: No such file or directory

I noticed that the nix version of python-pam, has a patch for the library path:

Initially I was not using poetry2nix, so I thought that poetry2nix might help. This is my simple test case

{
  description = "A very basic flake";

  inputs = {
    nixpkgs.url = "nixpkgs/nixos-22.11";
    flake-utils.url = "github:numtide/flake-utils";
  };
  outputs = { self, nixpkgs, flake-utils }:
  flake-utils.lib.eachDefaultSystem (system:
  let
    pkgs = nixpkgs.legacyPackages.${system};
  in
    {
      devShells.default = pkgs.mkShell {
        packages = with pkgs;
        [
          pam
          poetry
          (poetry2nix.mkPoetryEnv { projectDir = self; })
        ];
      };
    }
  );
}

Unfortunately using mkPoetryEnv does not help resolve the issue. Is there any way to use python-pam with poetry in a nix environment?

2

I think I’ve solved the problem myself, I saw a link in the docs to the defaultOverrides and then noticed that python-pam was not in the list. So I wrote my own override like so:

          (poetry2nix.mkPoetryEnv {
            projectDir = self;
            overrides = poetry2nix.overrides.withDefaults
              (self: super: {
                python-pam = super.python-pam.overridePythonAttrs (old: {
                  postPatch = ''
                    substituteInPlace src/pam/__internals.py \
                    --replace 'find_library("pam")' '"${pam}/lib/libpam.so"' \
                    --replace 'find_library("pam_misc")' '"${pam}/lib/libpam_misc.so"'
                  '';
                });
              });
          })

Is this the right approach?
I guess this could be contributed to nixpks?

1 Like
3

Is this the right approach?

So the reason you’re running into it is that the find_library function in setuptools (or wherever the python ecosystem has shuffled it to nowadays) won’t work on nix, I recall reading the implementation a few years ago and realizing that it simply cannot find libraries in /nix/store.

Hence, any python package that links to native libraries and is installed using setuptools will always fail to build on nix.

The patch you’re taking from nixpkgs simply hard-codes those paths, replacing the find_library calls. This is enough to fix the problem and allows the build to work. That is probably the best approach - having to patch stuff to work with nix is quite common (I wrote a long-form explanation as to why here if this confuses you).

I remember trying to patch the find_library function once but I don’t recall why I gave up on it. Either way, nix packages are intended to have library paths hard-coded (so that the output stays pure and always uses the same input packages you defined). Using whatever ldd finds at runtime is not desirable, so find_library can’t really be implemented reasonably I think.

I guess this could be contributed to nixpks?

The reason it’s broken for you is because you’re using poetry, which takes the normal python package and tries to link it against your nix libraries. You can’t make this change to the pip package, because it won’t work on other distros.

Given the package on nixpkgs already has those overrides, it will already work if you use nix instead of poetry to install it. What exactly do you intend to contribute to nixpkgs?

You might be able to contribute it to poetry2nix’ built-in override list, I’m not sure exactly what they accept.

1 Like
4

Thanks for the response.

Yes, to clarify, think this it could be added to poetry2nix/default.nix at 7b7956ff2325ada4ff6abb3a3e0fb7c00fdf6b61 · nix-community/poetry2nix · GitHub

Since other overrides of other packages seem to do very similar things.

5

Yep, I agree! That’s not nixpkgs though, it’s a separate project. @adisbladis will know if patches to random packages are taken, or just shoot them a PR and see if they accept it.

1 Like
6

I don’t quite understand how the project workflow works, I guess that’s why I said nixpkgs.
I’m not using the poetry2nix repository directly, so poetry2nix is just the toplevel object in nixpkgs.
I assume the contents of the poetry2nix project is periodically pulled into nixpkgs?

7

Ah, yes, pretty much. The author writes a PR to update it every once in a while. How often it’s updated also depends a bit on whether you’re on stable or unstable, because for the most part there’s a “no breaking changes on stable” rule in place.

You can see the history for unstable here: History for pkgs/development/tools/poetry2nix/poetry2nix - NixOS/nixpkgs · GitHub

This is how it works for pretty much all packages, by the way, only unusual thing is that the poetry2nix source is actually copied into the nixpkgs tree. nixpkgs itself contains only the NixOS modules, package definitions (i.e. build scripts which pull their source code from elsewhere) and some library functions.