Why is it so hard to use a Python package?

Help
42

I am using a small python wrapper that injects specified library paths via LD_LIBRARY_PATH into just the python executable. Additionally and critically, it passes the path to itself into the implicit zeroth parameter which Python reflects on quite heavily for venv handling.

This way, LD_LIBRARY_PATH fiddeling is localized and venvs, including scripts installed in the venv’s bin/ work as well (because pip injects the python wrapper as shebang instead of python itself).

This of course does not work well, if you mix pip installed and nix installed python packages. So for example ipython installed through nix will be unaffected (but could be handled similarly and pip install ipython works as expected). I guess, this setup can break in multiple ways, but it is easier than autoPatchelfing the venv dir every time I install a python package with binary extension (which I was doing successfully for quite some while).

Demo flake:

{
  description = "A basic flake demoing a python wrapper with lib injection";

  inputs = { nixpkgs.url = "nixpkgs/nixos-unstable"; };

  outputs = { self, nixpkgs }:
    let
      supportedSystems = [ "x86_64-linux" ];
      forAllSystems = nixpkgs.lib.genAttrs supportedSystems;
      pkgs = forAllSystems (system: import nixpkgs { inherit system; });
      pythonVer = "311";

    in {
      packages = { };

      devShells = forAllSystems (system:
        with pkgs.${system};
        let
          python = pkgs.${system}."python${pythonVer}";
          wrappedPython = (writeShellScriptBin "python" ''
            export LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
            SCRIPT_DIR=$(${coreutils}/bin/dirname $(${coreutils}/bin/realpath -s "$0"))
            exec -a "$SCRIPT_DIR/python" "${python}/bin/python" "$@"
          '');
        in with pkgs.${system}; {
          default = pkgs.${system}.mkShell {
            # adapt to your needs
            NIX_LD_LIBRARY_PATH =
              lib.makeLibraryPath [ stdenv.cc.cc openssl zlib ];
            packages = [ wrappedPython ];
          };

        });
    };
}

And in nix develop shell just create a venv as usual: python -m venv .venv. You can check if it worked, if .venv/bin/python points to the wrapper (that’s the doing of exec -a path/to/wrapper). You obviously have to create a fresh venv for this to work.

After that just pip install and it hopefully should just work.

Hope, this is useful for you - feel free to use it. It allows me to work with my unnixified colleagues.

1 Like