Installing a Python package from PyPI

Help
1

I’d like to install a Python package daktari as part of a Nix shell. It’s not available as a Nix package, so I’m attempting to use buildPythonPackage / fetchPypi in shell.nix:

{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
  nativeBuildInputs = [
    (
      let 
      my-python-packages = p: with p; [
        (
          buildPythonPackage rec {
            pname = "daktari";
            version = "0.0.74";
            src = fetchPypi {
              inherit pname version;
              sha256 = "sha256-HPpv5d7hyiMUymSKY/3flhX7eu/ygFEYGFaL35eFj2A=";
            };
            doCheck = false;
          }
        )
      ]; in pkgs.python3.withPackages my-python-packages
    )      
  ];
}

It fails with:

Executing setuptoolsBuildPhase
Traceback (most recent call last):
  File "/build/daktari-0.0.74/nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 4, in <module>
    with open("requirements.txt") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'

Hoping someone can advise on how I might get this to work.

2

If you download the source from pypi, you can see that there isn’t a requirements.txt in it (iirc this is because it isn’t listed in the manifest?)

I’m not sure if this is technically a bug on their end (it seems like it would keep source builds from working via pypi, but might be obscured by the wheels they provide?), but the file should be present if you use their github as the source.

3

Thanks - I’ve added requirements.txt added to the source distribution of the package (apologies, very much a newbie when it comes to Python packaging!).

That gets me a bit further, but I hit another problem:

error: builder for '/nix/store/y2hvl4s4n3i2gsyy4lcr8h50pjrbamm1-python3.10-daktari-0.0.75.drv' failed with exit code 1;
       last 10 log lines:
       > adding 'daktari-0.0.75.dist-info/RECORD'
       > removing build/bdist.linux-x86_64/wheel
       > Finished executing setuptoolsBuildPhase
       > installing
       > Executing pipInstallPhase
       > /build/daktari-0.0.75/dist /build/daktari-0.0.75
       > Processing ./daktari-0.0.75-py3-none-any.whl
       > ERROR: Could not find a version that satisfies the requirement requests-unixsocket==0.2.0 (from daktari) (from versions: none)
       > ERROR: No matching distribution found for requests-unixsocket==0.2.0
       >
       For full logs, run 'nix log /nix/store/y2hvl4s4n3i2gsyy4lcr8h50pjrbamm1-python3.10-daktari-0.0.75.drv'.
error: 1 dependencies of derivation '/nix/store/c7mlgblppavpf47y0mr7k0j79i56nhir-python3-3.10.7-env.drv' failed to build

Is that a Python problem, or something specific to buildPythonPackage thing? The package will normally install happily with a pip install in a virtual env. requests-unixsocket definitely exists on PyPI at that version.

4

Mostly the latter. The nix build itself doesn’t have network access (to limit a source of non-deterministic behavior), so dependencies aren’t dynamically available from pip or npm and such. (But you can still use pip like this interactively within a nix-shell if you need, I think.)

You generally need to add each of the dependencies to the propagatedBuildInputs attr. Here’s an example in nixpkgs: https://github.com/NixOS/nixpkgs/blob/b326f62dcf5713f1d10b4c03da36dfa06ee7a72d/pkgs/tools/system/auto-cpufreq/default.nix. This might go smoothly if they are all in nixpkgs, but it can get a little hairy if you have quite a few missing.

There are some other Nix-ecosystem tools that try to help when this is a PITA, but I’d recommend trying the above before branching out.

5

Thanks - what other ways might you recommend? (I can try manually providing the deps, but that feels unfortunate for a few reasons.)

6

Pip will use whatever version is pinned even if it is very old, insecure and causes other problems. In nixpkgs we mostly ignore those pins if the package works otherwise. I would recommend to remove the pinning or change it to a lower/upper bound that is way wider.

If the packages are already in nixpkgs I would use that. Otherwise try to use any of the many python2nix thingies.

7

I guess philosophies vary on pinning - I tend to prefer the predictability of getting the exact versions of dependencies I’m using so as to avoid surprise breaking changes, but I understand it’s a tradeoff.

If the packages are already in nixpkgs I would use that. Otherwise try to use any of the many python2nix thingies.

Have you got any recommendations? GitHub - proger/python2nix: Put a PyPI package name in, get a Nix expression out. and GitHub - nix-community/pypi2nix: Abandoned! Generate Nix expressions for Python packages seem to be abandoned projects.

1 Like
8

There’s a bit of a graveyard.

GitHub - DavHau/mach-nix: Create highly reproducible python environments is filling this role for a project I work on. I should note that the creator recently indicated that their energy is focused on eventually replacing it with the multi-language-ecosystem approach in GitHub - nix-community/dream2nix: Simplified nix packaging for various programming language ecosystems [maintainer=@DavHau]. IIRC they noted that they’ll still be merging PRs, so it may be possible for the ~community to ensure it’s useful until dream2nix’s python support is ready?

If the project can use Poetry, GitHub - nix-community/poetry2nix: Convert poetry projects to nix automagically [maintainer=@adisbladis] is also an active option.

2 Likes
9

yeah, really poetry2nix work well. afaik some people say poetry will be official packager, not requirments. so it good to go. all project i built have it.

so what lacks still generator of packages + versions.

because nixpkgs cannot update so easy and bloat in size. see pydantic/fastapi issue.

https://github.com/NixOS/nixpkgs/issues/263228