Packaging a python library that has pyproject.toml instead of setup.py

1

I’m trying to package the in-place python package at the moment (pypi, github). The repo does not include a setup.py file and instead has a pyproject.toml file. According to this SO thread this is a new official format for Python projects and will eventually supersede setup.py.

However, AFAICT nixpkgs has no support for this new project format. At least I’m getting an error:

...
patching sources
configuring
no configure script, doing nothing
building
Executing setuptoolsBuildPhase
Traceback (most recent call last):
  File "/build/source/nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "/nix/store/1g54cwl6m78rgrqsnn1pp29dbvldpvsd-python3-3.9.6/lib/python3.9/tokenize.py", line 392, in open
    buffer = _builtin_open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'setup.py'
builder for '/nix/store/bh1v55qnzafaf5lacvz642axyqxwfiym-python3.9-in-place-0.5.0.drv' failed with exit code 1

How should we go about packaging these kinds of projects in nixpkgs?

1 Like
2

Try this: GitHub - on-nix/python: Extensive collection of Python projects from PyPI, for Nix!

you don’t need to package something that’s already packaged for you :grin:

and it’s interoperable with nixpkgs:

# /path/to/my/expression.nix

let
  # import projects as explained in previous sections
  nixpkgs = import <nixpkgs> { };

  pythonOnNix = import (nixpkgs.fetchFromGitHub {
    owner = "kamadorueda";
    repo = "python-on-nix";
    # Pick a commit from this list:
    # https://github.com/kamadorueda/python-on-nix/commits/main
    rev = "2f850c4f86785fac9be357a46de11a3ce136aee8";
    # Update this manually
    sha256 = "0v38icc5zc0mr23g9pkm79hy52p2wrc82kk38dvz9ii2gsvzr09l";
  });

  # Create a Python on Nix environment as explained in previous sections
  env = pythonOnNix.python39Env {
    name = "example";
    projects = {
      "in-place" = "0.5.0";
      # Also works:
      # "in-place" = "latest";
    };
  };
in
nixpkgs.stdenv.mkDerivation {
  buildInputs = [ env ];
  builder = builtins.toFile "builder.sh" ''
    source $stdenv/setup

    set -x

    python --version
    python -c 'import in_place; print(in_place.__version__)'

    touch $out

    set +x
  '';
  name = "example";
}

$ nix-build --option sandbox true /path/to/my/expression.nix

+ python --version
Python 3.9.6

+ python -c 'import in_place; print(in_place.__version__)'
0.5.0

/nix/store/22nplnvdphzxb88m67fwi2pk4xrhyl2y-example

Future discussion:

I don’t know if nixpkgs maintainers feel comfortable with importing a third party (python-on-nix) on nixpkgs natively, but I think both projects could cooperate to achieve more together

I think python-on-nix has a lot of potential for changing the landscape for the better. Just to mention a few architectural changes with respect to nixpkgs:

  • we can have many versions of the same package
  • therefore updating a package does not break others
  • adding/updating an average package is only running $ just new <project> <version>, all automatic
  • we use pip directly, which knows how to install every package ever published to pypi, including pyproject.toml ones like the one you mention
  • etc
1 Like
3

I’m not against exploring alternative approaches to packaging the python ecosystem, but in this case I’m not interested in switching my project around to use a different packaging system with a whole different package set.

Btw, I noticed that you have jaxlib packaged but not jax. How does python-on-nix handle packages with CUDA/etc dependencies? Do you support the GPU/TPU versions of jaxlib?

1 Like
4

I just packaged jax so it’s available now (cpu flavor)

for the jax[cuda*] versions, we don’t currently support it, but it should be easy to implement, I added this issue: Support cuda releases of Jax · Issue #3 · on-nix/python · GitHub

thanks for the feedback!

1 Like
5

I think pyproject.toml projects can be packaged using poetry2nix. I haven’t had a chance to try it out myself yet.

1 Like
6

Search the Nixpkgs manual for pyproject. First hit explains you what to do.

3 Likes
7

Ah, ok yes format = "pyproject"; works.

4 Likes