PyO3, maturin: python native dependency management vs nixpkgs

Help
1

TLDR

  • Do you have any experience with using PyO3 (and mabye maturin) in Nix-controlled projects?

  • Would you recommend using maturin or avoiding it, in a Nix context?

  • Do you have any words of wisdom about using PyO3 to call Python from Rust, in Nix-controlled projects?

Background ramblings

PyO3 is a tool for creating bindings between Rust and Python. For quite some time, I have had an ad-hoc flake which provides the Rust toolchain and a Python interpreter with whatever packages I require, and allows me to use PyO3 to write a Python module in Rust and import it in Python. It’s a bit hacky, but it solved my short-term problems.

I now need to do the reverse: call Python from Rust. PyO3 has this covered, but in practice my initial attempts run into link-time errors.

Looking at the PyO3 docs, I see that, nowadays, PyO3 recommends maturin as the easiest way to get started with PyO3, and I’m wondering whether using maturn would be better than trying to lay everything out myself. What discourages me is that maturin uses Python’s virtualenvs, and the docs suggest all sorts of state-modifying CLI incantations, which make my Nix-attuned mind tremble with fear.

If my dependencies are fully controlled by Nix, I am confident that checking out an arbitrary commit will result in direnv automatically updating the versions of all dependencies to the ones with which that commit was tested. Admittedly, I allow (and trust) cargo to take care of this for the Rust dependencies. For a few years now, I’ve been using Nix to protect me from the nightmare that is Python packaging, so I have a (perhaps irrational) distrust of the Python tools. I certainly don’t trust the Python tools to do the right thing, to the extent that I trust cargo.

I was first made aware of maturin when I tried to install polars in Nix, and was confronted with a maturin failed error message. Progress on the polars package in nixpkgs has been limited to it getting marked as broken, and the experience has tainted maturin (perhaps unfairly) as guilty, by association.

2

I just succeeded to use nix to build a maturin-based project.
The idea is simple, get everything done using maturin (maturin init, etc…), and then use it in the nix build process to get the juicy wheel file.

Maturin with --offline and --target-dir won’t depend much on virtualenvs and stuff.
That being said, I only tried Rust-to-Python building, not the reverse so I cannot tell for this.

Here’s the template I made myself for this: nix-build-templates/maturin.nix at 6e4961dc56a9bbfa3acf316d81861f5bd1ea37ca · litchipi/nix-build-templates · GitHub

Hope it helps at least a little

2 Likes
3

Thanks for sharing your config! I had to make a few minor changes to get it working (perhaps because I upgraded to nixpkgs 23.05):

  • disable the separate artifact building step. I was running into some permission errors when pyo3 was trying to overwrite a file from the previous step.
  • change the wheel filename tail from cp310-cp310-linux_x86_64 to cp310-cp310-manylinux_2_34_x86_64. Not sure what that’s about.

Here’s my final working example: GitHub - OliverEvans96/maturin-nix-example

2 Likes