Are there any good examples for developing Python packages which will only be developed and deployed in Nix environment (so all the dependencies will be in possibly pinned nixpkgs)? Do I need non-Nix python development tools like poetry, etc. at all? How would I incorporate tools like black, mypy, sphinx, etc. in the workflow? Should I structure it as a flake or as an overlay for nixpgks? All the examples I have seen so far try to satisfy non-Nix developers (poetry2nix, match-nix, etc.) which makes them I think more complicated than what I need.
2 Likes
I found an example for running the additional tools (mypy, pylint, black) for checking a Python application (not a package) in nixpkgs: https://github.com/NixOS/nixpkgs/blob/33afbf39f6f2a6b37e99f070ba7d17a28c416d02/nixos/lib/test-driver/default.nix. The questions about structuring the package development work still remain.
It al depends on how you want to expose your package. Do you want to expose importable modules? Then you need to have the right site-packages module. Executables, then it’s recommended to have a bin folder.
I typically package things “as usual” with flit or hatchling as Python build backend but then stop bothering with version constraints (or even adding dependencies at all!) in pyproject.toml because Nix anyway provides the dependencies. I just use the build backend so all files get put in the right place.
1 Like
Yes, I want importable modules. It sounds like what I need is the most minimalistic build backend. My Python packaging experience is mostly with building somebody else’s Python packages (and going through the pains of creating Nix expressions for them!), so I don’t really have an opinion on which backend is sufficient but doesn’t try to manage the dependencies. The official documentation uses Hatchling, is that what the community considers the easiest? The only complication (aside from calling the additional tools like mypy which seems to be solved by that nixpkgs example) is in my world (data science) I often need to call c++ code (either from static of shared libraries – all provided via nix) from Python so whatever build backend I use should be able to deal with that.
Unfortunately there are no “community guidelines”, but yes, as easiest backend I suggest using flit or hatchling.
If you need to use native code as well I strongly recommend using meson. Scipy nowadays also uses meson (check the scipy .nix file!) though the build-backend meson-python.
In my opinion the PyPA should take the stance of recommending these 3 backends. It is very easy to just upgrade from flit/hatchling to meson, when you need it.
Thanks, so far my exposure to meson is this issue python3Packages.scipy can't be build with other blasProvider than openBlas · Issue #206866 · NixOS/nixpkgs · GitHub that broke my scipy due to the use of MKL as the BLAS provider everywhere in our environment. I am guessing your suggestion to use mesonFlags should work for my own packages because they are going to be built from scratch.