Hi! My team has been using Nix for Python development for a few years already, but I have kind of failed to find and connect with this greater Nix user community for feedback and ideas…
My use case for Nix in brief is
- reproducible development environments for developers to develop
- reproducible CI environment for CI to test, build and release
- Docker-based deployment with Nix-built docker images (no Nix on servers yet)
The main challenge from the very beginning has been, of course, Python packaging. While Python support on nixpkgs has been getting better and better, the main issue remains: Python in Nixpkgs is designed for Python users. When building software with Python, we need layer on top of Nixpkgs to pin specific versions and add missing or private packages into the mix (and I’m fine with this).
I started with writing everything by hand on top of Nixpkgs and building Docker images with custom approach. Then came various Python to Nix -generators and Nixpkgs dockerTools.
The last time I looked into Python to Nix -generators, pypi2nix seemed the most popular, but it also seemed to duplicate work by defining packages independently from the nixpkgs. I really wanted to be able to re-use nixpkgs’ well maintained rules for most of the Python packages.
Finally, from NixCon 2017 reports I learned about pip2nix, which did exactly what I wanted: it resolved Python packages from any pip-supported sources resulting an overlay on top of nixpkgs pythonPackages.
Then I wanted to abstract as much of the boilerplate as possible into a centrally managed versioned location, resulting in http://github.com/datakurre/setup.nix
Here’s my workflow, I’ve been designing setup.nix for:
each project has
requirements.txtdefining ALL development, build, test and runtime requiements fo a project (or environment)
most projects have
setup.cfgto define the project as a Python package with only real build, test and runtime requirements and console scripts (here
setup.pyis a just wrapper for executing setuptools to read package configuration from
pip2nixis used to turn requirements.txt into requirements.nix from PyPI or from private repository
finally, every project has
setup.nix-file, which usually just calls http://github.com/datakurre/setup.nix with current pkgs, pythonPackages, src and possible overrides (to be applied on top of requirements.nix).
So far so good. Because naming is hard, I named my wrapper
setup.py and followed that path to make it provide targets similar to some setup.py commands (develop, bdist_wheel), but also some extra commands. My usual use cases are:
nix-shell setup.nix -A developto give me nix-shell with all packages in
requirements.nixand Python package in development mode (re-using nixpkgs Python package development shell)
nix-build setup.nix -A envto build a Python environment with everything from
requirements.nixso that I can configure that as the project interpreter in PyCharm
nix-build setup.nix -A buildto build the evaluated version of the project
nix-build setup.nix -A bdist_wheelto build a Python wheel of the project
nix-build setup.nix -A bdist_dockerto build a dockerTools based image from the project
Then, time adds complexity.
Raw pip2nix requirements.nix overlay simply replaces nixpkgs packages with new ones, but setup.nix uses existing nixpkgs’ when available by updating their name and src with pip2nix generated data.
When package is not yet defined in nixpkgs, setup.nix cannot guess build dependencies for python packages and I still end up adding a lot of trivial (native) build dependencies such as pytest-runner, setuptools-scm, unzip, etc… for a lot of packages.
Finally, I have a use case with upstream packages with circular dependencies, which I handle by building packages without dependencies at all - assuming they are only used together anyway.
I guess my current main issues are the two last ones: Should I follow pypi2nix practice of adding usual build inputs for all packages whether they needed them or not? And is there already some “wheelhouse” style approaches for packaging Python products without making every package its own derivation at first?