…try
pip install, …trymach-nix, …trypython3Packages.virtualenv, …tryvenvShellHook, …trypoetry2nix, …tryconda-shell, …trymicromamba
That’s a little too many “tries” in entirely different directions, enough to overwhelm just anyone. This is exactly why python infrastructure in general is bad (could replace the list with “pip, distutils, setuptools, setup.py, pyproject, poetry, pipenv, pyenv, conda, …” and it would just as frustrating), and why the existing python UX in nixpkgs, plainly speaking, sucks - especially for newcomers. I don’t think a list of “tries” should be the first thing we suggest when explaining python in nixpkgs
To yours, @JonathanReeve, questions:
Why is this so hard to do on NixOS?
Personally, I’m thinking of two partial answers: for pre-built packages (e.g. wheels from pypi), and for nix-built packages. Disclaimer: I do not mean these as objective “facts”, other people may disagree in their evaluation
The pre-built packages are hard to use when they include or depend on any shared libraries (.so). This is because pre-built python packages are not cross-platform: in fact, they’re usually built for “manylinux”, which effectively means some old CentOS distribution with a FHS environment (including /usr/lib and a dynamic linker). The implication is that to use a pip install-ed package more complicated than pure sdist one has to run the python process in an environment that mimics that of “manylinux”. This is the mentioned conda-shell approach: one usually utilizes nixpkgs’ buildFHSUserEnv to create a (“chroot”) directory with all the needed dependencies in conventional FHS paths (/usr/lib, et c.) and spawns a shell in a mount namespace, that it would appear to see these paths
The nix-built python packages are much easier to consume. Somewhat ironically, however, I’d say they’re not nearly as convenient as nix-built binary libraries, in terms of handling transitive dependencies. When one consumes “native” libraries in Nix (say in buildInputs of mkShell or mkDerivation) one says “I want to make these (list) libraries visible” and that just works: e.g. nix will set up environment variables so that cmake can automatically discover the listed dependencies. When one uses an app that links dynamically, that will know where to find its dynamic dependencies through its header, and the dependencies will know where to find their transitive dependencies alike. Python story is quite different: rather than saying “I want to expose these libraries” one has to say “I want a python interpreter that can import these libraries” (python3.withPackages)
Of course mirroring all of pypi in nixpkgs would be infeasible and unsustainable. Solutions like mach-nix automate generating nix expressions for consuming pre-built wheels (and more, like conda packages). Other solutions, like poetry2nix, automate generating nix expressions with full build recipes, e.g. using poetry’s lock-file.
What do Python developers do on NixOS? Does everyone just do development in an Ubuntu VM? I’m at my wits’ end here
Most of the time I just use nix-built python packages. That includes heavy stuff that builds on top of pytorch/tensorflow and CUDA. When I recurrently need to use a package not provided in nixpkgs, I usually just write an expression for it: either right in the project tree, or in a nur repo.
Of course, quite often I need to just quickly try something out and cannot waste time on packaging.
For these cases I have a pre-built FHS environment (a la conda-shell) that I can enter and use pip and conda. I’m also running jupyterhub, which I spawn inside that FHS environment too. I install jupyter user kernels both from conda and from per-project nix-shells. I prefer to use nix-shell (in combination with nix-direnv and jupyter user kernels) whenever possible, because conda occasionally breaks (e.g. last time I checked it couldn’t detect GPU, although that used to work before)
I haven’t used mach-nix or poetry2nix much (I’m not even sure if I described what they do correctly), but it’s not surprising that they have rough edges: they’re solving a very ill-posed problem.