Practically speaking, is there specific things pip could change to make nix packaging easier (if we were pip maintainers what would we do)?
Issue warnings about old/bad practices perhaps? Things of that nature?
That pretty much summarizes my experience with conda as well. They use an SMT solver to solve a constraint programming problem but the versioning conventions they use (upstream version, conda channel, build properties like with/without MKL, etc.) have some very rough edges. The nightmare ended when I found Nix a few years ago…
2 Likes
Poetry does already a good job fixing some of the problems like install order. If pip was replaced with poetry we would already gain a lot.
Other than that they should have ditched that you can execute code to generate package metadata and enforce setup.cfg.
1 Like
Other aspect I think people were complaining about is that poetry itself depends on non-stdlib packages, which makes it really hard to bootstrap, at least without a good standardized way of getting the basic packages in place.
I imagine all distros have their own hacks for that by now, and nix needs to reimplement poetry’s parsing anyway, but it still is a bit of a problem.
I understand that conda has its issues, this is a big reason as to why e.g. mamba exists. I can only recommend to try it out. Also of course, conda can largely be used only to manage binary software that python packages need and the python packages can be installed with e.g. pip. Then conda becomes more like a venv, with easy support for binary dependencies.
Conda certainly isn’t perfect. But for most people, it is a lot easier to learn and use than nix. I really like nix, and it is a great inspiration on how it does things … but for most people in this space I know, conda is just a lot simpler, despite its rough edges.
Furthermore, when it comes to the solver, conda may have its issues, but so does poetry. I used poetry for 2 years, and recently gave up on it again. It happened to me more than once that the solver tried for 1 hour or more to find a solution for the lockfile (and didn’t find one), and that in instances where just months before it worked. Also, adding a package and finding other packages downgraded was a common occurence as well. As the poetry lockfile (as I understand it) is supposed to work for all python versions simultaneously, a new released python version alone can be a real headache.
Long story short - I don’t think there is a perfect solution here, just tradeoffs.
I got a video on this, I skipped to the point where I just create a venv leveraging nix: https://youtu.be/jXd-hkP4xnU?t=1295
3 Likes
I don’t know if there’s any implications for nix tooling, but I noticed there’s been some funded work in pip that implemented a new resolver in case that isn’t already on people’s radar. Already the default in pip 20.3.
One in-progress follow-up PR that seemed like it could potentially be relevant for generating nix expressions more easily/robustly:
https://github.com/pypa/pip/pull/10748
In general I get the impression they’re trying to take metadata more seriously but I’m honestly too confused to tell exactly what is going on.
1 Like
Wow TIL, those virtual env hooks are gonna be pretty cool to play with. Thanks for making that video
1 Like
Yes, this is an important new feature and will also be used by Hatch for lock-file support. It removes the need to download or build wheels to determine the metadata for like 99% of the cases.
It is my intention to also improve our Python package updater in Nixpkgs to utilize this metadata.
7 Likes
I think Devbox, together with Nixpacks, are the first real tools to come out of Nix that give Nix-naive or Nix-averse users everything they need to leverage the power of Nix without once touching the Nix language. Devbox gives a discrete, portable, environment and it just works. If you want to then ship an OCI image, Devbox gives you that power.
with DevBox - is it possible to install all pypi packages (mach-nix like) or only nixpkgs python packages (if those are not broken)?
Should be. Both pip and poetry are supported for fetching packages.
?
✓ Successfully installed devbox 🚀
Next Steps
1. Learn how to use devbox
Run devbox help or read the docs at https://github.com/jetpack-io/devbox
2. Get help and give feedback
Join our community at https://discord.gg/agbskCJXk2
$ devbox help
devbox: command not found
> devbox help
Instant, easy, predictable shells and containers
Usage:
devbox [flags]
devbox [command]
Available Commands:
add Add a new package to your devbox
build Build an OCI image that can run as a container
completion Generate the autocompletion script for the specified shell
help Help about any command
init Initialize a directory as a devbox project
plan Preview the plan used to build your environment
rm Remove a package from your devbox
shell Start a new shell or run a command with access to your packages
version Print version information
Flags:
-h, --help help for devbox
Use "devbox [command] --help" for more information about a command.
I found this to be a quick general solution, but not quite what OP wants. spacy-wordnet is not available in nixpkgs.
If you want to bundle python and some python module, such as requests, with it, we can do this.
configuration.nix
environment.systemPackages = with pkgs; [
(python311Packages.python.withPackages (p: [ p.requests ]))
];
withPackages attribute isn’t implemented for all nixpkgs though, this is python311Packages specific. Search python311Packages.<module-name> to see if some python module is available in nixpkgs.
Yeah, did you manage to find a solution that works for you?
I am currently trying to figure this out as a noob.
After trying many alternatives out there, I think the 2 most satisfying ones are dream2nix and pyproject.nix.
The 1st one is if you want to give priority to packages built by pypi.org. The 2nd one, if you want to prioritize packages built by nixpkgs.
That said, if you don’t want to learn the nix language, you should try devenv.sh. But learning it is very worth it.
5 Likes
Yes, I use python3Packages directly without any pypi integrations (no poetry2nix, no dream2nix, not anything else) and without conda or pip. When a package I need is absent in Nixpkgs I just nix-init it locally. I’m fairly happy with this solution.
The only obvious direction I see where this workflow could be improved is on the upstream’s side: I’d wish to support isolating transitive dependencies, e.g. s.t. one could include a package that depends on pydantic_1 without breaking the rest of the “environment” which depends on pydantic (v2), or s.t. one could let tensorflow depend on an older version of protobuf. Ignoring potential conflicts in the python interpreter’s address space, this is a choice and an inherent limitation of python’s import system which we can’t do much about without upstream changes.
3 Likes
I just run Debian on Distrobox now; this way I can learn Nix without a sword of Damocles over my head.
Ryan points us to Luc Perkins in this matter.
Python virtual environments rely on symlinks and environment variables manipulation, and the package you want to install depends on system libraries that have opinionated expectations for how LD_LIBRARY_PATH should be configured to access their runtime. These two don’t play well with nix store.
The idiomatic way would be to install both Python and C/C++ dependencies from nixpkgs. This is not always possible, or sometimes one might not want to rewrite requirement files to nix.
I have a flake template that I use to setup impure development environments for this type of projects.
It’s super fiddly, but satisfies my needs - which mostly are to create a virtual env and install deps from a requirements file (without bothering with containers or virtualization).
For this to work you’ll need to have flakes enabled in your system.
You could give it a try it with:
$ mkdir /tmp/project
$ cd /tmp/project
$ nix flake init --refresh --template github:gmodena/flake-templates#python
nix flake init will download flake.nix and requirements.txt template files. flake.nix declares the development environment (you can think of it sort of like a container).
You’ll need to customize it a bit to satisfy your project’s needs.
I gave build123d a go, and added these deps to systemPackages
in flake.nix:
...
systemPackages = with pkgs; [
taglib
openssl
git
libxml2
libxslt
libzip
zlib
stdenv.cc.cc.lib
libGL # required by build123d
xorg.libX11 # required by build123d
expat # required by build123d
];
...
Then I added the Python deps to requirements.txt
$ echo build123d >> requirements.txt
Finally, I created a development environment with
$ nix develop
If all works, nix develop should drop you in a shell with an active virtual environment (located at .venv).
$ which python
/home/gmodena/tmp/project/.venv/bin/python
$ python -c "from build123d import *"
This template thingy is not guaranteed to work (current state of testing is “works on my computer”), but I hope it helps 
5 Likes