Practically speaking, is there specific things pip could change to make nix packaging easier (if we were pip maintainers what would we do)?
Issue warnings about old/bad practices perhaps? Things of that nature?
That pretty much summarizes my experience with conda as well. They use an SMT solver to solve a constraint programming problem but the versioning conventions they use (upstream version, conda channel, build properties like with/without MKL, etc.) have some very rough edges. The nightmare ended when I found Nix a few years ago…
2 Likes
Poetry does already a good job fixing some of the problems like install order. If pip was replaced with poetry we would already gain a lot.
Other than that they should have ditched that you can execute code to generate package metadata and enforce setup.cfg.
1 Like
Other aspect I think people were complaining about is that poetry itself depends on non-stdlib packages, which makes it really hard to bootstrap, at least without a good standardized way of getting the basic packages in place.
I imagine all distros have their own hacks for that by now, and nix needs to reimplement poetry’s parsing anyway, but it still is a bit of a problem.
I understand that conda has its issues, this is a big reason as to why e.g. mamba exists. I can only recommend to try it out. Also of course, conda can largely be used only to manage binary software that python packages need and the python packages can be installed with e.g. pip. Then conda becomes more like a venv, with easy support for binary dependencies.
Conda certainly isn’t perfect. But for most people, it is a lot easier to learn and use than nix. I really like nix, and it is a great inspiration on how it does things … but for most people in this space I know, conda is just a lot simpler, despite its rough edges.
Furthermore, when it comes to the solver, conda may have its issues, but so does poetry. I used poetry for 2 years, and recently gave up on it again. It happened to me more than once that the solver tried for 1 hour or more to find a solution for the lockfile (and didn’t find one), and that in instances where just months before it worked. Also, adding a package and finding other packages downgraded was a common occurence as well. As the poetry lockfile (as I understand it) is supposed to work for all python versions simultaneously, a new released python version alone can be a real headache.
Long story short - I don’t think there is a perfect solution here, just tradeoffs.
I got a video on this, I skipped to the point where I just create a venv leveraging nix: https://youtu.be/jXd-hkP4xnU?t=1295
3 Likes
I don’t know if there’s any implications for nix tooling, but I noticed there’s been some funded work in pip that implemented a new resolver in case that isn’t already on people’s radar. Already the default in pip 20.3.
One in-progress follow-up PR that seemed like it could potentially be relevant for generating nix expressions more easily/robustly:
https://github.com/pypa/pip/pull/10748
In general I get the impression they’re trying to take metadata more seriously but I’m honestly too confused to tell exactly what is going on.
1 Like
Wow TIL, those virtual env hooks are gonna be pretty cool to play with. Thanks for making that video
1 Like
Yes, this is an important new feature and will also be used by Hatch for lock-file support. It removes the need to download or build wheels to determine the metadata for like 99% of the cases.
It is my intention to also improve our Python package updater in Nixpkgs to utilize this metadata.
7 Likes
I think Devbox, together with Nixpacks, are the first real tools to come out of Nix that give Nix-naive or Nix-averse users everything they need to leverage the power of Nix without once touching the Nix language. Devbox gives a discrete, portable, environment and it just works. If you want to then ship an OCI image, Devbox gives you that power.
with DevBox - is it possible to install all pypi packages (mach-nix like) or only nixpkgs python packages (if those are not broken)?
?
✓ Successfully installed devbox 🚀
Next Steps
1. Learn how to use devbox
Run devbox help or read the docs at https://github.com/jetpack-io/devbox
2. Get help and give feedback
Join our community at https://discord.gg/agbskCJXk2
$ devbox help
devbox: command not found
> devbox help
Instant, easy, predictable shells and containers
Usage:
devbox [flags]
devbox [command]
Available Commands:
add Add a new package to your devbox
build Build an OCI image that can run as a container
completion Generate the autocompletion script for the specified shell
help Help about any command
init Initialize a directory as a devbox project
plan Preview the plan used to build your environment
rm Remove a package from your devbox
shell Start a new shell or run a command with access to your packages
version Print version information
Flags:
-h, --help help for devbox
Use "devbox [command] --help" for more information about a command.
I found this to be a quick general solution, but not quite what OP wants. spacy-wordnet is not available in nixpkgs.
If you want to bundle python and some python module, such as requests, with it, we can do this.
configuration.nix
environment.systemPackages = with pkgs; [
(python311Packages.python.withPackages (p: [ p.requests ]))
];
withPackages attribute isn’t implemented for all nixpkgs though, this is python311Packages specific. Search python311Packages.<module-name> to see if some python module is available in nixpkgs.