Why is it so hard to use a Python package?

Practically speaking, is there specific things pip could change to make nix packaging easier (if we were pip maintainers what would we do)?

Issue warnings about old/bad practices perhaps? Things of that nature?

That pretty much summarizes my experience with conda as well. They use an SMT solver to solve a constraint programming problem but the versioning conventions they use (upstream version, conda channel, build properties like with/without MKL, etc.) have some very rough edges. The nightmare ended when I found Nix a few years ago…

2 Likes

Poetry does already a good job fixing some of the problems like install order. If pip was replaced with poetry we would already gain a lot.

Other than that they should have ditched that you can execute code to generate package metadata and enforce setup.cfg.

1 Like

Other aspect I think people were complaining about is that poetry itself depends on non-stdlib packages, which makes it really hard to bootstrap, at least without a good standardized way of getting the basic packages in place.

I imagine all distros have their own hacks for that by now, and nix needs to reimplement poetry’s parsing anyway, but it still is a bit of a problem.

I understand that conda has its issues, this is a big reason as to why e.g. mamba exists. I can only recommend to try it out. Also of course, conda can largely be used only to manage binary software that python packages need and the python packages can be installed with e.g. pip. Then conda becomes more like a venv, with easy support for binary dependencies.

Conda certainly isn’t perfect. But for most people, it is a lot easier to learn and use than nix. I really like nix, and it is a great inspiration on how it does things … but for most people in this space I know, conda is just a lot simpler, despite its rough edges.

Furthermore, when it comes to the solver, conda may have its issues, but so does poetry. I used poetry for 2 years, and recently gave up on it again. It happened to me more than once that the solver tried for 1 hour or more to find a solution for the lockfile (and didn’t find one), and that in instances where just months before it worked. Also, adding a package and finding other packages downgraded was a common occurence as well. As the poetry lockfile (as I understand it) is supposed to work for all python versions simultaneously, a new released python version alone can be a real headache.

Long story short - I don’t think there is a perfect solution here, just tradeoffs.

I got a video on this, I skipped to the point where I just create a venv leveraging nix: Nixpkgs - Python packaging, and development workflow. - YouTube

1 Like

I don’t know if there’s any implications for nix tooling, but I noticed there’s been some funded work in pip that implemented a new resolver in case that isn’t already on people’s radar. Already the default in pip 20.3.

One in-progress follow-up PR that seemed like it could potentially be relevant for generating nix expressions more easily/robustly:

In general I get the impression they’re trying to take metadata more seriously but I’m honestly too confused to tell exactly what is going on.

1 Like

Wow TIL, those virtual env hooks are gonna be pretty cool to play with. Thanks for making that video

1 Like

Yes, this is an important new feature and will also be used by Hatch for lock-file support. It removes the need to download or build wheels to determine the metadata for like 99% of the cases.

It is my intention to also improve our Python package updater in Nixpkgs to utilize this metadata.

5 Likes
Hosted by Flying Circus.