Currently some python packages require specific versions of other packages and overrides have to be created or some packages simply don’t exist and have to be added which adds significant overhead if packaging more complicated applications that use more uncommon packages, especially if most of those packages could’ve just aswell been generated automatically
Then there’s ruby’s approach for packaging where a set of overrides per package are definied globally with most of the common packages being handled automatically by the packaging tools and those that do need special tweaks get them delivered through the global overrides
It would be a lot easier if one could just “pypi2nix” by default and have those unadded packages be pulled in automatically, most of them might not need any crazy override logic besides the default python packaging scripts anyways.
Currently there exists a repo that just pulls the entirety of pypi into a nix expression which could additionally ease things and make this process not even require a special tool (but pulling in that large expression might cause problems of it’s own)
I recently started dream2nix which is a generic implementation of several concepts which I think are required for more scalable packaging workflows. @mkg20001 It basically covers all the points you mentioned and more.
In dream2nix individual packages manage their dependencies in isolation by default.
Dependency sharing between packages is considered an optimization, but not a hard requirement. This minimizes conflicts between packages and enables simpler workflows and automation.
Dream2nix also strictly separates:
generic logic (builders)
package specific logic (overrides)
data (generic lock)
It is done in a way so it would be compatible for integration into nixpkgs in the future.
The python subsystem of dream2nix did not get much attention yet, as I’m personally focusing on nodejs right now.
Currently there exists a repo that just pulls the entirety of pypi into a nix expression which could additionally ease things and make this process not even require a special tool (but pulling in that large expression might cause problems of it’s own)
This is what currently happens in pypi-deps-db, providing the basis for mach-nix.
The problem is that this is not implemented in a way compatible with nixpkgs.
Dream2nix tries to fix that.
As already mentioned, poetry2nix is a great option but it restricts in using a specific lock file format.
This is a addressed by dream2nix as well. The parsing/translation of metadata is decoupled from other parts, so that it can potentially support all metadata formats. It provides a modular system to attach translators for such metadata.
For example poetry2nix could be integrated as such a translator module for translating poetry lock files.
I am hoping for collaboration on this, as I cannot possibly maintain modules for all languages / build systems myself.
The general idea behind dream2nix is that instead of creating yet another tool for packaging language XY or lock-file format XY, we can focus efforts in one place, bringing scalable concepts to all build systems at once and not having to re-implement those over and over again.