Sorry for the long absence. I’ve been distracted and unfortunately neglected some things.
I’d like to try again with this change, but it’s obviously trickier than I originally thought. There are packages that depend on the current behaviour. :-/
The idea @ruro mentions above is fairly simple: instead of prefixing scripts with code to manipulate sys.path, we just rewrite the shebang to point to a symlink to the interpreter. Essentially, each package comes with a mini env for running its scripts. When packages depend on other packages, the envs get composed as usual. It’s simple, but invasive, so it’d take careful work to integrate it without a lot of breakage.
Then after that, the withPackages fix should be easier.
What sort of test do you have in mind? Ideally we’d have a test that runs all the scripts that are included with a python package, but I don’t know how “runs correctly” would be defined for such a large and diverse set of scripts.
@cwp I cherry-picked your commits on top of master as of 35a90c3f3dea9ae07f3cc70691f04fb4c9beb98c in GitHub - mcdonc/nixpkgs at python-env-venv and me and @domenkozar I think are going to try to figure out how to get reports from hydra runs before and after, to see what the actual breakage is, and then maybe we’ll have a better idea about what needs to be done. I think this is a reasonable step, anyway