Yes, you are right. pip can see those Nix packages, but unfortunately according my testing Poetry can’t and will install them again.
What’s the reason why you’d like to combine the two? Can’t everything be built using Nix or Poetry?
Yeah you might be right about this one. However that might be more an issue with poetry than with nix. I Personally always reinstall everything in venvs with poetry so I never had this issue before. And I use nix primarily as a replacement for something like pyenv.
But check this out:
The reason is that sometimes it is easier/faster/more convenient to install Python packages with C extensions from nixpkgs then from PyPi. And also, I am producing some custom Python packages with heavy C dependencies from my other Flake, so I want to install them from there.
@APCodes , thank you very much for pointing this out ! I am quite surprised that such a problem wasn’t fixed since then.
Hi, do you have plan for do the same thing with poetry like python.venv.enable, there are many problems when trying to install packages with poetry, like this and this, and I have to switch to use packages from ‘python3.withPackages’
I ran into this too. The problem is in the way the python interpreter packages are built. This PR fixes it.
6 Likes
For reference, 297628 is the PR from @cwp that was ultimately merged.
Update: see below for the link to the reversion of 297628 in 302385. Many thanks to @ruro and @chrism for identifying this.
1 Like
@cameronraysmith AFAIK, the PR you linked got reverted almost immediately, because there were some unexpected issues with it and there wasn’t enough time to fix it before ZHF. I am hopeful that it’ll get fixed eventually since the main idea of the fix was fine, the problem was mostly in the implementation details.
Oh ok thanks for clarifying. I updated my comment above to point to @chrism’s identification of the reversion below to minimize the potential for my original comment alone to create “mid-thread confusion” regarding the status of this work.
Ugh. This fixed so many problems. I’d be happy to help fix whatever is holding this back, if someone points me in the right direction. Or even a link to its reversion.
@chrism here is my current understanding.
Currently (without the PR applied), nix creates python package environments in such a way that they aren’t automatically picked up by the python interpreter. Instead, you have to do some ugly hacks using sitepackages.py and wrapper scripts that modify sys.argv[0] and put individual paths into site in order to ensure that the required packages are picked up at runtime.
PR #297628 basically tries to remove these hacks and instead make python just detect all the required packages like it would in a normal global python install. Unfortunately, the code in the PR tried to “unwrap” the already wrapped python scripts by just assuming that the third line in the wrapper would always be the line that modifies sys.argv[0] and site.
In reality, however, some wrappers may include extra lines (like a # -*- coding: utf-8 -*- comment, for example), which would make the “unwrapper” delete the wrong line.
A super simple, but dirty fix would be to match the lines that we want to delete by their content instead of by line number. So instead of
sed -e '1d' -e '3d' ".$prg-wrapped" >> "$out/bin/$prg"
we could do this
sed \
-e '/^#!\/nix\/store\//d' \
-e '/^import sys;import site;import functools;sys\.argv\[0\] = /d' \
".$prg-wrapped" >> "$out/bin/$prg"
This should be more robust to weird wrapper variations by only deleting the lines that start with #!/nix/store/ and import sys;import site;import functools;sys.argv[0] = instead of blindly expecting them to be the first and third line. (note: I haven’t tested this thoroughly, so there might still be some edge cases that this fix misses)
A more “proper” solution would probably be to avoid wrapping and unwrapping these scripts in the first place. In fact, in the original PR, @cwp mentioned that they had an idea regarding that. But “when the world needed him most, he vanished” and he hasn’t responded to my comments, so I am honestly not sure what was his idea.
4 Likes
Thank you, what a wonderful summary, It would be best to remove the need to do the sed at all.
I’ve reviewed Revert "Unwrap python scripts when building an environment" by SuperSandro2000 · Pull Request #302385 · NixOS/nixpkgs · GitHub and I’m going go try to identify and fix the actually impacted packages.
2 Likes
Thank you very much for this great summary. I very much suggest to copy this information to GitHub PR as well.
1 Like
Hi folks,
Sorry for the long absence. I’ve been distracted and unfortunately neglected some things.
I’d like to try again with this change, but it’s obviously trickier than I originally thought. There are packages that depend on the current behaviour. :-/
The idea @ruro mentions above is fairly simple: instead of prefixing scripts with code to manipulate sys.path, we just rewrite the shebang to point to a symlink to the interpreter. Essentially, each package comes with a mini env for running its scripts. When packages depend on other packages, the envs get composed as usual. It’s simple, but invasive, so it’d take careful work to integrate it without a lot of breakage.
Then after that, the withPackages fix should be easier.
3 Likes
I’d like this to be fixed, since devenv-nixpkgs is stuck due to this being reverted.
Shall we work together @cwp @chrism to get a set of tests and experiment on fixing it?
1 Like
Yeah, I’m down.
What sort of test do you have in mind? Ideally we’d have a test that runs all the scripts that are included with a python package, but I don’t know how “runs correctly” would be defined for such a large and diverse set of scripts.
-cwp
@cwp I cherry-picked your commits on top of master as of 35a90c3f3dea9ae07f3cc70691f04fb4c9beb98c in GitHub - mcdonc/nixpkgs at python-env-venv and me and @domenkozar I think are going to try to figure out how to get reports from hydra runs before and after, to see what the actual breakage is, and then maybe we’ll have a better idea about what needs to be done. I think this is a reasonable step, anyway 
1 Like