Nix-shell + buildPythonPackage + pyproject.toml does not seem to give editable installation correctly

fzakaria · August 1, 2023, 4:52pm

hello!

I’m having trouble getting a nix-shell with buildPythonPackage to recognize my Python module. Could someone help me debug it?

It looks to be a problem with the way it’s setting up the editable installation but my Python-fu is moderate.
My derivation (derivation.nix) is pretty simple:

# simplified to do an import from channel
let
  pkgs = import <nixpkgs> { };
in
with pkgs;
python3.pkgs.buildPythonPackage rec {
  name = "sqlelf";
  src = pkgs.nix-gitignore.gitignoreSource [ ] ./.;

  format = "pyproject";

  propagatedBuildInputs = with python3.pkgs; [
    setuptools
  ];
}

pyproject.toml:

[project]
name = "sqlelf"
version = "0.0.1"
requires-python = ">=3.10"
dependencies = [
]

[project.scripts]
sqlelf = "sqlelf.cli:start"

[tool.setuptools]
packages = ["sqlelf", "sqlelf.elf"]

When I run the script though it gives me “module not found” for the Python module

[nix-shell:~/code/github.com/fzakaria/sqlelf]$ sqlelf
Traceback (most recent call last):
  File "/run/user/780412/tmp.AReR3WtYpL/bin/sqlelf", line 5, in <module>
    from sqlelf.cli import start
ModuleNotFoundError: No module named 'sqlelf'

Doing nix-build derivation.nix works fine though which is confusing.

❯ nix-build derivation.nix
/nix/store/8j08phl545hc1cr3n53yh65230x5q1xm-python3.10-sqlelf
❯ ./result/bin/sqlelf
usage: sqlelf [-h] [-s SQL] FILE [FILE ...]
sqlelf: error: the following arguments are required: FILE

That makes me think the modules and source layout are correct so not sure why the nix-shell variant can’t find it.

firecat53 · August 1, 2023, 10:03pm

Have you tried with buildPythonApplication instead? My understanding is that should be used if there is a CLI component to your package.

Edit: Nix Manual - python

fzakaria · August 2, 2023, 3:33am

Same problem :S unfortunately even with buildPythonApplication

con-f-use · August 4, 2023, 9:30pm

My hunch would be that is a problem with your project (and python’s sometimes weird import behavior in edge-cases) in combination with nix, rather than with nix itself. Can you make a complete minimal example, i.e. including directory structure and python file(s)?

fzakaria · August 4, 2023, 9:59pm

Checkout GitHub - fzakaria/no-frills-nix-python-template: A simple Nix template for building and distributing Python applications

I try nix-shell derivation.nix and it looks like the module app should be installed in an editable fashion but doesn’t seem to work.

Here is me checking that there seems to be some site_packages that look like it’s trying to install it editably

[nix-shell:~/code/github.com/fzakaria/no-frills-nix-python-template]$ python
Python 3.10.11 (main, Apr  4 2023, 22:10:32) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/run/user/780412/tmp.aJmDO8Lmrx/lib/python3.10/site-packages', '/usr/local/buildtools/current/sitecustomize', '/nix/store/6jx4nc20v3cfphlbypqid30q41i5vh5x-python3-3.10.11/lib/python3.10/site-packages', '/nix/store/02aayiqjzhzwlxg32rn4408cp15ihjh6-python3.10-pip-23.0.1/lib/python3.10/site-packages', '/nix/store/j2h8wc4fd2j4gllrvxjx7kr2kq54m1y9-python3.10-wheel-0.38.4/lib/python3.10/site-packages', '/nix/store/j3h2j3hl4nqzdg7j37ixz8p8v109dkg6-python3.10-setuptools-67.4.0/lib/python3.10/site-packages', '/nix/store/6jx4nc20v3cfphlbypqid30q41i5vh5x-python3-3.10.11/lib/python310.zip', '/nix/store/6jx4nc20v3cfphlbypqid30q41i5vh5x-python3-3.10.11/lib/python3.10', '/nix/store/6jx4nc20v3cfphlbypqid30q41i5vh5x-python3-3.10.11/lib/python3.10/lib-dynload']
>>> 

[nix-shell:~/code/github.com/fzakaria/no-frills-nix-python-template]$ ls /run/user/780412/tmp.aJmDO8Lmrx/lib/python3.10/site-packages
__editable__.no_frills_nix_python_template-0.0.1.pth        __pycache__
__editable___no_frills_nix_python_template_0_0_1_finder.py  no_frills_nix_python_template-0.0.1.dist-info

Here it is failing:

[nix-shell:~/code/github.com/fzakaria/no-frills-nix-python-template]$ app
Traceback (most recent call last):
  File "/run/user/780412/tmp.E7FOrYN7JC/bin/app", line 5, in <module>
    from app.cli import start
ModuleNotFoundError: No module named 'app'

Doing a nix-build is fine though.

❯ nix-build derivation.nix
/nix/store/5l0amk2f0wm7qa4q6rk3izicc9d9315a-python3.10-no-frills-nix-python-template
❯ ./result/bin/app
Goodbye!

con-f-use · August 5, 2023, 9:26pm

Okay, one thing I can immediately see is, that you shouldn’t put the source directly in the project root directory where the pyproject.toml is. Because ./ is always in sys.path, so that will mask any import issues you might have, by virtue of import app working all the time, when app/ is in the current working directory, even if the package is not installed.
I.e. possible import errors will only show up when actually installing your package properly while simultaneously not having the project root as your working dir.
It’s not a problem here, but generally good practice.

But yeah, it seems line nix should be patching the resulting shell script from the editable install and isn’t or something.

con-f-use · August 5, 2023, 10:29pm

On a fresh checkout of your repo:

But that should never be needed with a properly editable installed package. When you run it in a python interpreter the cwd is in sys.path by default, when you run it via the app command (script generated by the installer, it usually not. That’s why it works via the interperter, and not via the app. And that is just pure coincidence because the usual import mechanism is bypassed then.

fzakaria · August 5, 2023, 10:44pm

Yea if you just add that current directory to PYTHONPATH that defeats the purpose of the editable installation

con-f-use · August 5, 2023, 11:10pm

I find this really strange, because, if I hack my now working venv (no nix) install to show me sys.path like this:

cat .venv/bin/app 
#!/home/confus/devel/python/no-frills-nix-python-template/.venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys
print("\n".join(sys.path))  # <---  HACKED
from app.cli import start
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start())

then I get:

'/home/confus/devel/python/no-frills-nix-python-template/.venv/bin'
'/nix/store/j1c268l85zbpdgbl5wlg0zc73zrfngvq-xonsh-0.14.0/lib/python3.10/site-packages'  # <-- contains the `.pth` file  which seems to be ignored
'/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/lib/python310.zip'
'/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/lib/python3.10'
'/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/lib/python3.10/lib-dynload'
'/home/confus/devel/python/no-frills-nix-python-template/.venv/lib/python3.10/site-packages'
'/home/confus/devel/python/no-frills-nix-python-template/src'  # <-- this makes importing work nonetheless
Goodbye!

but if I do the same in the nix shell, it’s:

[nix-shell:~/devel/python/no-frills-nix-python-template]$ which app
/run/user/1000/tmp.N5Nt9mMfSr/bin/app

[nix-shell:~/devel/python/no-frills-nix-python-template]$ cat /run/user/1000/tmp.N5Nt9mMfSr/bin/app
#!/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/bin/python3.10
# -*- coding: utf-8 -*-
import re
import sys
print("\n".join(sys.path))  # <---- likewise HACKED
import app
from app.cli import start
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start())

[nix-shell:~/devel/python/no-frills-nix-python-template]$ app
'/run/user/1000/tmp.N5Nt9mMfSr/bin'
'/run/user/1000/tmp.N5Nt9mMfSr/lib/python3.10/site-packages'  # <-- contains the `.pth` file  which seems to be ignored
'/nix/store/j1c268l85zbpdgbl5wlg0zc73zrfngvq-xonsh-0.14.0/lib/python3.10/site-packages'
'/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/lib/python3.10/site-packages'
'/nix/store/66qrgkmha6dmcac29nvxbjsbrh77jlin-python3.10-pip-23.0.1/lib/python3.10/site-packages'
'/nix/store/fj028lay9j9bsj58564bznjviqa4aapk-python3.10-wheel-0.38.4/lib/python3.10/site-packages'
'/nix/store/cr689gv3clfqhbl05pfqd9g9khzdidcb-python3.10-setuptools-67.4.0/lib/python3.10/site-packages'
'/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/lib/python310.zip'
'/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/lib/python3.10'
'/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/lib/python3.10/lib-dynload'
Traceback (most recent call last):
  File "/run/user/1000/tmp.N5Nt9mMfSr/bin/app", line 6, in <module>
    import app
ModuleNotFoundError: No module named 'app'

The sys.path in the two cases are different. So naturally, you’d think “there must be the problem, the .pth files must differ”, but when I diff the .pth files that are under sys.path for both, they are identical and link to the source directory:


[nix-shell:~/devel/python/no-frills-nix-python-template]$ diff \
  --report-identical-files \
  /run/user/1000/tmp.N5Nt9mMfSr/lib/python3.10/site-packages/__editable__.no_frills_nix_python_template-0.0.1.pth .venv/lib/python3.10/site-packages/__editable__.no_frills_nix_python_template-0.0.1.pth 

Files /run/user/1000/tmp.N5Nt9mMfSr/lib/python3.10/site-packages/__editable__.no_frills_nix_python_template-0.0.1.pth and .venv/lib/python3.10/site-packages/__editable__.no_frills_nix_python_template-0.0.1.pth are identical

So there’s no reason for that import to fail in one case, but not the other, damn it! Setuptools switched to PEP 660 edtiable installs a while ago, so that might be related. There’s legacy options.

con-f-use · August 5, 2023, 11:23pm

Something in our nix-shell python seems to ignore the .pth-files or not want to import from them:

>>> sys.executable
'/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/bin/python'
>>> sys.path = ['/run/user/1000/tmp.N5Nt9mMfSr/lib/python3.10/site-packages']
>>> import app
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'app'
>>> sys.path = ['/home/confus/devel/python/no-frills-nix-python-template/.venv/lib/python3.10/site-packages/']
>>> import app
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'app'
>>> sys.path = ['/home/confus/devel/python/no-frills-nix-python-template/src/']
>>> import app

While it doesn’t in a standard virtualenv and the issue was just maked by cwd in sys.path in some cases and app being directly in the project root. Because in your OP, result/bin/sqelf was probably wrapped and the path being set in the wrapper:

[nix-shell:~/devel/python/no-frills-nix-python-template]$ ll ./result/bin/
total 26
dr-xr-xr-x 2 root root   4 Jan  1  1970 .
dr-xr-xr-x 5 root root   5 Jan  1  1970 ..
-r-xr-xr-x 2 root root 715 Jan  1  1970 .app-wrapped
-r-xr-xr-x 2 root root 751 Jan  1  1970 app

You ran two completely different sqlelf in nix shell vs. ./result/bin/sqlelf !

con-f-use · August 6, 2023, 11:12am

Yet another obeservation. Again bypassing nix-shell and using a regular virtualenv for this, I noticed that just swapping the interpreter path in the shebang of your app script, has influence on sys.path and the import success:

Python simply has way too much magic for my taste! And mind you, the one in .venv/ is just a symlink to the absolute path, I used, so the environment and everything else is the same. It’s just python magic in the interpreter!

$ realpath /home/jan/devel/python/no-frills-nix-python-template/.venv/bin/python
/nix/store/1r6n7v2wam7gkr18gxccpg7p5ywgw551-python3-3.10.12/bin/python3.10

But the core problem remains, the .pth file in $tmp_path/<...>/site-packages is just ignored, when it shouldn’t be. I opened an issue in nixpkgs: python interpreter ignores `.pth` in certain circumstances on current unstable · Issue #247555 · NixOS/nixpkgs · GitHub
Hope it isn’t something in CPython in general.

fzakaria · August 6, 2023, 9:59pm

Thank you digging into this one with me – I learned a lot through the process.
I will be following along the issue to see what may have caused it.

I agree felt very suspicious and “magic” – everything looked OK on the outside

FRidh · August 7, 2023, 7:10am

It’s ignored because when using nix-shell you use PYTHONPATH and .pth are not recursed into through PYTHONPATH. For that, the folder containing the .pth needs to be part of a site directory (see site.addsitedir. We could try changing to NIX_PYTHONPATH instead which uses site to process.

con-f-use · August 7, 2023, 8:59am

Thanks @FRidh for the attention! Let’s discuss that in the github issue, so we don’t have to split our attention and run the risk of missing some conversation.

pbsds · October 9, 2023, 8:36pm

For those who land here via google, this is my current flake work-around to get scripts working, assuming the pyproject builder is poetry

devShells.${system}.default = self.packages.${system}.default.overridePythonAttrs (base: {
  # https://github.com/NixOS/nixpkgs/issues/247555
  nativeBuildInputs = base.nativeBuildInputs ++ pkgs.lib.mapAttrsToList (name: value:
      pkgs.writers.writePython3Bin name { flakeIgnore = [ "E401" "E501" ]; } ''
        import sys, importlib
        mod, attr = "${value}".split(":", 1)
        sys.exit(getattr(importlib.import_module(mod), attr)())
      ''
    ) (builtins.fromTOML (builtins.readFile ./pyproject.toml)).tool.poetry.scripts;
  shellHook = ''
    export PYTHONPATH="$(pwd):$PYTHONPATH"
  '';
});

phaer · March 29, 2024, 5:54pm

EDIT: I misremembered: Not going through .pth files in PYTHONPATH seems to be standard behaviour, according to site — Site-specific configuration hook — Python 3.12.2 documentation (“and exists in one of the four directories mentioned above”). So it’s most probably a better idea to go through NIX_PYTHONPATH for that
~~Do you know whether that behavior is intentional?~~ As everything on PYTHONPATH is intended to end up into sys.path anyways and (c)python seems to ensure that there are no duplicates, couldn’t it be safe to use site.addsitedir on PYTHONPATH in the same way as we do for NIX_PYTHONPATH?

It did seem to work for me in a small example (see this comment python3Packages.pythonImportsCheckHook: doesn't work with .pth files · Issue #289340 · NixOS/nixpkgs · GitHub), PYTHONPATH stays intact (isn’t unset) and everything that would later be added would still be prepended to the list. Happy to open a PR on nixpkgs if there no obvious drawbacks. (would cost some cpu cycles due to python-rebuild, so would target staging), but it might not be that easy after all?