PYTHONPATH missing dependencies for 3.11 build

I’ve been trying to package an update (https://github.com/NixOS/nixpkgs/pull/247618). As part of it, I added a few dependencies for pyside6 in buildInputs. The whole PR works fine with python310, but it breaks with python311.

It seems that PYTHONPATH is missing build inputs when using python311, even on nixpkgs:master:

$ nix develop .#python310Packages.pyside6 --command bash -c 'echo -e ${PYTHONPATH//:/\\n}'
/nix/store/jhflvwr40xbb0xr6jx4311icp9cym1fp-python3-3.10.12/lib/python3.10/site-packages
/nix/store/4a7gi10k8ncxzinf2bc2rp88s1ww718f-libselinux-3.3/lib/python3.10/site-packages
/nix/store/k6p3j60xvvgkiv0prrrrck8kdlhdak0g-shiboken6-6.5.0/lib/python3.10/site-packages

$ nix develop .#python311Packages.pyside6 --command bash -c 'echo -e ${PYTHONPATH//:/\\n}'
/nix/store/4a7gi10k8ncxzinf2bc2rp88s1ww718f-libselinux-3.3/lib/python3.10/site-packages
/nix/store/jhflvwr40xbb0xr6jx4311icp9cym1fp-python3-3.10.12/lib/python3.10/site-packages

(It’s also strange that those are 3.10 paths, right?)

How should I proceed to unblock this update?

1 Like

Can reproduce it on unstable.

Edit: Curios I tried it with another package and there it does not happens - so maybe somehow derivation related/specific?

warning: Git tree '/home/sebtm/Projects/Nix/nixpkgs/nixos-unstable' is dirty
/nix/store/2c7sgx69p6mmp76cvmi5j6c72dj76jj8-python3-3.10.12/lib/python3.10/site-packages
/nix/store/qg7z9pysfnzbsbmlbp5wlk92bg7kd351-python3.10-ninja-1.11.1/lib/python3.10/site-packages
/nix/store/3ajl7wpq0bpjzfshhdjj0n96skc4jik2-python3.10-packaging-23.1/lib/python3.10/site-packages
/nix/store/hpcmd1iwlzfjhqy74jmb21yrij3x61w9-python3.10-setuptools-67.4.0/lib/python3.10/site-packages
/nix/store/b4rjpcf3gkdh0mlx9kclrrbw37yxsdb2-libselinux-3.3/lib/python3.10/site-packages
❯ nix develop .#python311Packages.shiboken6 --command bash -c 'echo -e ${PYTHONPATH//:/\\n}'
warning: Git tree '/home/sebtm/Projects/Nix/nixpkgs/nixos-unstable' is dirty
/nix/store/9hwaihb8rhlvq8s7xq11m204jp51h6xi-python3-3.11.4/lib/python3.11/site-packages
/nix/store/cf28nlan3009pyv17bdrwp76qfxspxh1-python3.11-ninja-1.11.1/lib/python3.11/site-packages
/nix/store/wyxd8p52mpbrk9ls0v97a2dgm29ybid5-python3.11-packaging-23.1/lib/python3.11/site-packages
/nix/store/9nk0lgmr38d7m04gs6hi19a2xyxxxz6k-python3.11-setuptools-67.4.0/lib/python3.11/site-packages

(also for python311Packages.jq see edits, but seems less relevant - but pyside6/shiboken6 is not build with buildPythonPackage like jq is)

Update: The issue was that some qt6 modules pull in python3, but that dependency isn’t overridden when using python311 because qt6 isn’t a Python package. Fixed it in this commit.

Cool, nice catch and thanks for looking into it again - I was on the right track but couldn’t figure it out. Can you shortly explain how you get on the right track?

Is there a reason you preferred a patch over the substituteInPlace even it should be relative temporary as the code got changed for pillow > 10.0.0 which seems not backwards compatible? Should we suggest this change upstream for this reason as well?

I knew the issue affected pyside6 but not every Python package. So, I started removing pieces of the pyside6 derivation until I narrowed down the issue to the optional qt6 modules. Then, I checked for any python references in those modules, and saw that they are pulling in python3 but that qt6 is not part of pythonPackages. So, I checked how this is handled for other packages, and replicated that pattern for qt6.

As for the patch, I prefer the patch because it will cause a build failure when the upstream is updated, signalling that the patch can be removed. We’ll see what reviewers think.

Thanks for the explanation, always interesting to see how others approach such issues :slightly_smiling_face:

I tried to verify if the fix also would make it easier to prepare streamdeck-ui for future python upgrades by replicating the behavior from:

  deepsecrets = callPackage ../tools/security/deepsecrets {
    python3 = python311;
  };

what would result in:

  streamdeck-ui = libsForQt5.callPackage ../applications/misc/streamdeck-ui {
    python3Packages = python311.pkgs;
  };

I changed the version to latest develop release requiring release requiring at least python3.11. But despite different try’s (but this one was the most promising/comprehensible I always end up with:

nix log /nix/store/7zx1ydskf5597973acs81ijzay9fgz74-streamdeck-ui-3.1.0-develop.2.drv
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/7zx1ydskf5597973acs81ijzay9fgz74-streamdeck-ui-3.1.0-develop.2.drv^*'
Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing pip-build-hook
Using pipBuildPhase
Using pipShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing python-catch-conflicts-hook.sh
@nix { "action": "setPhase", "phase": "qtPreHook" }
qtPreHook
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/hjic3ls7h9pwmlxxf8ncdismdgnb28zm-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/tests/test_stream_deck_monitor.py
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
applying patch /nix/store/svlxvifkwni08pblaiaphiirxrw272zn-update-pillow.patch
patching file pyproject.toml
Hunk #1 succeeded at 14 with fuzz 2.
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Executing pipBuildPhase
Creating a wheel...
/nix/store/9hwaihb8rhlvq8s7xq11m204jp51h6xi-python3-3.11.4/bin/python3.11: No module named pip

does this make sense for you or would you just wait until it’s a stable release and have a look then?
Should we maybe do the change of qt-python before as it triggers far more rebuilds?