Trying to package a Python library with cythonized modules and have run into a bizzare problem. These modules can’t be imported at checkPhase (ModuleNotFoundError) in the test suite and manually, though they are perfectly importable at pythonImportChecks or in a nix shell. PYTHONPATH seems adequate.
Not sure what the root issue is, but it seems you have a typo, pythonImportChecks should be pythonImportsCheck which may explain why that “passes” when the check phase fails.
IIRC, cpythonized and native extensions will only appear in the installed site packages, so you need to do something to the affect of PYTHONPATH=$out/${python.sitePackages} for the native extensions to appear when the checkPhase runs.
You should see many examples in pkgs/development/python-modules where cython is used with checkphases
Thanks for the correction! Unfortunately, it didn’t help to explain the problem, at pythonImportsCheckPhase modules do get imported (simplex_tree is a one of cythonized modules):
pythonImportsCheckPhase
Executing pythonImportsCheckPhase
Check whether the following modules can be imported: gudhi gudhi.simplex_tree
pytestcachePhase
/nix/store/2a5aldh7vkh77vhkm68v48jzd712wm3y-python3.8-gudhi-3.4.1
Yes, that was my first idea too. But (1) the package gets properly installed at $out/${python.sitePackages}; (2) this path does appear in PYTHONPATH by default at check phase. Having the following diagnostics code at checkPhase:
Python 3.8.9
1.20.2
/nix/store/5iaqzmg7948ngl18n04mkh83ldv41bak-python3.8-gudhi-3.4.1/lib/python3.8/site-packages
/nix/store/5iaqzmg7948ngl18n04mkh83ldv41bak-python3.8-gudhi-3.4.1/lib/python3.8/site-packages:/nix/store/zv167zw7gwr9h0gpkfcikdfqnscrd0q6-python3-3.8.9/lib/python3.8/site-packages:/nix/store/d8j11v90ms2ymr8m770bfm31jm6zj2a6-python3.8-setuptools-54.2.0/lib/python3.8/site-packages:/nix/store/f1y3ykg4fjyrgns39j3m71w2py4m15yx-python3.8-wheel-0.36.2/lib/python3.8/site-packages:/nix/store/99f478yfy8jfilmrvmfcan7ldlv1nlk6-python3.8-pip-21.0.1/lib/python3.8/site-packages:/nix/store/ys7ffnghl2iaza8ly4mjpycfyargxv4z-python3.8-numpy-1.20.2/lib/python3.8/site-packages:/nix/store/klp1isshlzhpmnwdlxl9k0xw9zlbwkmq-python3.8-Cython-0.29.22/lib/python3.8/site-packages:/nix/store/28qrb63b4yiflv3bwhdp0sd548xygqz0-python3.8-pybind11-2.6.2/lib/python3.8/site-packages:/nix/store/ym2rmn08956qd5gkkkvhd119rfr5iqhn-python3.8-pytest-6.2.3/lib/python3.8/site-packages:/nix/store/1h74vgsxwfraq8jzv1553q0a3r13fnrz-python3.8-atomicwrites-1.4.0/lib/python3.8/site-packages:/nix/store/i3fgay5f8a9b22h94pmw7l2jy7iqk5pg-python3.8-attrs-20.3.0/lib/python3.8/site-packages:/nix/store/mk373mg1cywf1w7h9ns3gh9yld9indr7-python3.8-iniconfig-1.1.1/lib/python3.8/site-packages:/nix/store/d6c9596m7dx9d591aw5ain39hy0w439a-python3.8-more-itertools-8.7.0/lib/python3.8/site-packages:/nix/store/mah3d3892jir9xji04fy7wwyiaxzlaja-python3.8-six-1.15.0/lib/python3.8/site-packages:/nix/store/qsnkkqbyhckmh21x5vxh5svamwlp2wgf-python3.8-packaging-20.9/lib/python3.8/site-packages:/nix/store/5cd05wizlz87ynkwkvkk4rs1dzxfbpzq-python3.8-pyparsing-2.4.7/lib/python3.8/site-packages:/nix/store/ardnzjn9836rx8bw0n7c0sc856s7dgl5-python3.8-pluggy-0.13.1/lib/python3.8/site-packages:/nix/store/d4lx9h530kwji9cld8r8mgfm9ka7p896-python3.8-importlib-metadata-3.7.3/lib/python3.8/site-packages:/nix/store/mhnp0npp4zzwasa1yn30naj66wkx4dnh-python3.8-toml-0.10.2/lib/python3.8/site-packages:/nix/store/b0hhgkf25x2mg614a7x8b8ycsfj72nng-python3.8-zipp-3.4.1/lib/python3.8/site-packages:/nix/store/9m984jllp5cjdkb2g4p74l6fj8icf0x7-python3.8-py-1.10.0/lib/python3.8/site-packages:/nix/store/m3yqk753ihv5fihcw2865x07f0q12afy-python3.8-wcwidth-0.2.5/lib/python3.8/site-packages:/nix/store/3s6nsn23f110s88kd084q28ssscqdvzn-python3.8-scipy-1.6.1/lib/python3.8/site-packages
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'gudhi.simplex_tree'
I also looked at other packages, and it seems that Cython should work out of the box. E.g. python-modules/tesserocr has cythonized modules and tests without an explicit checkPhase, python-modules/tensorflow does have a custom checkPhase, but it looks like additional environment manipulations aren’t needed.
I’ve created a mock project with a Cython module, and it seems that I’ve localized the problem! Although PYTHONPATH was set correctly, there still was gudhi package in the build directory which was the current working directory for the shell. Python prioritizes current directory for imports (sys.path[0] == "" normally), and GUDHI uses absolute imports in its top-level __init__.py, so the package left after building gets imported, not the one from $out/${python.sitePackages}. To keep ctest happy as CTestTestfile.cmake is still in the build dir, I just added rm -r gudhi to the top of checkPhase to make Python use the first PYTHONPATH entry. Otherwise something like cd .. might work too.
After some additional patching in CMakeLists.txt I got the package built with all tests passing! Thanks to everybody for help!
oh yea, I forgot to mention that was well. It’s common to delete or move the local "<module>" directory, or cd into the test directory. This avoids python’s default behavior of adding $PWD to sys.path