Can't import cythonized modules at checkPhase

Help
1

Trying to package a Python library with cythonized modules and have run into a bizzare problem. These modules can’t be imported at checkPhase (ModuleNotFoundError) in the test suite and manually, though they are perfectly importable at pythonImportChecks or in a nix shell. PYTHONPATH seems adequate.

Here is default.nix I was using: http://pastie.org/p/1JUbt5C9FnIZZTy9JSqKw6

Build command: nix-build --keep-failed -E 'with import <nixpkgs> {}; with python3.pkgs; callPackage ./. {}'

Shell command (with checks turned off): nix-shell -E 'with import <nixpkgs> {}; (let gudhi = with python3.pkgs; callPackage ./. {}; in python3.withPackages(ps: with ps; [ps.ipython gudhi])).env'

How to fix this problem and why has it arisen?

2

Not sure what the root issue is, but it seems you have a typo, pythonImportChecks should be pythonImportsCheck which may explain why that “passes” when the check phase fails.

3

IIRC, cpythonized and native extensions will only appear in the installed site packages, so you need to do something to the affect of PYTHONPATH=$out/${python.sitePackages} for the native extensions to appear when the checkPhase runs.

You should see many examples in pkgs/development/python-modules where cython is used with checkphases

4

Thanks for the correction! Unfortunately, it didn’t help to explain the problem, at pythonImportsCheckPhase modules do get imported (simplex_tree is a one of cythonized modules):

pythonImportsCheckPhase
Executing pythonImportsCheckPhase
Check whether the following modules can be imported: gudhi gudhi.simplex_tree
pytestcachePhase
/nix/store/2a5aldh7vkh77vhkm68v48jzd712wm3y-python3.8-gudhi-3.4.1
5

Yes, that was my first idea too. But (1) the package gets properly installed at $out/${python.sitePackages}; (2) this path does appear in PYTHONPATH by default at check phase. Having the following diagnostics code at checkPhase:

${python.interpreter} --version
${python.interpreter} -c 'import numpy; print(numpy.__version__)'
echo $out/${python.sitePackages}
echo $PYTHONPATH
${python.interpreter} -c 'import gudhi.simplex_tree; print(gudhi.simplex_tree)' || true

gives the following output:

Python 3.8.9
1.20.2
/nix/store/5iaqzmg7948ngl18n04mkh83ldv41bak-python3.8-gudhi-3.4.1/lib/python3.8/site-packages
/nix/store/5iaqzmg7948ngl18n04mkh83ldv41bak-python3.8-gudhi-3.4.1/lib/python3.8/site-packages:/nix/store/zv167zw7gwr9h0gpkfcikdfqnscrd0q6-python3-3.8.9/lib/python3.8/site-packages:/nix/store/d8j11v90ms2ymr8m770bfm31jm6zj2a6-python3.8-setuptools-54.2.0/lib/python3.8/site-packages:/nix/store/f1y3ykg4fjyrgns39j3m71w2py4m15yx-python3.8-wheel-0.36.2/lib/python3.8/site-packages:/nix/store/99f478yfy8jfilmrvmfcan7ldlv1nlk6-python3.8-pip-21.0.1/lib/python3.8/site-packages:/nix/store/ys7ffnghl2iaza8ly4mjpycfyargxv4z-python3.8-numpy-1.20.2/lib/python3.8/site-packages:/nix/store/klp1isshlzhpmnwdlxl9k0xw9zlbwkmq-python3.8-Cython-0.29.22/lib/python3.8/site-packages:/nix/store/28qrb63b4yiflv3bwhdp0sd548xygqz0-python3.8-pybind11-2.6.2/lib/python3.8/site-packages:/nix/store/ym2rmn08956qd5gkkkvhd119rfr5iqhn-python3.8-pytest-6.2.3/lib/python3.8/site-packages:/nix/store/1h74vgsxwfraq8jzv1553q0a3r13fnrz-python3.8-atomicwrites-1.4.0/lib/python3.8/site-packages:/nix/store/i3fgay5f8a9b22h94pmw7l2jy7iqk5pg-python3.8-attrs-20.3.0/lib/python3.8/site-packages:/nix/store/mk373mg1cywf1w7h9ns3gh9yld9indr7-python3.8-iniconfig-1.1.1/lib/python3.8/site-packages:/nix/store/d6c9596m7dx9d591aw5ain39hy0w439a-python3.8-more-itertools-8.7.0/lib/python3.8/site-packages:/nix/store/mah3d3892jir9xji04fy7wwyiaxzlaja-python3.8-six-1.15.0/lib/python3.8/site-packages:/nix/store/qsnkkqbyhckmh21x5vxh5svamwlp2wgf-python3.8-packaging-20.9/lib/python3.8/site-packages:/nix/store/5cd05wizlz87ynkwkvkk4rs1dzxfbpzq-python3.8-pyparsing-2.4.7/lib/python3.8/site-packages:/nix/store/ardnzjn9836rx8bw0n7c0sc856s7dgl5-python3.8-pluggy-0.13.1/lib/python3.8/site-packages:/nix/store/d4lx9h530kwji9cld8r8mgfm9ka7p896-python3.8-importlib-metadata-3.7.3/lib/python3.8/site-packages:/nix/store/mhnp0npp4zzwasa1yn30naj66wkx4dnh-python3.8-toml-0.10.2/lib/python3.8/site-packages:/nix/store/b0hhgkf25x2mg614a7x8b8ycsfj72nng-python3.8-zipp-3.4.1/lib/python3.8/site-packages:/nix/store/9m984jllp5cjdkb2g4p74l6fj8icf0x7-python3.8-py-1.10.0/lib/python3.8/site-packages:/nix/store/m3yqk753ihv5fihcw2865x07f0q12afy-python3.8-wcwidth-0.2.5/lib/python3.8/site-packages:/nix/store/3s6nsn23f110s88kd084q28ssscqdvzn-python3.8-scipy-1.6.1/lib/python3.8/site-packages
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'gudhi.simplex_tree'

I also looked at other packages, and it seems that Cython should work out of the box. E.g. python-modules/tesserocr has cythonized modules and tests without an explicit checkPhase, python-modules/tensorflow does have a custom checkPhase, but it looks like additional environment manipulations aren’t needed.

6

I’ve created a mock project with a Cython module, and it seems that I’ve localized the problem! Although PYTHONPATH was set correctly, there still was gudhi package in the build directory which was the current working directory for the shell. Python prioritizes current directory for imports (sys.path[0] == "" normally), and GUDHI uses absolute imports in its top-level __init__.py, so the package left after building gets imported, not the one from $out/${python.sitePackages}. To keep ctest happy as CTestTestfile.cmake is still in the build dir, I just added rm -r gudhi to the top of checkPhase to make Python use the first PYTHONPATH entry. Otherwise something like cd .. might work too.

After some additional patching in CMakeLists.txt I got the package built with all tests passing! Thanks to everybody for help!

7

oh yea, I forgot to mention that was well. It’s common to delete or move the local "<module>" directory, or cd into the test directory. This avoids python’s default behavior of adding $PWD to sys.path