Packaging a python library with C++ distutils extension?

Help
1

I’m trying to package the ffcv python library. I have a draft PR that almost works. It’s just getting hung up trying to import the C++ extension in tests:

E   ModuleNotFoundError: No module named 'ffcv._libffcv'

Does anyone have any idea what could be going on here? Why isn’t the extension automatically included in PYTHONPATH?

2

Two updates:

  • It seems to work perfectly fine if I just skip the test suite, so there’s some divergence between the test environment and the built package.
  • The built .so appears to live in build/lib.linux-x86_64-3.9/ffcv/_libffcv.cpython-39-x86_64-linux-gnu.so. Why is that not in PYTHONPATH when running tests?
3

I would assume that during the tests all the modules are found through . (i.e. nobody’s adding anything special to PYTHONPATH)?

4

That seems to be the case… But I figured that using distutils would be common enough that it would be handled by buildPythonPackage

5

Hey @samuela @Sandro

I found this thread and the related issue while working on another Python package (roboticstoolbox and related packages). I hacked on this for a while myself but could not figure it out. I reached out to Jon Ringer and we figured it out after a couple of hours. I submitted a PR to fix the ffcv package so the tests run (well at least the ones that behave in a sandbox).

The fix is pretty easy. Just have a preCheck that moves the directory into the tests (or anywhere really). I put the following explanation above the preCheck in the default.nix file.

C/C++ python modules are only in the installed output and not in the build directory. Since tests are run from the build directory python prefers to import the local module first which does not contain the C/C++ python modules and results in an import error. By changing the directory to ‘tests’ the build directory is no long available and python will import from the store which does contain the C/C++ python modules.

I initially used the same fix for my code but ran into an infinite recursion via circular dependency issue. Two of the packages I am trying to submit depend on each other for their tests. So what we did was create a testsout output, copy the tests directory there, set doCheck = false, and have the tests as a passthru that can be built separately. This was primarily done to solve the infinite recursion problem but it also solves the C/C++ module discovery at the same time.

2 Likes