Python package fails to find 'setup.py' with cmake out-of-source build

This is somewhat of a continuation of a previous question, but the problems and solutions discussed there are no longer relevant here.
I’m packaging cgal-bindings for python. I’m installing it with the typical python.withPackages approach using the following package definition:

{ lib, stdenv
, buildPythonPackage
, fetchFromGitHub
, cmake
, swig
, cgal_5
, numpy
#cgal_dependencies
, gmp
, mpfr
, boost
, eigen
}:

buildPythonPackage rec {
  pname = "cgal-python";
  version = "2479e35";

  src = fetchFromGitHub {
    owner = "CGAL";
    repo = "cgal-swig-bindings";
    rev = version;
    sha256 = "1ahy6qykba6455jd3xlx93khkv1c17h071p7z3vyqdy44przh9f2";
  };

  buildInputs = [ cgal_5 gmp mpfr boost eigen ];
  propagatedBuildInputs = [ numpy ];
  nativeBuildInputs = [ swig cmake ];

  meta = with lib; {
    description = "CGAL bindings using SWIG";
    homepage = "http://cgal.org";
    license = with licenses; [ gpl3Plus lgpl3Plus];
    platforms = platforms.all;
  };

  cmakeFlags =  ["-DCGAL_DIR=${cgal_5}/lib/cmake/CGAL" "-DBUILD_JAVA=OFF"];
}

Installing this package fails at the setuptoolsBuildPhase with the following output:

Executing setuptoolsBuildPhase
Traceback (most recent call last):
  File "/build/source/build/nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "/nix/store/cgxc3jz7idrb1wnb2lard9rvcx6aw2si-python3-3.9.6/lib/python3.9/tokenize.py", line 392, in open
    buffer = _builtin_open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'setup.py'
builder for '/nix/store/hlnhah1vvl4dzkvp1z4rv1bj9bjlav48-python3.9-cgal-python-2479e35.drv' failed with exit code 1
cannot build derivation '/nix/store/zvs0xf6iny1lfzizvwbcf2jc47mvjbs9-python3-3.9.6-env.drv': 1
dependencies couldn't be built
cannot build derivation '/nix/store/i7zf826cig9x7dm0541xd3if7a3rj0wf-test-fhs-env-usr-target.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/n6aasgnhh0v5lb1wiyvgssswnj3m5bwl-test-fhs-env-fhs.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/qirkzpcxyxvpf3b18fs1jaz96vzqqjbj-test-fhs-env-init.drv': 1
dependencies couldn't be built
error: build of '/nix/store/qirkzpcxyxvpf3b18fs1jaz96vzqqjbj-test-fhs-env-init.drv' failed

The reason for this turns out to be pretty straightforward; this phase is executed in the wrong directory. setup.py is located in /build/source and the cmake command is executed in /build/source/build. The installation instructions indicate that the package can either be built in-source, where cmake is executed in the project root, or out-of-source, where cmake is executed in some other directory and is given the location of the project root. The default behaviour seems to be a hybrid approach where it moves out of the project root but looks for all files in the working directory as if it’s the project root. This is probably caused by some weird interplay between buildPythonPackage and cmake but since both of these interact with the packaging process by implicitly adding code, it’s very difficult to debug or understand what’s going on.
By a process of trial and error I was able to get the installation to run to completion by adding the following code:

  preBuild = ''
    cd /build/source #equivalent to `cd ..`
    mkdir -p build/CGAL 
    ln -sf build/CGAL .
  '';

However, aside from being pretty ugly and almost certainly not the correct way to fix this problem, it doesn’t completely work: When attempting to import the library after the installation I get he following error:

>>> from CGAL.CGAL_Kernel import Point_2
Traceback (most recent call last):
  File "/nix/store/07jjbr2fr7cpnmz68jncw575sy4568ik-python3-3.9.6-env/lib/python3.9/site-packag
    return importlib.import_module(mname)
  File "/nix/store/cgxc3jz7idrb1wnb2lard9rvcx6aw2si-python3-3.9.6/lib/python3.9/importlib/__ini
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 565, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1173, in create_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
ImportError: libCGAL_Kernel_cpp.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/07jjbr2fr7cpnmz68jncw575sy4568ik-python3-3.9.6-env/lib/python3.9/site-packag
    _CGAL_Kernel = swig_import_helper()
  File "/nix/store/07jjbr2fr7cpnmz68jncw575sy4568ik-python3-3.9.6-env/lib/python3.9/site-packag
    return importlib.import_module('_CGAL_Kernel')
  File "/nix/store/cgxc3jz7idrb1wnb2lard9rvcx6aw2si-python3-3.9.6/lib/python3.9/importlib/__ini
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_CGAL_Kernel'

This seems like it’s caused by the same problem, since it’s essentailly a “file not found” error. I suspect that if I could just add .. or /build/source to thew very end of the cmake command it would work, but I could only find doucmentation for inserting arguments into the middle of the command (which, if I may, is something of a microcasm of the understandability problems plaguing NixOS). So what am I doing wrong here? And what, if anything, is the correct way to fix this problem?

2 Likes

me too facing sam eproblem :: setup.py file not found, but it is there in place.

I eventually solved this problem. It was a while ago and I don’t remember the exact line of reasoning I had to follow but this was the package definition that worked:

{ lib
, stdenv
, python3
, toPythonModule
, fetchFromGitHub
, cmake
, swig
, cgal_5
, numpy
#cgal_dependencies
, gmp
, mpfr
, boost
, eigen
}:

toPythonModule (stdenv.mkDerivation rec {
  pname = "cgal-python";
  version = "2479e35";

  src = fetchFromGitHub {
    owner = "CGAL";
    repo = "cgal-swig-bindings";
    rev = version;
    sha256 = "1ahy6qykba6455jd3xlx93khkv1c17h071p7z3vyqdy44przh9f2";
  };

  buildInputs = [ cgal_5 gmp mpfr boost eigen python3 ];
  propagatedBuildInputs = [ numpy ];
  nativeBuildInputs = [ cmake swig ];

  cmakeFlags =  [
    "-DCGAL_DIR=${cgal_5}/lib/cmake/CGAL"
    "-DBUILD_JAVA=OFF"
    "-DPYTHON_OUTDIR_PREFIX=${placeholder "out"}/${python3.sitePackages}/CGAL"
  ];

  meta = with lib; {
    description = "CGAL bindings using SWIG";
    homepage = "https://github.com/CGAL/cgal-swig-bindings";
    license = licenses.boost;
    platforms = platforms.all;
  };
})

The main difference is that instead of using buildPythonPackage, I’m building an ordinary derivation and then using it as a Python module with toPythonModule. This frees my from the assumptions that buildPythonPackage makes about the source.
I haven’t run this code in two years so I can’t verify that it still works, but I may as well share it.

1 Like