How to add include-path to python-package's header-files?

I am working on a derivation of a C++ application (built with autotools) that includes an embedded python-interpreter.

In addition, it uses python’s numpy package. I have added python3 and numpy to the buildInputs and nativebuildinput and to propagatedBuildinputs for good measure, not really knowing what I am doing. (as python310Packages.numpy and python3.withPackages (ps: with ps: [numpy]) )

The build fails with

pyremote.C:1507:10: fatal error: numpy/arrayobject.h: No such file or directory
1507 | #include “numpy/arrayobject.h”

Indeed the CFLAGS do not contain the proper nix-store path to the package (as shown in the nix log output).

Examination of the nix-store yields its true location:

/nix/store/n39mz2vn1q786hd0b2pigqfnjk8jzr5c-python3.10-numpy-1.23.3/lib/python3.10/site-packages/numpy/core/include/numpy/arrayobject.h

How do I have to introduce the path to the python-package numpy in the derivation for it to be added to the include paths?

You need to use something like (python3.withPackages (ps: with ps; [ numpy ])) to create a python with numpy, rather than just adding python and numpy as separate packages.

EDIT: Ah, nvm, you said you tried that.

Could you consider uploading your nix derivation, I’m not an expert in C bindings in python, so it’s much easier to debug this way as this error could potentially come from multiple reasons.

For instance, if the library uses a setup.py and forgets to add include_dirs=[numpy.get_include()] in its setup then it can cause this error.

Then, you might want to look at this similar question Fatal error: numpy/arrayobject.h: No such file or directory while building package, it seems that adding python310Packages.pybind11 in the nativeBuildInput might help.

edit
Finally, a dirty hack could be to add the CFLAGS manually using something like CFLAGS = "-I${python310Packages.numpy}/${pkgs.python310.sitePackages}/numpy/core/include", but I’m sure there are cleaner methods to properly add them.

OK, adding python310Packages.pybind11 did not help.
There is no python library and no setup.py in this project.
The python-libraries are included as pure C/C++ as documented here: 1. Embedding Python in Another Application — Python 3.11.1 documentation
I will try the CFLAGS hack, but I am still interested in the proper way…
I will post the entire nix file soon. It contains a lot of cruft that I want to remove first.

Hum… It might be helpful to try to add pkg-config as it usually automatically configures the include dir, but since I don’t see any .pc file in the numpy derivation I can’t guarantee that it will work. Note that some packages provide options to configure this path, for instance in rdkit (pkgs/development/python-modules/rdkit/default.nix) you have the option PYTHON_NUMPY_INCLUDE_PATH:

preConfigure = ''
  cmakeFlags="$cmakeFlags -DPYTHON_NUMPY_INCLUDE_PATH=$(${python}/bin/python -c 'import numpy; print(numpy.get_include())')"
'';

and in blender (pkgs/applications/misc/blender/default.nix) you have another option PYTHON_NUMPY_INCLUDE_DIRS:

cmakeFlags = [
  "-DPYTHON_NUMPY_INCLUDE_DIRS=${python310Packages.numpy}/${python.sitePackages}/numpy/core/include"
];

Your software might provide a similar option to configure this path.

I guess we can investigate more once we have a working derivation to try with.

I had seen the blender derivation, too; but I didn’t know how to translate a cmake-specific keyword to autotools/generic CFLAGS.
I am currently trying to figure out how to introduce my CFLAGS…
I will keep you informed.

This specific option is specific to blender, so you might need to find in the documentation of your package a similar instruction. Regarding the CFLAGS hack I posted above, you should just be able to copy/paste this line in the set of the derivation directly.

Hmm, neither CFLAGS nor NIX_CFLAGS_COMPILE in the mkDerivation are used in the compilation step. I have tried stuffing the hardcoded path into it. The build process prints the CFLAGS, and the path doesn’t end up there.

Try to set NIX_DEBUG=1 to see what flags gcc is using, NIX_CFLAGS_COMPILE flags is sent directly to the compiler gcc, and are not seen by the build process directly. See also NIX_LDFLAGS to send commands to the linker ld.

For more help, you should provide your .nix file, or at least the name of the project you package.

Ha, it’s a physics application, hosted here: https://gxsm.sourceforge.net/, but forked here:
GitHub - StefanSchroeder/Gxsm3: Gnome X Scanning Microscopy -- see also https://gxsm.sf.net
The first challenge was that it’s using an obsolete version of the netcdf-library which is not in nixpkgs. I added it to the nixpkgs successfully after some help here in discourse.

It uses more than a dozen libraries.

  pname = "gxsm3";
  version = "3.0.2";

  src = fetchFromGitHub {
    owner= "StefanSchroeder";
    repo= "Gxsm3";
    rev ="a5b3c2ec8629bf9e9927f103b7c5cb89c9fbba6a";
    sha256= "gxaT+qIoPYWsTqALmKiMOxs55+qeNfHSEwSO7eFVvoU=";
  };

Not all the includes are actually used:

  lib, stdenv,
  fetchFromGitHub, 
  gtk3, 
  pkg-config, 
  fftw, 
  pythonSupport ? false, 
  python2Packages,
  python3,
  python310Packages,
  gnome2,
  autoconf, libtool, automake, fetchpatch,
  openexrSupport ? true, openexr,
  libzipSupport ? true, libzip,
  libwebpSupport ? true, libwebp,
  # libXmu is not used if libunique is.
  libXmuSupport ? false, xorg,
  libxsltSupport ? true, libxslt,
  fitsSupport ? true, cfitsio,
  zlibSupport ? true, zlib,
  libuniqueSupport ? true, libunique,
  libpngSupport ? true, libpng,
  openglSupport ? !stdenv.isDarwin,
  intltool,
  libxml2,
  gtksourceview,
  bison,
  hostname,
  libsoup,
  yelp-tools,
  libglvnd,
  icu,
  itstool,
  glm,
  freeglut, libGLU, libGL,
  glew,
  gsl,
  nettools, 
  netcdf, 
  netcdfcxx4, 
  netcdfcxx4-legacy // my addition, not in nixpkgs

This is buildinputs:

nativeBuildInputs = [
    pkg-config 
    autoconf
    automake
    pkg-config
    intltool
    libtool
    libxml2
    bison
    yelp-tools
    itstool
    libglvnd
    nettools
    icu
    netcdf
    netcdfcxx4
    netcdfcxx4-legacy
    python310Packages.numpy
    python310Packages.pybind11
  ];

This is embarrassingly crufty, but you asked for it. :wink:
I should clean this up. I have taken the advice from the manual to use some other project’s derivation as a starting point and haven’t taken the time yet to remove unneeded functions. (The other derivation that I used as a base was gwyddion which is somewhat similar to this app.)

I’m not sure, but stdenv may not pick up includes and libs from nativeBuildInputs. Even if it does, you should put those things in buildInputs.

Ahah, can’t you give the full .nix file to copy/paste, instead of chunks? ^^ The goal is to allow us to simply debug your issue by trying ourselve. For instance the buildInputs and netcdfcxx4-legacy are missing, and I’m not sure if you just use other stuff in the derivation. If you have multiple files, you might find it easier to use git gist to send them. Also, most libraries (e.g. libxml2) should probably be in buildInputs instead of nativeBuildInputs.

Okay, okay, this is the current working state.
You have been warned. :sweat_smile:

pkgs/applications/science/physics/gxsm3/default.nix

{ lib, stdenv, 
  fetchFromGitHub, 
  gtk3, 
  pkg-config, 
  fftw, 
  pythonSupport ? false, 
  python2Packages,
  python3,
  python310Packages,
  gnome2,
  autoconf, libtool, automake, fetchpatch,
  openexrSupport ? true, openexr,
  libzipSupport ? true, libzip,
  libwebpSupport ? true, libwebp,
  # libXmu is not used if libunique is.
  libXmuSupport ? false, xorg,
  libxsltSupport ? true, libxslt,
  fitsSupport ? true, cfitsio,
  zlibSupport ? true, zlib,
  libuniqueSupport ? true, libunique,
  libpngSupport ? true, libpng,
  openglSupport ? !stdenv.isDarwin,
  intltool,
  libxml2,
  gtksourceview,
  bison,
  hostname,
  libsoup,
  yelp-tools,
  libglvnd,
  icu,
  itstool,
  glm,
  freeglut, libGLU, libGL,
  glew,
  gsl,
  nettools, 
  netcdf, 
  netcdfcxx4, 
  netcdfcxx4-legacy
}:

let
    inherit (python310Packages) pygtk pygobject2 python;
in

stdenv.mkDerivation rec {

  hardeningDisable = [ "all" ];

  pname = "gxsm3";
  version = "3.0.2";

  src = fetchFromGitHub {
    owner= "StefanSchroeder";
    repo= "Gxsm3";
    rev ="a5b3c2ec8629bf9e9927f103b7c5cb89c9fbba6a";
    sha256= "gxaT+qIoPYWsTqALmKiMOxs55+qeNfHSEwSO7eFVvoU=";
  };

  nativeBuildInputs = [
    pkg-config 
    autoconf
    automake
    pkg-config
    intltool
    libtool
    libxml2
    bison
    yelp-tools
    itstool
    libglvnd
    nettools
    icu
    netcdf
    netcdfcxx4
    netcdfcxx4-legacy
    python310Packages.numpy
    python310Packages.pybind11
  ];

  buildInputs = with lib;

    [ python310Packages.numpy gsl icu glm freeglut libGLU libGL glew gtk3 fftw autoconf libxml2 libxml2 gtksourceview libsoup ] ++

    optional openglSupport gnome2.gtkglext ++
    /*optional openexrSupport openexr ++*/
    optional libXmuSupport xorg.libXmu ++
    optional fitsSupport cfitsio ++
    optional libpngSupport libpng ++
    optional libxsltSupport libxslt ++
/*    optional libwebpSupport libwebp ++*/
    optional zlibSupport zlib ++
    optional libuniqueSupport libunique ++
    optional libzipSupport libzip;

/*  propagatedBuildInputs = with lib;
    optionals pythonSupport [ numpy pygtk pygobject2 python gnome2.gtksourceview ];
*/
  propagatedBuildInputs = [
    (python3.withPackages (ps: with ps; [ numpy ]))
  ];

  configureScript = "./autogen.sh";
#  patches = [ ./codegen.patch ];

  /*NIX_CFLAGS_COMPILE = "-I/nix/store/n39mz2vn1q786hd0b2pigqfnjk8jzr5c-python3.10-numpy-1.23.3/lib/python3.10/site-packages/numpy/core/include/numpy/";*/

/*CFLAGS = "-I/nix/store/n39mz2vn1q786hd0b2pigqfnjk8jzr5c-python3.10-numpy-1.23.3/lib/python3.10/site-packages/numpy/core/include/numpy/";*/

  meta = {
    homepage = "https://sourceforge.net/projects/gxsm/";

    description = "Scanning probe microscopy control and analysis";

    longDescription = ''
	GXSM -- Gnome X Scanning Microscopy: A multi-channel image and
	vector-probe data acquisition and visualization system designed for
	SPM techniques (STM,AFM..), but also SPA-LEED/LEED/LEEM data analysis.
	A plug-in interface allows any user add-on data-processing and special
	hardware and instrument support. Latest: NC-AFM and related
	explorative methods as SQDM can be configured. High-Speed external
	PAC-PLL hardware option with digital DSP link.
    '';
    license = lib.licenses.gpl2;
    platforms = with lib.platforms; linux ++ darwin;
    maintainers = [ lib.maintainers.todo ];
    # never built on aarch64-darwin since first introduction in nixpkgs
    broken = stdenv.isDarwin && stdenv.isAarch64;
  };
}

and

./pkgs/development/libraries/netcdf-cxx4-legacy/default.nix

{ lib, stdenv, fetchurl, netcdf, hdf5, curl }:
stdenv.mkDerivation rec {
  pname = "netcdf-cxx4-legacy";
  version = "4.2";

  src = fetchurl {
    url = "https://downloads.unidata.ucar.edu/netcdf-cxx/4.2/netcdf-cxx-4.2.tar.gz";
    sha512 = "11ycy7x3j12f5ba7y4pnw6zwpfxq7jsp14a8rwgdadkp80jwpvplr56r7sn6c63xmapr3x8qdgjhvhlwm176hb9in38c8dc332xnz9l";
  };


  preConfigure = ''
  '';

  buildInputs = [ netcdf hdf5 curl ];

  doCheck = false;
  enableParallelChecking = false;

  meta = {
    description = "C++ API to manipulate netcdf files";
    homepage = "https://www.unidata.ucar.edu/software/netcdf/";
    license = lib.licenses.free;
    platforms = lib.platforms.unix;
  };
}

Much better, thanks.

So here you go, your library compiles fine now. I just did a bit of cleaning and added:

  NIX_CFLAGS_COMPILE = [
    "-I${python3.pkgs.numpy}/${python3.sitePackages}/numpy/core/include"
  ];

To fix the error (I’m not sure if there is a cleaner way to solve it, but at least it works).

Then to compile the rest of the library I just added a bunch of needed dependencies and packaged libquicktime as well (no difficulty). Note that it seems like a lot of the library you added are useless (not sure why you added so many libraries, usually when you get an error about a missing library, you can use nix-locate nameofthefile.h to find the name of the missing library. Anyway, I’ll let you do the cleaning.

gxsm3/default.nix

{ lib, stdenv, 
  fetchFromGitHub, 
  gtk3, 
  pkg-config, 
  fftw, 
  pythonSupport ? false, 
  python3,
  gnome2,
  autoconf, libtool, automake, fetchpatch,
  openexrSupport ? true, openexr,
  libzipSupport ? true, libzip,
  libwebpSupport ? true, libwebp,
  # libXmu is not used if libunique is.
  libXmuSupport ? false, xorg,
  libxsltSupport ? true, libxslt,
  fitsSupport ? true, cfitsio,
  zlibSupport ? true, zlib,
  libuniqueSupport ? true, libunique,
  libpngSupport ? true, libpng,
  openglSupport ? !stdenv.isDarwin,
  intltool,
  libxml2,
  gtksourceview,
  bison,
  hostname,
  libsoup,
  yelp-tools,
  libglvnd,
  icu,
  itstool,
  glm,
  freeglut, libGLU, libGL,
  glew,
  gsl,
  nettools, 
  netcdf, 
  # netcdfcxx4, 
  netcdfcxx4-legacy,
  libquicktime,
  opencv2,
  glibc, # Or error on ld: cannot find -lnlopt: No such file or directory. https://unix.stackexchange.com/questions/371050/ld-cannot-find-lc-on-nixos
  nlopt, # or errors on ld: cannot find -lnlopt
  popt,
}:

stdenv.mkDerivation rec {

  # hardeningDisable = [ "all" ];

  pname = "gxsm3";
  version = "3.0.2";

  src = fetchFromGitHub {
    owner= "StefanSchroeder";
    repo= "Gxsm3";
    rev ="a5b3c2ec8629bf9e9927f103b7c5cb89c9fbba6a";
    sha256= "gxaT+qIoPYWsTqALmKiMOxs55+qeNfHSEwSO7eFVvoU=";
  };

  nativeBuildInputs = [
    pkg-config 
    autoconf
    automake
    # glibc.static
    bison
  ];

  dontDisableStatic = true;
  # configurePlatforms = [ "build" "host" "target" ];
  postPatch = ''
    # Or we get an error:
    # error: format not a string literal and no format arguments [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wformat-security-Werror=format-security8;;]
    substituteInPlace gxsm3/app_view.C --replace "g_message (line);" 'g_message ("%s", line);'
  '';
  
  buildInputs = with lib;
    [
      gsl
      icu
      glm
      freeglut
      libGLU
      libGL
      glew
      gtk3
      fftw
      autoconf
      libxml2
      libxml2
      gtksourceview
      libsoup
      intltool
      libtool
      libxml2
      yelp-tools
      itstool
      libglvnd
      nettools
      icu
      netcdf
      # netcdfcxx4
      netcdfcxx4-legacy
      libquicktime
      opencv2.dev
      nlopt
      popt
    ] ++
    optional openglSupport gnome2.gtkglext ++
    # optional openexrSupport openexr ++
    optional libXmuSupport xorg.libXmu ++
    optional fitsSupport cfitsio ++
    optional libpngSupport libpng ++
    optional libxsltSupport libxslt ++
    # optional libwebpSupport libwebp ++
    optional zlibSupport zlib ++
    optional libuniqueSupport libunique ++
    optional libzipSupport libzip;

  propagatedBuildInputs = with python3.pkgs; [
    pygobject3
    numpy
    pybind11
  ];
  doCheck = false;

  NIX_CFLAGS_COMPILE = [
    "-I${python3.pkgs.numpy}/${python3.sitePackages}/numpy/core/include"
  ];
  preConfigure = ''
    ./autogen.sh
  '';
  
  meta = {
    homepage = "https://sourceforge.net/projects/gxsm/";

    description = "Scanning probe microscopy control and analysis";

    longDescription = ''
    	GXSM -- Gnome X Scanning Microscopy: A multi-channel image and
    	vector-probe data acquisition and visualization system designed for
    	SPM techniques (STM,AFM..), but also SPA-LEED/LEED/LEEM data analysis.
    	A plug-in interface allows any user add-on data-processing and special
    	hardware and instrument support. Latest: NC-AFM and related
    	explorative methods as SQDM can be configured. High-Speed external
    	PAC-PLL hardware option with digital DSP link.
    '';
    license = lib.licenses.gpl2;
    platforms = with lib.platforms; linux ++ darwin;
    maintainers = [ lib.maintainers.todo ];
    # never built on aarch64-darwin since first introduction in nixpkgs
    broken = stdenv.isDarwin && stdenv.isAarch64;
  };
}

libquicktime/default.nix

{
  lib,
  stdenv,
  fetchurl,
  doxygen,
  zlib,
}:
stdenv.mkDerivation rec {
  pname = "libquicktime";
  version = "1.2.4";
  src = fetchurl {
    url = "mirror://sourceforge/${pname}/${pname}/${version}/${pname}-${version}.tar.gz";
    sha256 = "HFM1nDOzE0e017ANNhFGP+XpQsrj7A/v4NL9QT/Uc2g=";
  };
  buildInputs = [
    doxygen
    zlib
  ];
}

netcdf-cxx4-legacy/default.nix:

{ lib, stdenv, fetchurl, netcdf, hdf5, curl }:
stdenv.mkDerivation rec {
  pname = "netcdf-cxx4-legacy";
  version = "4.2";

  src = fetchurl {
    url = "https://downloads.unidata.ucar.edu/netcdf-cxx/4.2/netcdf-cxx-4.2.tar.gz";
    sha512 = "11ycy7x3j12f5ba7y4pnw6zwpfxq7jsp14a8rwgdadkp80jwpvplr56r7sn6c63xmapr3x8qdgjhvhlwm176hb9in38c8dc332xnz9l";
  };

  buildInputs = [ netcdf hdf5 curl ];

  doCheck = false;
  enableParallelChecking = false;

  meta = {
    description = "C++ API to manipulate netcdf files";
    homepage = "https://www.unidata.ucar.edu/software/netcdf/";
    license = lib.licenses.free;
    platforms = lib.platforms.unix;
  };
}

default.nix

{ pkgs ? import <nixpkgs> {} }:
let
  netcdfcxx4-legacy = pkgs.callPackage ./netcdf-cxx4-legacy/default.nix {};
  libquicktime = pkgs.callPackage ./libquicktime {};
  gxsm3 = pkgs.callPackage ./gxsm3/default.nix { inherit netcdfcxx4-legacy libquicktime; };
in gxsm3

Compile with nix-build

Strangely enough, patching the hardcoded path to the reportedly missing header files into the CFLAGS did not resolve the issue of them being reported as missing… There must be something else going on.

I’m not sure if it answers your question, but using NIX_CFLAGS_COMPILE it works fine (see my post above), I’m not sure why CFLAGS fails here… but I’m not an expert of cmake at all. If you want to see precisely what is going on, you might want to set export NIX_DEBUG = 1 for instance it the preConfigure phase, or I guess it should work as well if you put it in mkDerivation { NIX_DEBUG = 1; …}.

I’ll try it out first thing in the morning tomorrow. :+1:

The hint re. NIX_CFLAGS_COMPILE actually resolved the error!
This question is solved. Now onto the next error. Thank you, @tobiasBora for your great help and for not giving up on nudging me to provide the necessary details. :grinning:

You still have errors? Using the above code everything compiled smoothly on my side.

Hm. While I marked this question as ‘solved’, since the error abt python/numpy was resolved, there are other issues that I am now attending to. The next error was a lack of libquicktime, which is not packaged in Nix, probably because the library hasn’t been updated since 2012 (https://libquicktime.sourceforge.net/). this is not a big deal, because the main application uses a plugin-infrastructure and I don’t need the quicktime import feature. Next on the list is opencv, but I know that this is included in Nixpkgs and I just need to look up the package names. Next steps: Get the missing deps resolved; then possibly add libquicktime to (local) nixpkgs; evaluate if it’s worthwhile to commit upstream.

That being said, I am surprised to hear that the compilation passed on your end. Go figure.