OpenCV with Cuda in nix-shell

dsxmachina · May 25, 2020, 4:09pm

Hello wonderful People :),
I am new to the whole nix ecosystem but super happy that I found it.
For one week now I am stuck with the following problem:

I want to use OpenCV4 with Cuda support in my project.
For the purpose of testing i kept it simple:

int main(){
   cuda::GpuMat gpfoo;
   Mat test = Mat::eye(Size(600, 600), CV_32FC3);
   gpfoo.upload(test);
   return 0;
}

My default.nix looks like this:

{ pkgs ? import <nixpkgs> {} } :
with pkgs;
let 
	 opencv4cuda = opencv4.override {  enableGtk3 = true; 
                                    enableFfmpeg = true; 
                                    enableCuda = true; 
                                    enableUnfree = true;};
in
stdenv.mkDerivation {
  name = "cvCuda_test";
  src = ./.;

  buildInputs = [ cudatoolkit opencv4cuda ];
  nativeBuildInputs = [ pkgconfig libconfig cmake ];
}

I can compile inside a nix-shell with:

nix-shell --pure --command \
	"cmake -H. -Bbuild/ -DCMAKE_BUILD_TYPE=$btype && cmake --build build/. -j12"

And it compiles fine - using the “right” version of cv and cuda from the nix-store. Yay! … BUT:
When i start the program it crashes with:

OpenCV(4.3.0) /build/source/modules/core/src/cuda/gpu_mat.cu:116: error: (-217:Gpu API call) CUDA driver version is insufficient for CUDA runtime version in function 'allocate'

The error code is pretty straightforward. If i compile the project “normally” outside the nix-shell, everything works fine (with identical cuda and cv versions!)
So my Idea was, maybe he did not find the necessary drivers for Cuda in the nix-store;

So I read this: CUDA setup on NixOS - #6 by Aeschylus
And this: Nvidia - NixOS Wiki

But any suggested changes to my default.nix resulted in the same error
(in the end i got this abomination:

{ pkgs ? import <nixpkgs> {} } :
with pkgs;
let 
	 opencv4cuda = opencv4.override {  enableGtk3 = true; 
												enableFfmpeg = true; 
												enableCuda = true;
												enableUnfree = true; };
in
stdenv.mkDerivation {
  name = "cvCuda_test";
  src = ./.;
   buildInputs = [ git gitRepo gnupg autoconf curl
                    procps gnumake utillinux m4 gperf unzip
                    cudatoolkit linuxPackages.nvidia_x11
                    xorg.libXi xorg.libXmu freeglut
                    xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib 
                    ncurses5 stdenv.cc binutils
							opencv4cuda
                   ];
   shellHook = ''
      export CUDA_PATH=${pkgs.cudatoolkit}
       export LD_LIBRARY_PATH=${pkgs.linuxPackages.nvidia_x11}/lib:${pkgs.ncurses5}/lib:/run/opengl-driver/lib
		  export EXTRA_LDFLAGS="-L/lib -L${pkgs.linuxPackages.nvidia_x11}/lib"
		  export EXTRA_CCFLAGS="-I/usr/include"
   '';          
	postFixup = ''
    addOpenGLRunpath $out/bin/add  # i guess this isn't needed but whatever
    wrapProgram $out/bin/add --prefix LD_LIBRARY_PATH ":" "/run/opengl-driver/lib"
  '';	
  nativeBuildInputs = [ pkgconfig libconfig cmake ];
}

)

Has anyone some experience with this topic ? Or even uses opencv with cuda ?
I would be happy for any suggestions or links to further reading that get’s me closer to a solution

Just as info: I use nix version 2.3.4 on ArchLinux

Thanks!

FRidh · May 25, 2020, 8:16pm

Maybe the cudatoolkit that was used to compile the Nixpkgs opencv4 is too new for the drivers of your system?

dsxmachina · May 25, 2020, 8:32pm

Sorry, that i did not mention - I have the exact same versions (cuda toolkit 10.2 and opencv 4.3) installed normally on my system and everything works fine (with the same CMakeLists.txt)

doronbehar · May 26, 2020, 10:28am

Maybe something relevant will be found at Design discussion about nixGL (OpenGL / Cuda / OpenCL wrapper for nix) - #6 by Alejandro .

dsxmachina · May 29, 2020, 7:15pm

Woah indeed, that is super helpful - thank you!

For anyone whos interested - my default.nix is down to this:

{ pkgs ? import <nixpkgs> {} } :
with pkgs;
let 
  opencv4cuda = opencv4.override {  
    enableGtk3   = true; 
    enableFfmpeg = true; 
    enableCuda   = true;
    enableUnfree = true; 
  };
  nixGL = (callPackage ./home/machina/Software/nixGL/nixGL.nix {
          inherit pkgs;
          nvidiaVersion = "440.82.00";
          nvidiaHash    = "edd415acf2f75a659e0f3b4f27c1fab770cf21614e84a18152d94f0d004a758e";
  }).nixGLNvidia;
in
stdenv.mkDerivation {
  name = "cvCuda_test";
  src  = ./.;

  buildInputs = [ 
    cudatoolkit 
    linuxPackages.nvidia_x11
    opencv4cuda
  ];

  shellHook = ''
    export CUDA_PATH=${pkgs.cudatoolkit}
  '';          

  nativeBuildInputs = [ 
    pkgconfig 
    libconfig 
    cmake 
  ];
}

To actually run the program, i use

nixGLNvidia ./cvCuda_test

In the post you linked the idea of a build time wrapper was posted.
Does anyone know how one would do that ?
I tried to call CMake && make within nixGL, but it didn’t had any effect

mjlbach · May 29, 2020, 8:02pm

You can also add a shell hook for nixgl

  shellHook = ''
      export LD_LIBRARY_PATH=$(nixGLNvidia printenv LD_LIBRARY_PATH):$LD_LIBRARY_PATH
    '';

See: Integration with nix-shell? · Issue #46 · nix-community/nixGL · GitHub

dsxmachina · May 30, 2020, 12:08pm

Okay. For some reason nixGLNvidia is not found - although it is installed, so I have to do:

export LD_LIBRARY_PATH=$(${nixGL}/bin/nixGLNvidia printenv LD_LIBRARY_PATH):$LD_LIBRARY_PATH

But that seems to work. Inside a nix-shell i can execute the binary, without wrapping it
To be able to execute it from a normal shell, i append nixGL LD_LIBRARY_PATH to my normal LD_LIBRARY_PATH inside my .bashrc.
Is this the way to do it?

mjlbach · May 30, 2020, 6:23pm

Yeah, the solution I gave assumes you’ve installed nixGLNvidia declaratively (it’s what’s recommended by nixGL in the readme). Your way is better if you have nixGL defined within your nix-shell I think so, if you look at the github link Guibou mentioned this is the current way to do it.

breakds · September 9, 2020, 4:52pm

Not sure whether it is related - I ran in to the problem that cmake’s findOpenGL.cmake fails to find openGL with error message like

CMake Error at /nix/store/cn6awz9ppdqanlyzmrni37pbwq148bjz-cmake-3.16.3/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenGL (missing: OPENGL_opengl_LIBRARY OPENGL_glx_LIBRARY
  OPENGL_INCLUDE_DIR)
Call Stack (most recent call first):
  /nix/store/cn6awz9ppdqanlyzmrni37pbwq148bjz-cmake-3.16.3/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /nix/store/cn6awz9ppdqanlyzmrni37pbwq148bjz-cmake-3.16.3/share/cmake-3.16/Modules/FindOpenGL.cmake:397 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  /nix/store/cn6awz9ppdqanlyzmrni37pbwq148bjz-cmake-3.16.3/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /nix/store/nyssvbqfbc9pj5xq6v9mwq0pqikj9mm7-g2o-unstable-2019-04-07/lib/cmake/g2o/g2oConfig.cmake:4 (find_dependency)

I am running NixOS with nix-shell for this project. And as suggested in this post my shell.nix looks like

let pkgs = import <nixpkgs> {
      config.allowUnfree = true;
    };

    nixGLSrc = builtins.fetchGit {
      url = "https://github.com/guibou/nixGL";
      rev = "210c6a8a547b4a548b89b08bd46ffedc396bc4f4";
    };

    nixGLNvidia = (pkgs.callPackage nixGLSrc {}).nixGLNvidia;

in pkgs.mkShell rec {
  name = "osm";
  buildInputs = with pkgs; [
    cmake
    cmakeCurses
    pkgconfig

    nixGLNvidia

    spdlog
    eigen # >= 3.3
    libyamlcpp
    opencv4
    g2o
  ];
  shellHook = ''
    export PS1="$(echo -e '\uf277') {\[$(tput sgr0)\]\[\033[38;5;228m\]\w\[$(tput sgr0)\]\[\033[38;5;15m\]} (${name}) \\$ \[$(tput sgr0)\]"
    export LD_LIBRARY_PATH=$(nixGLNvidia printenv LD_LIBRARY_PATH):$LD_LIBRARY_PATH
    export CC=clang
    export CXX=clang++
  '';
}

Basically the problem persists after I put in nixGL and populate the LD_LIBRARY_PATH.

mjlbach · September 9, 2020, 6:30pm

LD_LIBRARY_PATH won’t help cmake find build-time dependencies. Have you tried adding libGLU to the build inputs?

breakds · September 9, 2020, 10:39pm

Thanks a lot for the prompt reply. Yes adding libGLU does make it work!