OpenCV with Cuda in nix-shell

Hello wonderful People :),
I am new to the whole nix ecosystem but super happy that I found it.
For one week now I am stuck with the following problem:

I want to use OpenCV4 with Cuda support in my project.
For the purpose of testing i kept it simple:

int main(){
   cuda::GpuMat gpfoo;
   Mat test = Mat::eye(Size(600, 600), CV_32FC3);
   gpfoo.upload(test);
   return 0;
}

My default.nix looks like this:

{ pkgs ? import <nixpkgs> {} } :
with pkgs;
let 
	 opencv4cuda = opencv4.override {  enableGtk3 = true; 
                                    enableFfmpeg = true; 
                                    enableCuda = true; 
                                    enableUnfree = true;};
in
stdenv.mkDerivation {
  name = "cvCuda_test";
  src = ./.;

  buildInputs = [ cudatoolkit opencv4cuda ];
  nativeBuildInputs = [ pkgconfig libconfig cmake ];
}

I can compile inside a nix-shell with:

nix-shell --pure --command \
	"cmake -H. -Bbuild/ -DCMAKE_BUILD_TYPE=$btype && cmake --build build/. -j12"

And it compiles fine - using the “right” version of cv and cuda from the nix-store. Yay! … BUT:
When i start the program it crashes with:

OpenCV(4.3.0) /build/source/modules/core/src/cuda/gpu_mat.cu:116: error: (-217:Gpu API call) CUDA driver version is insufficient for CUDA runtime version in function 'allocate'

The error code is pretty straightforward. If i compile the project “normally” outside the nix-shell, everything works fine (with identical cuda and cv versions!) :confused:
So my Idea was, maybe he did not find the necessary drivers for Cuda in the nix-store;

So I read this: CUDA setup on NixOS - #6 by Aeschylus
And this: Nvidia - NixOS Wiki

But any suggested changes to my default.nix resulted in the same error :frowning:
(in the end i got this abomination:

{ pkgs ? import <nixpkgs> {} } :
with pkgs;
let 
	 opencv4cuda = opencv4.override {  enableGtk3 = true; 
												enableFfmpeg = true; 
												enableCuda = true;
												enableUnfree = true; };
in
stdenv.mkDerivation {
  name = "cvCuda_test";
  src = ./.;
   buildInputs = [ git gitRepo gnupg autoconf curl
                    procps gnumake utillinux m4 gperf unzip
                    cudatoolkit linuxPackages.nvidia_x11
                    xorg.libXi xorg.libXmu freeglut
                    xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib 
                    ncurses5 stdenv.cc binutils
							opencv4cuda
                   ];
   shellHook = ''
      export CUDA_PATH=${pkgs.cudatoolkit}
       export LD_LIBRARY_PATH=${pkgs.linuxPackages.nvidia_x11}/lib:${pkgs.ncurses5}/lib:/run/opengl-driver/lib
		  export EXTRA_LDFLAGS="-L/lib -L${pkgs.linuxPackages.nvidia_x11}/lib"
		  export EXTRA_CCFLAGS="-I/usr/include"
   '';          
	postFixup = ''
    addOpenGLRunpath $out/bin/add  # i guess this isn't needed but whatever
    wrapProgram $out/bin/add --prefix LD_LIBRARY_PATH ":" "/run/opengl-driver/lib"
  '';	
  nativeBuildInputs = [ pkgconfig libconfig cmake ];
}

)

Has anyone some experience with this topic ? Or even uses opencv with cuda ?
I would be happy for any suggestions or links to further reading that get’s me closer to a solution

Just as info: I use nix version 2.3.4 on ArchLinux

Thanks! :slight_smile:

1 Like

Maybe the cudatoolkit that was used to compile the Nixpkgs opencv4 is too new for the drivers of your system?

Sorry, that i did not mention - I have the exact same versions (cuda toolkit 10.2 and opencv 4.3) installed normally on my system and everything works fine (with the same CMakeLists.txt)

Maybe something relevant will be found at Design discussion about nixGL (OpenGL / Cuda / OpenCL wrapper for nix) - #6 by Alejandro .

Woah indeed, that is super helpful - thank you!

For anyone whos interested - my default.nix is down to this:

{ pkgs ? import <nixpkgs> {} } :
with pkgs;
let 
  opencv4cuda = opencv4.override {  
    enableGtk3   = true; 
    enableFfmpeg = true; 
    enableCuda   = true;
    enableUnfree = true; 
  };
  nixGL = (callPackage ./home/machina/Software/nixGL/nixGL.nix {
          inherit pkgs;
          nvidiaVersion = "440.82.00";
          nvidiaHash    = "edd415acf2f75a659e0f3b4f27c1fab770cf21614e84a18152d94f0d004a758e";
  }).nixGLNvidia;
in
stdenv.mkDerivation {
  name = "cvCuda_test";
  src  = ./.;

  buildInputs = [ 
    cudatoolkit 
    linuxPackages.nvidia_x11
    opencv4cuda
  ];

  shellHook = ''
    export CUDA_PATH=${pkgs.cudatoolkit}
  '';          

  nativeBuildInputs = [ 
    pkgconfig 
    libconfig 
    cmake 
  ];
}

To actually run the program, i use

nixGLNvidia ./cvCuda_test

In the post you linked the idea of a build time wrapper was posted.
Does anyone know how one would do that ?
I tried to call CMake && make within nixGL, but it didn’t had any effect

1 Like

You can also add a shell hook for nixgl

  shellHook = ''
      export LD_LIBRARY_PATH=$(nixGLNvidia printenv LD_LIBRARY_PATH):$LD_LIBRARY_PATH
    '';

See: Integration with nix-shell? · Issue #46 · nix-community/nixGL · GitHub

2 Likes

Okay. For some reason nixGLNvidia is not found - although it is installed, so I have to do:

export LD_LIBRARY_PATH=$(${nixGL}/bin/nixGLNvidia printenv LD_LIBRARY_PATH):$LD_LIBRARY_PATH

But that seems to work. Inside a nix-shell i can execute the binary, without wrapping it :slight_smile:
To be able to execute it from a normal shell, i append nixGL LD_LIBRARY_PATH to my normal LD_LIBRARY_PATH inside my .bashrc.
Is this the way to do it?

Yeah, the solution I gave assumes you’ve installed nixGLNvidia declaratively (it’s what’s recommended by nixGL in the readme). Your way is better if you have nixGL defined within your nix-shell :slight_smile: I think so, if you look at the github link Guibou mentioned this is the current way to do it.

Not sure whether it is related - I ran in to the problem that cmake’s findOpenGL.cmake fails to find openGL with error message like

CMake Error at /nix/store/cn6awz9ppdqanlyzmrni37pbwq148bjz-cmake-3.16.3/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenGL (missing: OPENGL_opengl_LIBRARY OPENGL_glx_LIBRARY
  OPENGL_INCLUDE_DIR)
Call Stack (most recent call first):
  /nix/store/cn6awz9ppdqanlyzmrni37pbwq148bjz-cmake-3.16.3/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /nix/store/cn6awz9ppdqanlyzmrni37pbwq148bjz-cmake-3.16.3/share/cmake-3.16/Modules/FindOpenGL.cmake:397 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  /nix/store/cn6awz9ppdqanlyzmrni37pbwq148bjz-cmake-3.16.3/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /nix/store/nyssvbqfbc9pj5xq6v9mwq0pqikj9mm7-g2o-unstable-2019-04-07/lib/cmake/g2o/g2oConfig.cmake:4 (find_dependency)

I am running NixOS with nix-shell for this project. And as suggested in this post my shell.nix looks like

let pkgs = import <nixpkgs> {
      config.allowUnfree = true;
    };

    nixGLSrc = builtins.fetchGit {
      url = "https://github.com/guibou/nixGL";
      rev = "210c6a8a547b4a548b89b08bd46ffedc396bc4f4";
    };

    nixGLNvidia = (pkgs.callPackage nixGLSrc {}).nixGLNvidia;

in pkgs.mkShell rec {
  name = "osm";
  buildInputs = with pkgs; [
    cmake
    cmakeCurses
    pkgconfig

    nixGLNvidia

    spdlog
    eigen # >= 3.3
    libyamlcpp
    opencv4
    g2o
  ];
  shellHook = ''
    export PS1="$(echo -e '\uf277') {\[$(tput sgr0)\]\[\033[38;5;228m\]\w\[$(tput sgr0)\]\[\033[38;5;15m\]} (${name}) \\$ \[$(tput sgr0)\]"
    export LD_LIBRARY_PATH=$(nixGLNvidia printenv LD_LIBRARY_PATH):$LD_LIBRARY_PATH
    export CC=clang
    export CXX=clang++
  '';
}

Basically the problem persists after I put in nixGL and populate the LD_LIBRARY_PATH.

LD_LIBRARY_PATH won’t help cmake find build-time dependencies. Have you tried adding libGLU to the build inputs?

1 Like

Thanks a lot for the prompt reply. Yes adding libGLU does make it work!