Correct way to Nix-ify shared native libraries?

adelbertc · February 22, 2020, 10:15pm

Hey folks,

I am trying to setup a Nix build for a C++ CMake-based project and am wondering what the correct way to set this up yes. This is what I currently have which works:

let
  nixpkgs = fetchGit {
    url = "https://github.com/NixOS/nixpkgs.git";
    rev = "8130f3c1c2bb0e533b5e150c39911d6e61dcecc2";
  };

  pkgs = import nixpkgs {};
in
  pkgs.stdenv.mkDerivation {
    name = "tvm";
    src = pkgs.lib.cleanSource ./.;
    nativeBuildInputs = [ pkgs.cmake ];

    dontUseCmakeConfigure = true;
    enableParallelBuilding = true;
    NIX_BUILD_CORES = 4;

    installPhase = ''
      mkdir $out
      cp -r build/* $out/
    '';
  }

This seems to approximately work, I am primarily looking to get at the *.dylib (macOS) and *.so (Linux) shared libraries. In the result directory I see these files among a couple other CMake files so it looks OK.

However in my experience with Nix packages I know some packages have a separate *-dev package and there is some support for a $outputDev placeholder variable when building with multiple outputs? Given what I am building doesn’t have an executable and is just a shared library is the way I have up there fine, or should I be using another way instead?

Also curious for any suggestions to improve the expression as well.

Thanks!
-Adelbert

knedlsepp · February 24, 2020, 8:49pm

This looks a little off by me. Why did you resort to using dontUseCmakeConfigure = true;? This looks to me that you will have to manually build the project before being able to call nix-build, which will be both not reproducible and also result in incorrect *.so files not containing the correct references to depending libraries.

Usually you would nixify your project in such a way that nix-build produces correct outputs and then using nix-shell to interactively build and develop the package should work mostly out of the box.

I would assume that removing both the dontUseCmakeConfigure and the manually defined installPhase should get you somewhat further. If cmake fails during configuring the project you might want to add possible other dependencies of your project

adelbertc · October 25, 2020, 9:20pm

Sorry for the extremely late reply! I am back to trying to get this project done the right Nix way.

The reason I set dontUseCmakeConfigure = true was purely experimentally, without it when I try to nix-build I get:

...
CMake Warning at cmake/modules/Git.cmake:39 (message):
  Git not found
Call Stack (most recent call first):
  CMakeLists.txt:353 (include)


-- Performing Test SUPPORT_CXX14
-- Performing Test SUPPORT_CXX14 - Success
-- Building with TVM Map...
-- Build with thread support...
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Configuring done
CMake Error at CMakeLists.txt:369 (add_library):
  Cannot find source file:

    src/target/var/empty/build_cuda_off.cc

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++ .hm
  .hpp .hxx .in .txx


CMake Error at CMakeLists.txt:372 (add_library):
  No SOURCES given to target: tvm


CMake Error at CMakeLists.txt:369 (add_library):
  No SOURCES given to target: tvm_objs


CMake Generate step failed.  Build files cannot be regenerated correctly.
builder for '/nix/store/23xbzrdg7y9rnylwk584r16dlwh9vmhg-tvm.drv' failed with exit code 1
error: build of '/nix/store/23xbzrdg7y9rnylwk584r16dlwh9vmhg-tvm.drv' failed

Somehow I found that flag and I can confirm with that flag on and a fresh repo checkout everything builds fine.

Separate from that I am also curious, after a nix-build in my result/ directory I see:

CMakeCache.txt	CMakeFiles  Makefile  cmake_install.cmake  libtvm.dylib  libtvm_runtime.dylib

… so the shared libraries are in there, but not sure if that is the right place as I often see other Nix C++ projects have a -dev folder with include/ or share/ folders?

knedlsepp · October 25, 2020, 11:27pm

The “var/empty” looks suspiciously like something went wrong in detecting a target or an environment variable being unset. This means there is a bug in your CMakelists.txt

As for why you’ve got CMakeCache.txt in your $out: in your custom installPhase you’re copying everything in the build directory to the output directory.
This is not a good idea since the build directory also will contain some things that should not go to the output.

Instead normally nixpkgs is set up to call “make install”. So as long as your CMakeLists.txt contains a “install(…)” command everything should work fine if you don’t override the installPhase manually.

jtojnar · October 25, 2020, 11:44pm

github.com

NixOS/nixpkgs/blob/3825e74193a6494996dd7f7cfb17f9942f0f2b86/pkgs/development/tools/build-managers/cmake/setup-hook.sh#L5-L13

    
      
          fixCmakeFiles() {
              # Replace occurences of /usr and /opt by /var/empty.
              echo "fixing cmake files..."
              find "$1" \( -type f -name "*.cmake" -o -name "*.cmake.in" -o -name CMakeLists.txt \) -print |
                  while read fn; do
                      sed -e 's^/usr\([ /]\|$\)^/var/empty\1^g' -e 's^/opt\([ /]\|$\)^/var/empty\1^g' < "$fn" > "$fn.tmp"
                      mv "$fn.tmp" "$fn"
                  done
          }

knedlsepp · October 26, 2020, 12:23am

Wow. I wasn’t aware of that. Is this really a good idea? Seems that every occurrence of ‘/usr’ will be replaced. Even legitimate usages will be removed?

adelbertc · October 26, 2020, 1:54am

Ahh yeah, in this case it was “incorrectly” picking up the /opt as it was referencing a directory in the project itself as opposed to a global one. Setting dontFixCmake = true; seems to fix it - I see a couple other derivations in nixpkgs do the same - thank you!

EDIT Toggling dontFixCmake and then removing the installPhase override works beautifully! It results in the expected include and lib subdirectories in result woot - thanks everyone!