Nix-shell, C++ developer environments, custom libraries and pkg-config

I’m switching to NixOS on my workstation (coming from Gentoo), and the nix-shell looks like an awesome tool. I’m trying to figure out how to integrate that with the way how we’re building some C and C++ projects at work. As a rule, our projects require a mixture of boring system packages (think PCRE, OpenSSL, etc) as well as some bleeding-edge libraries. For those bleeding-edge dependencies, we’re pinning their specific versions via this git submodule. We’re using CMake, and we’re almost exclusively relying on pkg-config to find paths to all dependencies. Here’s how a typical CI build script looks like if you’re curious.

On my development machine, I would like to be able to switch between different “build environments” – say, GCC 9, GCC 9 with ASAN, or CLANG 10 with TSAN. On my old Gentoo system I was doing this by using system packages for my boring dependencies, and installing my apps and bleeding-edge dependencies into a custom prefix, once per each “build environment”, say, ~/proj/gcc-9 and ~/proj/clang-10-tsan. This required setting up a few variables such as PKG_CONFIG_PATH and PATH so that they include my prefix. I tried to replicate this with nix-shell by doing roughly this:

#!/run/current-system/sw/bin/env nix-shell
#!nix-shell -i bash -p cmake ninja gcc pkg-config swig pcre bison flex libev openssl libssh python38 python3.pkgs.pytest

PREFIX=${TOP_BUILD_DIR}/target
export PATH=${PREFIX}/bin:$PATH
export PKG_CONFIG_PATH=${PREFIX}/lib64/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}

CMAKE_OPTIONS="-GNinja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 \
    -DCMAKE_INSTALL_RPATH:INTERNAL=${PREFIX}/lib64 \
    -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:INTERNAL=ON \
    -DCMAKE_PREFIX_PATH=${PREFIX} \
    -DCMAKE_INSTALL_PREFIX=${PREFIX}"

build_dep_cmake() {
    BUILD_DIR=${TOP_BUILD_DIR}/$(basename $1)
    mkdir -p ${BUILD_DIR}
    pushd ${BUILD_DIR}
    cmake ${CMAKE_OPTIONS} $1
    ninja install
    ctest --output-on-failure
    popd
}

build_dep_cmake ${PROJECT_ROOT}/project/submodules/dependencies/A
build_dep_cmake ${PROJECT_ROOT}/project/submodules/dependencies/B
build_dep_cmake ${PROJECT_ROOT}/project/submodules/dependencies/C

In the code above, all of the A, B and C dependencies provide some *.pc files which will be needed by the later stages of the build. On a regular Linux system, this works fine because pkg-config can find systemwide packages (such as libev) as well as my custom A, B and C libraries which are coming from my $PREFIX. Under the nix-shell, however, the $PKG_CONFIG_PATH gets created as a union of all library dependencies, which makes sense. That variable eventually gets renamed to $PKG_CONFIG_PATH_x86_64_unknown_linux_gnu, and that variable is then used as-is for overriding my own user-provided env variables in the pkg-config wrapper. Here’s how the final pkg-config that is in my $PATH when under nix-shell looks like:

# ...
if (( ${#role_suffixes[@]} > 0 )); then
        # replace env var with nix-modified one
    PKG_CONFIG_PATH=$PKG_CONFIG_PATH_x86_64_unknown_linux_gnu exec /nix/store/52360vwgzlv2h27pyzidv542gk6dvp1c-pkg-config-0.29.2/bin/pkg-config "$@"
else
        # pkg-config isn't a bonafied dependency so ignore setup hook entirely
        exec /nix/store/52360vwgzlv2h27pyzidv542gk6dvp1c-pkg-config-0.29.2/bin/pkg-config "$@"
fi
#...

The TL;DR version is that Nix’ pkg-config wrapper overrides my modifications to $PKG_CONFIG_PATH. I can hack that around by setting $PKG_CONFIG_PATH_x86_64_unknown_linux_gnu in my own wrapper, but then pkg-config complains that it cannot find my systemwide libraries. Do I have to write my own pkg-config wrapper which calls something like this?

#!/bin/sh
# file: ${PREFIX}/bin/pkg-config
PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${PKG_CONFIG_PATH_x86_64_unknown_linux_gnu} \
   pkg-config-unwrapped

…and stash that into my $PATH?

Perhaps the Nix way of working is to package all of my dependencies as derivations, and always enter an environment via nix-shell. That sounds awesome, and it will probably also allow me to build the whole environment with ASAN or clang’s libstdc++ (which only works on my old system because I just happened not to use any systemwide C++ library). That would be super cool. However, I’m often working on these libraries themselves, I’m editing them in my text editor, and for that it is very useful to have the compilation database so that my editor can offer tab completion and what not. Can I instruct nix-shell and/or nix-build to keep the build directory around, so that I still have access to the compile_commands.json? Should I put some src = stdenv.fetchGit { url = "file:///path/to/my/checkout"; } override blocks into the nix-shell file that I’m using? How do I inform nix-shell that I’ve changed these revisions on disk, and that the dependencies should be rebuilt?

How do I do this while I’m actively developing my dependencies? I guess I’ll have to commit every time I want to regenerate my shell environment, right?

How are people solving these problems on NixOS?

1 Like

When I do C++ developing I create a shell.nix file inside the project folder like this:

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {
  name = "cpp project";
  buildInputs = with pkgs; [
     cmake
     ninja
     pkg-config
     swig
     pcre
     bison
     flex
     openssl
  ];
  
  shellHook = ''
   # Some bash command and export some env vars.
   export FOO=BAR
   echo "Starting new shell";
  '';
}

To activate it I use the command nix-shell. It’ll start a new shell with all packages. You can create more than one nix shell file, just active the one you want running nix-shell <nix_shell_file>. If you need to export or run some commands when start the shell use the shellHook. Now you are in the shell, run the command you need, like the cmake.

3 Likes

Thanks, @brogos, but that only explains how to prepare a build environment for a single package. The is a problem with this approach: I cannot extend this environment in a way that pkg-config recognizes due to the way the pkg-config-wrapper works. It just silently overwrites my own $PKG_CONFIG_PATH. How do you handle a mixture of systemwide C++ packages and packages that you’ve built yourself? How do you make pkg-config recognize them all?

1 Like

@jkt if you’ve define your packages as overlays, you can import them this way:

{ pkgs ? import <nixpkgs> { overlays = [ (import ./nix_files/overlay.nix) ]; } }:

maybe you can get your shell environment clear from the system wide stuff with the pure option:

nix-shell --pure

It’s a bit inconvenient though, you need to explicitly declare all systempackages that you do need to include in every project. But then you have full control over the environment.

@KernelPanic, well, I think that the --pure option is not what I’m missing. I do not have a problem that I see too much stuff within my nix-shell session. I’m also happy that I can do nix-shell -p libssh to get access to the libssh library in my build environment, and I like this modularity. What I am not happy with is the fact that Nix’s pkg-config wrapper doesn’t let me extend the list of places to look for the *.pc files via the standard $PKG_CONFIG_PATH env variable.

If I’m simultaneously working on libA, which needs OpenSSL, and projectB, which needs libA and libssh, then that’s a workflow where Nix is making my experience rather complicated. Of course I’m already starting my shell via nix-shell -p openssl libssh, but the problem is that I don’t know how to build my own libA within this nix-shell environment in a way that a subsequent call to pkg-config will find it. Once again, I need pkg-config to recognize and find openssl, libssh (both from Nixpkgs and enabled via nix-shell -p ...) and my libA that I just installed into my own location at the same time.

Try this @jkt:

  1. Create a shell.nix inside your source code:
{ pkgs ? import <nixpkgs> { } }:

pkgs.mkShell {
  name = "cpp_project";
  buildInputs = with pkgs; [
    cmake
    ninja
    swig
    pcre
    bison
    flex
    openssl
  ];

  nativeBuildInputs = [ pkgs.pkg-config ];

  shellHook = ''
    export PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$HOME/project/libmanolo/"
  '';
}

Change buildInputs to your needs. Keep pkgs.pkg-config in nativeBuildInputs. In shellHook you put any variable you want to export or some commands you want to run on the creation of the new shell. In this example I’m extending $PKG_CONFIG_PATH.

  1. Inside the folder with the shell.nix file run the command nix-shell.

  2. If you print PKG_CONFIG_PATH it’ll show you that PKG_CONFIG_PATH was extended with $HOME/project/libmanolo/.

echo $PKG_CONFIG_PATH 
/nix/store/izfyh5hr9as4ihx2vwi2yrnkiz6gn4qi-pcre-8.44-dev/lib/pkgconfig:/nix/store/lwcrmj44j6s5ww3j0ybar2jc7kf9ddzq-openssl-1.1.1g-dev/lib/pkgconfig:/nix/store/izfyh5hr9as4ihx2vwi2yrnkiz6gn4qi-pcre-8.44-dev/lib/pkgconfig:/nix/store/lwcrmj44j6s5ww3j0ybar2jc7kf9ddzq-openssl-1.1.1g-dev/lib/pkgconfig:/home/thiago/project/libmanolo/
1 Like

@brogos, thanks, that did the trick. Apparently the shellHook can extend these variables just as you’ve shown, and the wrappers such as pkg-config then respect these overrides. Awesome, thanks!

1 Like

Also, since you’re new, I’d highly recommend taking a look at direnv. I started using it a bit late and regretted it.

Its 3 steps.

  1. Install direnv.
  2. Add a line to shell to hook into cd.
  3. Add a .envrc to your project to say use nix.

There are also (unofficial, and IMHO imperfect) solutions to persistence so your cd is instant.

2 Likes

@jkt good to know that it helped you.