Per file derivations with C++

Growpotkin · May 31, 2022, 1:23am

I have a science experiment to share that I imagine some of you will enjoy.

Recently I’ve been trying out writing derivations directly, and limiting my reliance on Nixpkgs ( within reason ). Largely this is motivated by a need for fine grain caching at work and also making Nix more approachable to coworkers by avoiding the complexity of setup.sh.

I wanted to try getting real pedantic with derivations, and use them to process commands like cc, ln, ar, and cp directly. What I cooked up was a minimal set of helpers for C++ builds.

I took it for a spin on a relatively simple set of packages to produce static and shared libraries, as well as dynamic and static executables.

These depend on a mixture of Nixpkgs and “from scratch” libs.

The scale here is too small to make any sweeping performance claims, but because it’s this fine grained, changes to sources only trigger rebuilds of the effected files. It would be easy to extend this to mark individual headers as inputs as well using cpp traces, and to library linkage by using traces in ld ( there’s potentially a large performance advantage for linkage ).

The README in the repo covers limitations and possible extensions, but overall I’m really pleased with this little experiment.

Here’s the helper file:

github.com

aakropotkin/gourou-nix/blob/2a347f4194a25caa11d25b6e1a4342a6be9d3748/support.nix

{ stdenv, linkFarmFromDrvs }:
let

  baseName = p: with builtins;
    let
      bp = baseNameOf ( toString p );
      isBaseName = mb: ( baseNameOf mb ) == mb;
      isNixStorePath = nsp:
        let prefix = "/nix/store/"; plen = stringLength prefix; in
        prefix == ( substring 0 plen ( toString nsp ) );
      removeNixStorePrefix = nsp:
        let m = match "/nix/store/[^-]+-(.*)" ( toString nsp ); in
        if m == null then nsp else ( head m );
    in baseNameOf ( removeNixStorePrefix ( p ) );
  baseName' = p: builtins.unsafeDiscardStringContext ( baseName p );

  baseNameOfDropExt = p: builtins.head ( builtins.split "\\." ( baseName' p ) );
  objName = file: ( baseNameOfDropExt file ) + ".o";
  objNames = sources: builtins.concatStringsSep " " ( map objName sources );

This file has been truncated. show original

And my example package :

github.com

aakropotkin/gourou-nix/blob/2a347f4194a25caa11d25b6e1a4342a6be9d3748/flake.nix

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils/master";
    gourou-src = {
      #url   = "github:BentonEdmondson/the-scroll/main";
      url   = "git://soutade.fr/libgourou.git";
      type  = "git";
      ref   = "master";
      flake = false;
    };
    updfparser-src = {
      url   = "git://soutade.fr/updfparser.git";
      type  = "git";
      ref   = "master";
      flake = false;
    };
    base64-src = {
      url = "git+https://gist.github.com/f0fd86b6c73063283afe550bc5d77594.git";
      flake = false;

This file has been truncated. show original

edolstra · May 31, 2022, 8:46am

You may be interested in https://github.com/edolstra/nix-ccache, which provides a gcc wrapper that runs the input through the preprocessor and then uses a Nix derivation to perform the actual compilation. This enables caching and distributed builds similar to ccache (hence the name - it doesn’t actually use ccache).

There is also the ancient https://github.com/edolstra/nix-make which provides functions for building C/C++ source files, using some import-from-derivation magic to figure out the header file dependencies automatically.

Growpotkin · May 31, 2022, 4:15pm

Thanks for pointing these out. I’ve seen nix-ccache floating around the repo list for ages but never checked it out, and this is the first time I’ve seen nix-make.

I’ll take the dive into those soon. I imagine some of my symbol table and linkage analysis scripts could fit right in with nix-ccache.

What I really want one day is for Nix to dry run cpp and ld to create derivations - and it sounds like nix-ccache has cpp already covered.