Help using ccache for kernel build

Help
1

I’m exploring using NixOS systems for Linux kernel benchmarking. So far I’ve arrived at this flake which I am able to build from a non-NixOS system. I’m planning to use linuxPackages_custom and just point it at my own kernel branch.

Now, I expect that very often I will be comparing two kernel builds that differ only in one or two .c files, which means an incremental build of the second would normally be fast. But IIUC with Nix in principle the whole thing has to be built from scratch each time.

I think the best workaround for this is ccache (please interject if there’s some better way to share incremental results between builds). The Wiki seems to suggest:

  1. Add extra-sandbox-paths = /nix/var/cache/ccache to nix.conf
    2… Add nixpkgs.overlays = ... to… something
    3… Then there’s a separate section titled “Derivation CCache” which has a different example of overriding nixpkgs.overlays.

Apologies, I’m completely lost here, I’m afraid I can’t really ask good questions so I have to just spray out a bunch of vague ones:

  • Does part 1 mean update your system’s /etc/nix/nix.conf, and if so doesn’t that mean your global Nix configuration now has a coupling with your Flake
  • What is the relationship between parts 2 and 3, are they alternative ways to achieve the same thing, do they achieve separate things, or are they two complementary steps for achieving a single thing? What does “Derivation CCache” mean?
  • … and how do I apply this to my Flake?
2

I still haven’t figured this out but I think when I wrote this, my brain was a bit fried, coming back to it, things seem nowhere near as abjectly unknowable as they did before! The explanation of overlays here is pretty helpful.

I think the ccacheWrapper thing is configuring the details of the ccacheStdenv which is a feature of nixpkgs that gives you a replacement of the stdenv where ccache is enabled. I don’t know why it requires this extra configuration. My best guess is that it’s because it’s a little bit hacky due to the coupling with /nix/var/cache/ccache that I mentioned before?

I found this example on GitHub, I was able to cargo-cult that by just adding this into one of my NixOS modules:

  nixpkgs = {
    config = { replaceStdenv = { pkgs }: pkgs.ccacheStdenv; };
    overlays = [
      (final: prev: {
        ccacheWrapper = prev.ccacheWrapper.override {
          extraConfig = ''
            <omitted for brevity>
          '';
        };
      })
    ];
  };

And I believe that replaceStdenv biz then just tells nixpkgs to build everything in the ccacheStdenv. When I then ran my build again, it started compiling what I assume was literally all of NixOS. Aside from being quite a burden on my poor little 5800X, some derivations failed, e.g. alsa gave me a configure: error: C compiler cannot create executables. The fact that stuff broke seems like evidence that I actually did something with my change.

So, I think my next job is just to get this to apply only to the kernel build. My guess is that the “Derivation CCache” bit in the wiki is illustrating how you’d do that for ffmpeg (i.e. instead of using replaceStdenv at the top level you just set stdenv in the package-specific overlay).

So, now I think the only barrier left is that the kernel package build is rather special and wrapped up in various layers that I don’t understand. So I probably “just” have to understand those layers so I can figure out how to do the same thing there.

3

OK yeah so I followed the definition of the kernel packages in nixpkgs and it turns out the buildLinux function accepts a field called stdenv, and if I set that to pkgs.ccacheStdenv, it just gets applied directly to the kernel.

The immediate result is a bit confusing, because that wrapper script that’s embedded in the extraConfig (omitted in my comment) doesn’t behave anything like a C compiler if it doesn’t see /var/cache/ccache, so the kernel build system says “sorry, your C compiler isn’t supported” and you don’t actually get the nice helpful error message.

Anyway, once I created it and set extra-sandbox-paths = /var/cache/ccache in /etc/nix/nix.conf, it started building the kernel. Weirdly, the ccache statistics don’t show much happening:

❯❯  CCACHE_DIR=/var/cache/ccache sudo ccache -s
Local storage:
  Cache size (GiB): 0.0 / 5.0 ( 0.00%)

But I can see files appearing in /var/cache/ccache… so hopefully something is working.

So yeah, some remaining awkwardness, i.e. now my flake won’t compile without setup on the build host, and in that case it prints an extremely unhelpful message. But, it seems like it should be achievable to make the ccache usage optional, and print a more helpful message. In the meantime maybe I have something that Works On My Machine.

Here’s the commit where I think I got things working.