Implementing ccache in Flake-based configurations

Help
1

NixOS offers the use of ccache for building packages by configuring programs.ccache.packageNames, which will in turn configure nixpkgs.overlays. This will replace the pkgs attribute of the system configuration with a new one that now also includes this ccache overlay.

In my Flake-based configuration, I prefer to import nixpkgs as rarely as possible and configuring ccache this way would mean that I would add another evaluation of nixpkgs to my build.

Right now, I have something like this in my flake.nix:

...
let
  system = "x86_64-linux";
  pkgs = import nixpkgs {
    inherit system;
    config = {allowUnfree = true;};
    overlays = [overlay1 overlay2 overlay3 ...];
  };
in {
  nixosConfigurations = {
    myHost = nixpkgs.lib.nixosSystem {
      inherit system;
      inherit pkgs;
      modules = [
        # ...
      ];
    };
  };
}
...

If I wanted to add a ccache overlay here, I would probably create an overlay similar to this:

  (self: super: {
    ccacheWrapper = super.ccacheWrapper.override {
      extraConfig = ''
        export CCACHE_COMPRESS=1
        export CCACHE_DIR="/nix/var/cache/ccache"
        export CCACHE_UMASK=007
        if [ ! -d "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' does not exist"
          echo "Please create it with:"
          echo "  sudo mkdir -m0770 '$CCACHE_DIR'"
          echo "  sudo chown root:nixbld '$CCACHE_DIR'"
          echo "====="
          exit 1
        fi
        if [ ! -w "$CCACHE_DIR" ]; then
          echo "====="
          echo "Directory '$CCACHE_DIR' is not accessible for user $(whoami)"
          echo "Please verify its access permissions"
          echo "====="
          exit 1
        fi
      '';
    };
    ffmpeg = super.ffmpeg.override { stdenv = super.ccacheStdenv; };
  })

Using this overlay would make assumptions about the target system, like the presence of and access to /nix/var/cache/ccache.

Additionally, we would need to add this path to nix.settings.extra-sandbox-paths, which would mean that every host needs to have an additional option here.

Usually I would solve issues like these using common NixOS modules, but we are dealing with importing nixpkgs here before we even declare a NixOS configuration.

I also wonder if I would be creating a chicken and egg problem here, as a (new?) system, that doesn’t yet include its cache path in the Nix sandbox wouldn’t be able to build overidden packages until nix.settings.extra-sandbox-paths has been adjusted.

How are other people dealing with this, or are there maybe alternative approaches?

2

No it doesn’t. A NixOS configuration imports nixpkgs just once, using the arguments in the nixpkgs.* NixOS options, including nixpkgs.overlays. The overlays argument just injects functions into the fixed point of nixpkgs and doesn’t result in re-importing or re-evaluating nixpkgs as a whole.

3

I wasn’t even sure if the nixpkgs.* options work at all in my setup.

I wanted to try a minimal example of this by overriding two packages that I added using an overlay in my Flake like this:

...
let
  system = "x86_64-linux";
  pkgs = import nixpkgs {
    inherit system;
    config = {allowUnfree = true;};
    overlays = [prismlauncher.overlays.default my-custom-overlay ...];
    # pkgs.prismlauncher and pkgs.linux_zen_scrumplex
  };
in {
  nixosConfigurations = {
    myHost = nixpkgs.lib.nixosSystem {
      inherit system;
      inherit pkgs;
      modules = [
        # ...
      ];
    };
  };
}
...

{...}: {
  programs.ccache = {
    enable = true;
    packageNames = [
      "prismlauncher"
      "linux_zen_scrumplex"
    ];
  };
}

After switching to the new configuration, both prismlauncher and linux_zen_scrumplex were not rebuilt, even though the inputs changed, which should obviously result in a rebuild.

5

The overlays go in the nixpkgs.overlays options in your NixOS module. You do not want to write pkgs = ... and then inherit pkgs in the nixosSystem call. You want one of the modules in modules = [ ... ] to include nixpkgs.overlays = [/* the overlays you want */]

let
  system = "x86_64-linux";
in {
  nixosConfigurations = {
    myHost = nixpkgs.lib.nixosSystem {
      inherit system;
      modules = [
        {
          nixpkgs.config = {allowUnfree = true;};
          nixpkgs.overlays = [prismlauncher.overlays.default my-custom-overlay ...];
        }
        # ...
      ];
    };
  };
}

(Or move those nixpkgs.* lines into the same file as the programs.ccache lines)

What you did almost works, because you can just unilaterally set the pkgs for a nixosSystem by passing it a pkgs argument. But that’s really not the best way. When you leave that out, NixOS instantiates pkgs itself using the NixOS module options nixpkgs.* This allows the modules to set custom nixpkgs arguments like overlays. And the programs.ccache option you’re trying to use is implemented by a module that uses the nixpkgs.overlays option. By passing pkgs to nixosSystem, you’re telling it to ignore the nixpkgs.overlays option and just use whatever pkgs you gave it instead, negating the modules trying to set overlays.

6

Personally, I prefer instantiating pkgs in the flake since I will usually need it in other parts of the flake, e.g. devShells and want to avoid instantiating it multiple times. It also feels cleaner from an architectural point of view (inversion of control/dependency injection):

...
let
  system = "x86_64-linux";
  pkgs = import nixpkgs {
    inherit system;
    config = {allowUnfree = true;};
    overlays = [prismlauncher.overlays.default my-custom-overlay ...];
    # pkgs.prismlauncher and pkgs.linux_zen_scrumplex
  };
in {
  nixosConfigurations = {
    myHost = nixpkgs.lib.nixosSystem {
      inherit system;
      modules = [
        nixpkgs.pkgs = pkgs;
      ];
    };
  };
}
...

Of course, the downside is that most nixpkgs.* NixOS options will not work any more. overlays are one of the exceptions.

7

I would also prefer this latter solution.

It also seems to work this time.

Though one question still remains for me:

If I bootstrap a system using my flake, the ccache directory will not be present yet, so any package that uses ccache will fail to build. I assume there isn’t really much I can do here, though?