Populating binary cache from multiple build servers

Help
1

Hi!

In our organization, we have multiple Nix build servers that are used via different frontends (Hydra, Gitlab, remote builds). Our goal is to have a central binary cache for everything these build servers produce. As far as I can tell, there doesn’t really seem to be a good solution for this and we are looking for the best way to solve this problem.

We’ve come up with two general stragegies:

In the pull model we would set up a server as the binary cache that knows all builders and uses a cron job to nix copy --all store paths from each builders. This feels like it would work, but also a bit hacky and not necessarily scalable.

In the transitive build model we would hide all actual build servers behind a central “gateway” server that is used for all Nix building. This central gateway would distribute jobs to the actual build nodes and at the same time act as the central binary cache, because all store paths pass through it. This is a bit the nixbuild model and it is nice, because the actual build servers can be scaled up or down. My question here would be whether Nix actually forwards remote build requests or whether we would have to develop custom tooling here.

Does anyone have experiences with building scalable build setups with Nix and if so, what would you propose as best practice here?

Thanks!
Julian

2 Likes
2

I guess you could try setting max-jobs = 0 in /etc/nix/nix.conf on the gateway and hope that it will forward them. It will definitely refuse to perform local builds.

3

If you’re willing to give Cachix a try:

Using cachix watch-store it will watch /nix/store for new paths and push them to your binary cache.

It does that by using two queues, one to mass query for missing narinfos and one for pushing each store path.

If you’re running NixOS:

 systemd.services.cachix-watch-store = {
    description = "Cachix store watcher service";
    wantedBy = [ "multi-user.target" ];
    after = [ "network.target" ];
    path = [ config.nix.package ];
    environment.XDG_CACHE_HOME = "/var/cache/cachix-watch-store";
    serviceConfig = {
      Restart = "always";
      CacheDirectory = "cachix-watch-store";
      ExecStart = "${pkgs.cachix}/bin/cachix watch-store mycache";
    };
  };

You’ll also need to export $CACHIX_AUTH_TOKEN secret :slight_smile:

1 Like