In our organization, we have multiple Nix build servers that are used via different frontends (Hydra, Gitlab, remote builds). Our goal is to have a central binary cache for everything these build servers produce. As far as I can tell, there doesn’t really seem to be a good solution for this and we are looking for the best way to solve this problem.
We’ve come up with two general stragegies:
In the pull model we would set up a server as the binary cache that knows all builders and uses a cron job to
nix copy --all store paths from each builders. This feels like it would work, but also a bit hacky and not necessarily scalable.
In the transitive build model we would hide all actual build servers behind a central “gateway” server that is used for all Nix building. This central gateway would distribute jobs to the actual build nodes and at the same time act as the central binary cache, because all store paths pass through it. This is a bit the nixbuild model and it is nice, because the actual build servers can be scaled up or down. My question here would be whether Nix actually forwards remote build requests or whether we would have to develop custom tooling here.
Does anyone have experiences with building scalable build setups with Nix and if so, what would you propose as best practice here?