We need a binary cache which is very reliable.
We thought about this structure:
We run 3 VMs in three different providers/datacenters.
They should not share data.
nixcache1.example.com
nixcache2.example.com
nixcache3.example.com
Question 1: cachix or attic or something else?
Question 2: How to sync the three caches?
We could do attic push three times, but caches could get out of sync easily.
What do you mean by “They should not share data.” when you want the caches to be in sync?
I implemented something like that. What I did is push to all the mirrors synchronously, but in case that fails, each mirror also syncs with the others at an interval.
I did that in a bit hacky way, I essentially do ‘ls /nix/store’ (or rather something like this: cachix-action/dist/list-nix-store.sh at ee79db531bfb851a7cf47035057b7cca88d26449 · cachix/cachix-action · GitHub) on both the source and target, compute the diff, then use ‘nix copy’ to send the missing store paths.
With “They should not share data.” I mean: Each VM should have its own storage (disk is enough). For example no shared s3 storage.