We just released yensid, a proxy for remote builders that:
- does global load balancing, not just load balancing based on individual clients
- has various possible load balancing strategies, as well as a fully customizable (in lua) option. You can based load balancing on actual CPU, disk, and memory usage, for example, and even dynamically change builders (if you want to implement autoscaling)
- has optionally short-lived SSH-based certificates for remote builders so that adding or removing builders is easy, and doesn’t involve coordinating all clients (and so key compromises are simpler to handle). There are modules that do all the renewal logic for you, so you just need to enable it in your builders.
In addition to the repo, we also have a short blog post describing it. The repo has the entire infra as VMs, so you can play with it locally (and conceivably even benchmark different load balancing strategies). It also comes with a NixOS configuration that you can deploy in 5 minutes with garnix. You can even just run it on your own dev machine instead of deploying it somewhere (though then you would need to do a bit more work to have global load balancing).
Let me know if you have any questions or thoughts!