I keep tiptoeing around Hercules CI. A bunch of my projects currently use Github Actions + Cachix, and could well benefit from the better ergonomics of Hercules CI.
But having to host my own runner somehow always put me off. I am trying to maintain less services on my machines, not more…. Also, for some one of my projects where there is a slight hope that a company might pay for CI resources, I’d have a much easier time telling them to simply sign up and pay, rather than having to explain that they need to maintain a build host (no matter how simple in practice).
Now I stumbled over nixbuild.net, which seems to provide exactly the missing bits: A pay-as-you-go nix-builder-as-a-service offering!
But it seems that they can’t directly talk to each other. Seems like a missed opportunity to me? Maybe nixbuild.net might be interested in providing native support for running the Hercules CI agent?
(Also, darwin support on nixbuild.net might be needed for some of my projects.)
Hercules CI is still two or three significant features away from being a fully hosted service, agent-wise, but I’ve been considering to offer managed agents in the meanwhile.
Hosting the agent yourself is about as easy as it can get, as they’re stateless and require little configuration, all of which can be installed with agenix or similar. It’s designed to be simple and fully adapted to NixOS and nix-darwin. I suppose build servers have some stigma around them? Anyway, I shouldn’t be telling you what you want , so:
You can PM me if your team is interested in managed agents. Same for anyone else who prefers not to run their own agents, hit me up!
But it seems that they can’t directly talk to each other.
Building is only one part of what Hercules CI does. Most significantly, it also evaluates and runs effects, which are a way to run deployments and such, with access to configured secrets and the network.
If nixbuild.net is up for it, I’ll be happy to help them.
As Robert points out, the Hercules agent performs evaluation and runs effects. Those things are tricky to outsource to nixbuild.net, since they depend on local context. Arguably, one of the nice selling points of Hercules is that it keeps evaluation local and therefore makes it easier to use secrets and internal stuff.
However, running your own sizeable Nix build farm is simply not a fun thing to do. You’ll soon run into problems with auto-scalability, copying closures back and forth between builders, garbage collection etc. That is why I built nixbuild.net
If you want to use Hercules but avoid maintaining build servers, my recommendation would be to provision a small/cheap server where you run the Hercules agent and then configure it to send all builds to nixbuild.net.
Alternatively, if you really would like to avoid self-hosting Hercules, maybe it somehow would be possible to run the Hercules agent in GitHub Actions? @roberth Have you thought anything about that scenario? I agree that it is probably a weird thing to do because I assume the agent is meant to continuously poll the Hercules server for work? But, at least in theory, the agent could trigger jobs and push results to Hercules on its own, when it starts inside a GitHub Action… People already trusts GitHub with their sources so it wouldn’t be far-fetched to run evaluation and effects in GitHub, and then get the great dashboard and Nix integration on Hercules CI.
As Robert points out, the Hercules agent performs evaluation and runs effects. Those things are tricky to outsource to nixbuild.net, since they depend on local context.
I admit I keep ignoring effects so far, but at least evaluation should, ideally, not depend too much on local context, shouldn’t it (assuming the projects build pin their nixpkgs dependency etc.)?
I would find that ideal. Especially if I am already used to putting secrets into github actions, and I want to use effects from github actions (e.g. commenting on github, pushing code fixed or github pages) after having built something.
What would be left of Hercules CI then, though? The nice dashboard certainly. Nice reporting of build status, I guess. And the deduplication of concurrent builds. (Although, does nix build --store ssh-ng://eu.nixbuild.net deduplicates concurrent builds?)
Yeah, you are correct that evaluation can be done remotely. Hydra does it. And there’s no problem running Nix evaluations in GitHub Actions. But I don’t think evaluation is in the scope of nixbuild.net for the moment. Evaluation would have to be “opinionated” to a certain extent, and I much rather let users decide how to evaluate their Nix expressions and then just use nixbuild.net (or any other Nix machine) for running the builds.
Yes, all builds are deduplicated on nixbuild.net no matter how you run them. You really can look at nixbuild.net as a single Nix machine, and use it as such. This is what makes it so nice to use from GitHub Actions, for example. You can run many concurrent jobs in GHA, all evaluating different (or the same) Nix attributes and then sending off builds to nixbuild.net. No duplicate work will be done, all dependencies of the complete set of Nix builds will be built in the correct order without you having to think about it at all. You can even mix in interactive use of nixbuild.net at the same time, they will wait on any running CI builds if it turns out they need to build the same derivations.
I’m coming back to this because I am experimenting with garnix.io right now, which provides the kind of fully integrated CI experience that I find very attractive (and it supports Darwin, which is crucial for one of my open source projects).
But Hercules-CI seems to have a nicer dashboard, and nixbuild.net has the appeal that I can use it locally too, among other goodies, and a fair and sustainable cost model.
I still think that both Hercules-CI and nixbuild.net would benefit from more adoption if there was a way to get the two to work together without people having to run their own agent.
, so: