Github Actions and/or/with Hercules CI

I have a few mildly complex projects that use nix as their build system (some using non-standard targets or compiler toolchains, some mixing build products from different architectures, some using Import-From-Derivation), and spent some time thinking about the ideal CI system.

Because of the custom dependencies, any CI system must use and fill a custom nix cache. But thanks to the great Cachix, that part is solved.

I looked at @roberth’s Hercules CI and set it up for one of my open source projects. The setup was smooth, I was able to quickly run a runner on one of my Debian servers, and it it mostly did its job. But I quickly noticed that while it was great at building derivations and filling the nix cache, I was rather limited in my control of what to build, and what to do after the build.

So I mostly moved back to plain Github Actions (with cachix, of course). This would allow me to do things like

  • Run a set different set derivations depending on, say, Github labels.
  • Do “differential CI”, where I compare the build output against the base branch (imagine a compiler project, and any PR with label refactoring should still produce identical output on the test files)
  • Have a wealth of Github Action to choose from to do something after the build (post comment, create Github releases, push code lint fixes to the branch, upload documentation etc.).

I saw that Hercules CI has a beta feature called Effects, but it seems like it would require a lot reengineering to be able to do what I can do with Github Actions.

So pragmatically, I am currently inclined to got with Github Actions.

Yet it pains me, because when it comes to the “build this derivation” part, it’s just so dumb:

  • A simple nix-build -A foo will, if the output is in the cache, still download it for no good reason, instead of just reporting success. (This part can probably be improved.)
  • Two jobs in parallel that build the same or overlapping derivations will both build it, instead of one waiting for the other.
  • Parallelism is limited to the Github runner
  • A dedicated Hercules CI runner will likely have a warm /nix/store, making builds much faster

So what I think I really want is a Github Action step “Build on Hercules CI”, which builds a given path and attribute on Hercules CI, failing if it cannot be built. This would give me most of the benefits of Hercules CI, and allow me to combine them with most of the benefits of Github Actions. Would that be feasiable?

Taking this thought one step further, it seems that Hercules CI is, in a way, just a smarter remote builder. So maybe one can have a hci build command that works like nix build, but builds stuff on Hercules CI, which may be useful even for local development sometimes?

1 Like

Hi @nomeata,

I have to point out that I’m no longer associated with Hercules CI as of the beginning of 2020.

Those are excellent questions and I can clarify around GitHub Actions:

  • A simple nix-build -A foo will, if the output is in the cache, still download it for no good reason, instead of just reporting success. (This part can probably be improved.)

@Mic92 wrote a utility for this GitHub - Mic92/nix-build-uncached: A CI friendly wrapper around nix-build..

It would be really great to have this feature as part of Nix: Make `nix-build-if-changed` a first class feature of nix · Issue #4364 · NixOS/nix · GitHub & Avoid substituting top-level attributes if that's not needed (CI mode) · Issue #3428 · NixOS/nix · GitHub

  • Two jobs in parallel that build the same or overlapping derivations will both build it, instead of one waiting for the other.

You can order jobs if they depend on each other, but there’s no way around this (see the next answer).

  • Parallelism is limited to the Github runner

GitHub plans to allow different sizes of the runner.

  • A dedicated Hercules CI runner will likely have a warm /nix/store , making builds much faster

Yes! You can self-host GH runner but at that point, you’re better of with a better UI and first-class support for IFD going with Hercules CI.

I do have some ideas to speed up warming up the cache with some improvement to Cachix (mostly waiting on Nix 2.4 release).

1 Like

Thanks! I’ll definitely look at nix-build-uncached once that itch becomes itchy enough. (I also edited my post to correctly mention @roberth.)

Hi @nomeata,

It’s good to hear your experience with Hercules CI.

You can run anything in the Effects sandbox, but you can’t respond to other events than pushes yet.
So far, Hercules CI has aimed to be a simple but well-integrated solution for CI and CD (beta). More flexibility will be added over time.

That’s certainly an important part of it. (Ignoring the neat Nix integration, dashboard, CD features, etc. , but I’ll follow along :slight_smile: )

Something like this could make its way into a future version.

If you want to stay up to date about these features, you can subscribe to the issues:

2 Likes

Wells the nix integration and dashboard is what makes it a great smart remote builder! :smiley: