Nix CI (besides Hydra and Hercules)

Is there any Nix-aware CI besides Hydra and Hercules?

Or maybe plugins for Jenkins or GitLab?

By Nix-awareness I mainly mean making a graph of native tasks from nix-instantiate and distribute them to CI workers, and showing those graphs as a single CI-tasks to avoid pollution of the root namespace. Plus import of Nix build logs into CI system.

Why not Hydra/Hercules: to integrate with things unrelated to Nix (uploading to Maven, etc). It seems easier to add Nix support to Jenkins than to enrich Hydra with all it lacks.

4 Likes

I was thinking about this while designing a build/ci/cd library in elixir.

Iā€™ve also thought about just running a script in a tool like Jenkins or Gocd that would effectively do something like ā€œif .nix file, run nix-buildā€ and at least with gocd you can tie a pipeline to a build machine that has nix installed.

In one project I worked on we used circleci to do these nix builds and it worked.

Ultimately we ended up creating a *-nixpkgs pattern, and using hydra, because we were building a whole Nixos release.

1 Like

Another idea is to use hydra, and have other build tool check and trigger on successful build to get and upload artifacts to maven etc hydra has an api that can both trigger jobs and check job status

> Or maybe plugins for Jenkins or GitLab?

Nix is typically used to avoid using systems like Jenkins with that despondent butler! Writing a plugin for Jenkins would be irony I think :cowboy_hat_face:

However, more CI that is Nix-Aware is a good thing, no matter what.

BuildKite pipelines can be updated dynamically so itā€™s possible to generate a nix build graph on the fly.

How I see it, there would be a pipeline evaluation step that pushes all the sources to the binary cache and generates the build graph. As an optimization, it could only schedule jobs for derivations that donā€™t have results in the cache.

README.md - depot - Sourcegraph does something like that but I havenā€™t looked into their implementation specifically.

4 Likes

It is still highly experimental, but we are working on bitte-ci at work to run nix ci jobs on a nomad cluster.

3 Likes

In the nix build ā†’ graph ā†’ jobs for other CI platform modelā€¦

are you re-implementing scheduling? How do you handle a toplevel of hundreds of derivations, some of which rely on another? Am I over/under-thinking this?

I think the scheduling is usually handled by the platform (letā€™s say Jenkins, Gocd, etc)

Basically, if you have a nixos based build node, then nix-build will take care of these issues you mention, you run nix-build as task on the machine against the nix expressions. This doesnā€™t take care of where you will cache your binaries, etc. But nix-build on a collection of nix expressions will take care of all dependencies, etc.

Gitlab CI has a feature to build pipelines dynamically based on scripts nowadays: https://docs.gitlab.com/ee/ci/pipelines/parent_child_pipelines.html#dynamic-child-pipelines.

I have yet to use it but it seems decent enough.

1 Like

If this is in response to my post on bitte-ci, I can answer a few points. So scheduling is handling by nomad itself, it does a good job of that on itā€™s own. We also made a special patch to nomad so it understands nix flakes and can build an isolated environment around a flake attribute. In order to pass in multiple required derivations, we recently expanded this patch to accept multiple flakes. Really, all the heavy lifting was done by @manveru. Iā€™m mostly just a consumer :sweat_smile:

2 Likes

Did they mean deployment scheduling or ci job scheduling. Does bitte-ci actually manage the scheduling of builds and testing with nomad too?

I guess so now that I re-read with that context in mind. Very interesting

Let me try again:

Something is going to convert a nix build with N derivations into M jobs on the underlying platform, no? Hydra knows about itā€™s runners, evals, and divides those derivations somewhat intelligentally across runners.

Is there not a need for a similar component in an imagined layered-CI system?

edit: for example, building my toplevel is 200 derivations. Hydra might fan those out over my x86-64 nodes. Does bitte-ci do this somehow? I guess I sort of imagine that it must make one flake output = one nomad job?

(again I may be over thinking this)

2 Likes

Gitlabā€™s dynamic parent-child pipelines are indeed interesting, but from the docs I see that only two level of parent-child relationships are permitted, seems like bummer to me. Iā€™m not a really expert in gitlabā€™s pipelines (although Iā€™m using them) and while the problem of knowing what to build before can be solved with something like a toposort, handling the parallelization and requirements without a hierarchical structure seemsā€¦ hard. I would be very interested though, because gitlab runners then can be arranged in various way, as native services or even kubernetes jobs (and maybe even nomad tasks).

Iā€™ve not come across buildkite at all, is a something you can deploy yourself and self host, or just another platform ā€˜trust usā€™ scenario? which is find depending what your use case is.

Hydra seems to be everything you need to get distributed builds working well, well until trustnix gets going.

buildkite is similar to hercules-ci in that regard. The user self-hosts the agents and machines. The agents connect to the SaaS API, which hosts their proprietary scheduler and web UI.

1 Like

ok, that makes senseā€¦ thanks for your clarifcation my llama friendā€¦, your not working on a sunday are you?

OT but no, Nix doesnā€™t count as working :grin:

1 Like

i think i know what you meanā€¦

itā€™s easy to become ā€˜addicted to nixā€™.

probably needs a cover doing for the next nix con, if thereā€™s any musicians among us.

1 Like