Flake check integration with other systems?

Development
1

Right now I’m using nix flake check in both Jenkins and GitLab CI and in each case the result is a bit of an opaque yea/nay popping out the end, with little indication of what went on or what was actually validated.

Has there been any work on a more transparent “executor” for flake check tests? Specifically, I’m thinking of stuff like:

  • Evaluate the flake and emit a list of per-check invocations, so that an outer CI system could spin off separate jobs that run each check in parallel with their own separate log, timeout, status, etc. (See for example GitLab’s dynamic child pipeline capability: https://www.youtube.com/watch?v=C5j3ju9je2M)
  • Run all the checks together, but generate a JUnit XML report that lists the names of the checks that were run and populates each with the correct log of what their output was. Unfortunately the particulars of the XML format are far from standardized, but nonetheless there are common elements that basically every CI system knows how to display in their UI.

If nothing like this exists, is it something there would be interest in?

2

What speaks against you building an executable that does all you want, and running it with nix run?

There has been recent discussion around whether Nix should even do or know all these application-level things. In my opinion: no, it’s a maintenance liability. Remember, derivations are just cached execve, that’s essentially all Nix is about.

2 Likes
3

Oh yeah, I’m definitely not expecting Nix itself to support something like this— I’m picturing an external executor that uses nix eval --json, nix log, and similar to do this integration. And perhaps that’s trivial enough that I really should just build it myself, but I was curious if there was anything out there already.

4

If you don’t mind GitHub Actions example this is a repo that does a bit of this:
https://github.com/nix-community/NixOS-WSL/actions/runs/7604952861

I did test something similar for self hosted GitLab, but I found speed was more important than visibility. With the dynamic jobs having a lot of overhead making things much slower.