Right now I’m using nix flake check
in both Jenkins and GitLab CI and in each case the result is a bit of an opaque yea/nay popping out the end, with little indication of what went on or what was actually validated.
Has there been any work on a more transparent “executor” for flake check tests? Specifically, I’m thinking of stuff like:
- Evaluate the flake and emit a list of per-check invocations, so that an outer CI system could spin off separate jobs that run each check in parallel with their own separate log, timeout, status, etc. (See for example GitLab’s dynamic child pipeline capability: https://www.youtube.com/watch?v=C5j3ju9je2M)
- Run all the checks together, but generate a JUnit XML report that lists the names of the checks that were run and populates each with the correct log of what their output was. Unfortunately the particulars of the XML format are far from standardized, but nonetheless there are common elements that basically every CI system knows how to display in their UI.
If nothing like this exists, is it something there would be interest in?