Can we make ofborg show broken tests as red?

More and more packages get .passthru.tests added, which is fantastic. They get run by ofborg in PRs, also fantastic.

However, when a test gets completely broken by a change, it goes green (bad!) with text:

Success

Attempted: consul

The following builds were skipped because they don’t evaluate on x86_64-linux: consul.passthru.tests

How can we make such cases go red on Github, and show “Failed”?

CC @grahamc who might know best, but probably others as well.

https://github.com/NixOS/nixpkgs/pull/107798#issuecomment-751856433:

3 Likes

Example where this just went wrong: neomutt: 20230407 -> 20230512 by Ma27 · Pull Request #231638 · NixOS/nixpkgs · GitHub

The tests (correctly) failed but CI ignored the failures instead of flagging them. The PR was merged, broken.

2 Likes
Hosted by Flying Circus.