More and more packages get
.passthru.tests added, which is fantastic. They get run by ofborg in PRs, also fantastic.
However, when a test gets completely broken by a change, it goes green (bad!) with text:
The following builds were skipped because they don’t evaluate on x86_64-linux: consul.passthru.tests
How can we make such cases go red on Github, and show “Failed”?
CC @grahamc who might know best, but probably others as well.
Example where this just went wrong: neomutt: 20230407 -> 20230512 by Ma27 · Pull Request #231638 · NixOS/nixpkgs · GitHub
The tests (correctly) failed but CI ignored the failures instead of flagging them. The PR was merged, broken.
Here is another example: rclone: 1.62.2 -> 1.63.0 by SuperSandro2000 · Pull Request #240743 · NixOS/nixpkgs · GitHub
aarch64-darwin tests correctly failed, indicating a failure to build rclone on M1 macs. It went undetected and got merged.
Maybe @lilyinstarlight would be interested in this
It’s a repeated question, e.g.