When are PRs adding NixOS tests appropriate?

I’ve recently extended a game package, and I think a NixOS test could be helpful here. But they seem somewhat “heavyweight” to me and I was wondering if there are guidelines or practices for when to use them.

If PRs testing previously untested packages are appreciated in general I’d have a couple of things I could add :slight_smile:

Related questions: suppose there’s a NixOS test for package xyz that suddenly fails. Does that block some of the automatic nixpkgs processes/workflows, or is xyz simply marked as broken and things move on?

I have been under an impression that the causal chain is: NixOS VM tests are heavy → they are slow → they are annoying to debug → there are not enough of them…

Hum, so then I’ll probably create a PR or two. But how about the “block” thing? I’d like to understand the “consequences” first.

1 Like

I am pretty sure all auto-mark-broken processes are manually triggered… How to make sure a test does not block any channel is a good question.

(It looks like it could be possible not to include the test in the channels at all, though)

No blocking by default, except for waiting for everything (including the test) to finish. Just look at the amount of failing tests: https://hydra.nixos.org/eval/1767792?filter=nixos.tests#tabs-still-fail

1 Like