Policy for acceptable number of failures in a jobset


I’m a relatively new contributor and wondering about the criteria to consider a branch in a releasable state. I know about the hydra jobsets that need to be mostly green, but I can also see that some of these jobsets are never entirely green (which is unavoidable with such a large amount of packages of course).

So I guess my question is, is this evaluated “manually” by a human being seeing that there are less failures than on average, or is there an automated criteria?

That question came to mind after I opened this issue and tracked down the upgrade that broke it to this PR. @hexa linked the hydra jobset testing changes to these python packages in the PR, and I can see in the jobset that the fava build is indeed failing.

Of course I’m not expecting the awesome maintainers we have to give even more of their time to fix every single broken build on such an upgrade, but I’m wondering what’s the threshold for “ok enough of the jobset is green, let’s merge”, if that’s clear :sweat_smile: .


I also found this comment from @jonringer on the python package set upgrade doc PR, so I’m guessing parts of my question could be answered in that PR soon :eyes:

Fava was fixed ~12 hours ago on master fava: 1.18 -> 1.19 · NixOS/nixpkgs@9a6cc8c · GitHub there should be a channel update soon with the updated package.