Behavior of undeclared broken packages is confusing

#1

@timokau suggested in https://github.com/NixOS/nixpkgs/issues/55648 that I start a discussion here.

When trying to install a (non-widely-popular) package for the first time, if it starts building, someone can easily think that this is simply because it isn’t cached by Hydra. Then you wait a few minutes / hours, only to find that the build (or the tests) are broken and the time spent on your machine was just wasted. When you start digging in the nixpkgs repository / Hydra, you discover that this package was supposed to build on Hydra, it’s just that it was broken there as well, and thus no binary was available.

It would be nice to get a warning and a confirmation prompt before a package starts building when a build of the same derivation was attempted on Hydra but failed eventually.

Another solution would be to refuse advancing a channel if there are undeclared broken packages. Then a bot could add the broken status automatically if adding it by hand is too much work (although it shouldn’t: maintainers are usually aware of broken packages, so they should either immediately fix them, or mark them as broken).

#2

Worsening the issue is that maintainers are not notified of breakages.

Also somewhat relevant is that nix used to have the option to cache failed builds. Having that plus the option to also substitute failed builds would solve part of this problem.

I think the best solution would be to mark everything that is broken as broken. Currently we pretty much only mark packages as broken if they have been failing for a while and nobody wants to fix them. It would be better to just mark stuff once it starts failing. But automating that is hard.

#3

Timo Kaufmann via Nix community nixos1@discoursemail.com writes:

I think the best solution would be to mark everything that is broken as broken. Currently we pretty much only mark packages as broken if they have been failing for a while and nobody wants to fix them. It would be better to just mark stuff once it starts failing. But automating that is hard.

Also, there’s the issue that as far as I remember if a package is marked
as broken then it won’t be built by hydra, which would mean that it
wouldn’t be automatically marked as non-broken if another package update
incidentally fixes it.

That’d mean that packages that fail because master is completely broken
would then have to be all manually unmarked as broken… imagine what
would happen if mkDerivation was failing at one hydra eval.

#4

Just to complicate this even further (sorry), build failures on Hydra are sometimes transient (e.g. a timeout). I’ve seen several times (in the ~ 6 months I’ve been using NixOS) where a package isn’t cached, but the local build succeeds. Any approach to automating package marking would need to account for this as well.

That said, assuming that transient and non-transient build failures can be reliably distinguished, I agree it would be nice to get a warning.

#5

That said, assuming that transient and non-transient build failures can be reliably distinguished

I think I have seen a build fail two or three times in a row in a same weird (not resource-related) way on Hydra, with multiple people being able to build exactly the same derivation locally (with sandbox)…

#6

I’ve long thought that the binary cache “protocol” should support negative caching as well, so the clients could consume that information easily, at least as a warning. Of course, this whole thing has multiple aspects (you touched some of them), and we’ll probably want to attack more of them, eventually.

#7

In defense of the current behavior, I think it’s nice to show people why something is failing. Especially if this is a trivial issue, it encourages users to attempt to fix it themselves. If there is a good reason for something to fail, it makes sense to mark it as broken. But lots of times it’s fixable.

#8

Well sure. But in the case that prompted this thread, the package ran tests for several hours on my machine. And the fix was apparently to disable one of them.

#9

True, didn’t consider that. It probably wouldn’t be that bad though, as long as we only mark the “root” package as broken and not all its reverse dependencies. Those rarely fix themselves without manual intervention anyways. Still, the issue of automatically changing something in nixpkgs would probably make this a non-starter.

#10

I think if there is a transient error, it wouldn’t be wrong to propagate a failure. There is an issue after all that may or may not affect the user. Either way, manual intervention required.

#11

Haven’t looked into the exact failure you experienced, but I’m pretty sure the fix wasn’t just to disable a test.

If failures would be substituted, nix could download the log from the substitutor (as nix log already does) and display that.

#12

I like the sound of a warning. If it’s a transient failure, the package might build just fine for the user. If it’s a big package, the user might decide it’s not worth it and they can kill the build there and then rather than wait a few hours to confirm.

#13

Perhaps off-topic, but I would think bisection is one of the big use-cases of cached build failures.