When trying to install a (non-widely-popular) package for the first time, if it starts building, someone can easily think that this is simply because it isn’t cached by Hydra. Then you wait a few minutes / hours, only to find that the build (or the tests) are broken and the time spent on your machine was just wasted. When you start digging in the nixpkgs repository / Hydra, you discover that this package was supposed to build on Hydra, it’s just that it was broken there as well, and thus no binary was available.
It would be nice to get a warning and a confirmation prompt before a package starts building when a build of the same derivation was attempted on Hydra but failed eventually.
Another solution would be to refuse advancing a channel if there are undeclared broken packages. Then a bot could add the broken status automatically if adding it by hand is too much work (although it shouldn’t: maintainers are usually aware of broken packages, so they should either immediately fix them, or mark them as broken).
Also somewhat relevant is that nix used to have the option to cache failed builds. Having that plus the option to also substitute failed builds would solve part of this problem.
I think the best solution would be to mark everything that is broken as broken. Currently we pretty much only mark packages as broken if they have been failing for a while and nobody wants to fix them. It would be better to just mark stuff once it starts failing. But automating that is hard.
I think the best solution would be to mark everything that is broken as broken. Currently we pretty much only mark packages as broken if they have been failing for a while and nobody wants to fix them. It would be better to just mark stuff once it starts failing. But automating that is hard.
Also, there’s the issue that as far as I remember if a package is marked
as broken then it won’t be built by hydra, which would mean that it
wouldn’t be automatically marked as non-broken if another package update
incidentally fixes it.
That’d mean that packages that fail because master is completely broken
would then have to be all manually unmarked as broken… imagine what
would happen if mkDerivation was failing at one hydra eval.
Just to complicate this even further (sorry), build failures on Hydra are sometimes transient (e.g. a timeout). I’ve seen several times (in the ~ 6 months I’ve been using NixOS) where a package isn’t cached, but the local build succeeds. Any approach to automating package marking would need to account for this as well.
That said, assuming that transient and non-transient build failures can be reliably distinguished, I agree it would be nice to get a warning.
That said, assuming that transient and non-transient build failures can be reliably distinguished
I think I have seen a build fail two or three times in a row in a same weird (not resource-related) way on Hydra, with multiple people being able to build exactly the same derivation locally (with sandbox)…
I’ve long thought that the binary cache “protocol” should support negative caching as well, so the clients could consume that information easily, at least as a warning. Of course, this whole thing has multiple aspects (you touched some of them), and we’ll probably want to attack more of them, eventually.
In defense of the current behavior, I think it’s nice to show people why something is failing. Especially if this is a trivial issue, it encourages users to attempt to fix it themselves. If there is a good reason for something to fail, it makes sense to mark it as broken. But lots of times it’s fixable.
Well sure. But in the case that prompted this thread, the package ran tests for several hours on my machine. And the fix was apparently to disable one of them.
True, didn’t consider that. It probably wouldn’t be that bad though, as long as we only mark the “root” package as broken and not all its reverse dependencies. Those rarely fix themselves without manual intervention anyways. Still, the issue of automatically changing something in nixpkgs would probably make this a non-starter.
I think if there is a transient error, it wouldn’t be wrong to propagate a failure. There is an issue after all that may or may not affect the user. Either way, manual intervention required.
I like the sound of a warning. If it’s a transient failure, the package might build just fine for the user. If it’s a big package, the user might decide it’s not worth it and they can kill the build there and then rather than wait a few hours to confirm.
The issue is that the Nix client is not able to make the distinction between a failed build that is custom, and one that Hydra has also attempted to build. If it did, it could pre-emptively abort the build and even display the uploaded logs to the user.
This would require extending the binary cache protocol so that build failures are also uploadable, and queryable. Extend the Nix client with a setting option that tells it to/not to retry cached failures. Extend the Nix client to display the cache failure with a helpful message.
A .narinfo with no NAR URL could be considered a failure (this might break back-compat though). The .narinfo could also be extended to include a link back to the hydra build, which can then be displayed in the error message to the user. nix copy also supports uploading logs on build success, and could be extended to also upload the failed build logs.