“Never build the same thing twice” is a great tagline for nix, but it’s really only true when your build succeeds. What about when the build fails? Why doesn’t nix just cache the failure code and failure log?
This would be lightweight (much less to store than the successful build results!) and would save countless hours when debugging/bisecting.
(I know that users will want to add a flag to retry a build anyway. Things like network errors, etc. are hard to eliminate, but the point remains…)
Because it’s not really meaningful. The current design just saves the realized “validPaths” which were created.
Hydra does cache this, but it has a lot of machinery around restarting builds.
It does, you can do nix log <failed .drv> to see the logs.
I think for most people who are consuming nix from stable channels, this isn’t a super common scenario.
But I agree that it would be amazing for package maintenance to constantly try a build.
Because it’s not really meaningful
It’s true, but we also discard just too much sometimes: for example when after a very long successful build we encounter a minor error in the checkPhase. We have many packages that would benefit greatly from splitting into individually cached phases
I’ve wondered whether the storage implications of this are prohibitive, as I also see a potential use-case:
nixpkgs could probably support some interesting tools/levers if it was trivial to get a reference to the ~prepared pre-build source, or to the intermediate artifacts generated by the build process but otherwise thrown away.
(This said, I waffle a little bit on whether it would be better to have a system for hooking in standard analyses to be performed globally and stored as outputs. Both? 
)
If a package is failing to build, and Hydra/cache.nixos.org knows this, how do I save myself the electrons locally? This would be invaluable when it comes to bisecting failures. Is there any way to automatically pull the failures from cache.nixos.org the same way that we pull the successful builds?
there’s a hydra-check utility. I’m not aware of anything else
Also if I run
$ nix-build -A my-failing-package
$ nix-build -A my-failing-package
why doesn’t nix just spit out the cached, failure logs on the second invocation?
If you were using flakes it did.
Maybe this is my cue to start using flakes more…
My main use case for this is bisecting nixpkgs failures. Hydra already knows which things failed to build (and the failure logs!), but instead I have to blow hours upon hours getting them to re-fail locally. It’s a huge waste of resources when you’re bisecting on something like pytorch or tensorflow.
That use case is not covered by flakes eval cache, as it uses the commit sha as one of the cache keys.
And when bisecting nixpkgs I often have to build stuff that never has been built by hydra. This happens much more often than having to build something that would have been known to fail in advance.
Is there a reason this functionality is limited to flakes? Or is it just a matter of no one has gotten around to it?
I should also clarify that even when it’s something Hydra has never built before, once I spend half an hour failing to build something once, I then tend to continue to fail to build it 7 more times. So it would be very convenient to have these failures cached locally, even if Hydra/cache.nixos.org cannot be helpful.
Non flakes nix isn’t self contained, and therefore has to assume that each and every run of a build could result in a different outcome as something could have changed the imperative parts of the system.
It wouldn’t even be aware of dirtiness of your nixpkgs repository you are bisecting.
Indeed, but not even flakes can save you here 
Flakes cache evaluation results based on a flakes git SHA.
Nothing there remembers that a certain derivation with name X and input hash Y failed.
Though indeed this might be a nice feature that should even be possible to implement in non-flakes nix.
Though then we need a way to enforce rebuilds, I often have problems with disk full either because /tmp or /nix not having enough free space. I don’t want those to block a build forever and neither do I want to have to manually delete cache entries.
The original question is whether the build of the same resulting derivation has failed before, flake evaluation should not matter for this.
Do flakes actually have negative build caching, or is it just about flake evaluation cache (which indeed needs some form of hermetic evaluation, although tying things specifically to be Git-only is one of the reasons I won’t touch flakes as long as I can avoid them)?
I guess for network failures one could also special-case fixed-output derivations, at least; not sure if there is a robust way to detect the out of space condition, however (som heuristics could probably help a bit)
This would presume that the build failure was reproducible which it doesn’t have to be. In fact in presence of limited resources failures due to, say, OOM are quite common which are not always reproducible.
cache.nixos.org does not know this – it just knows which builds it has cached and not everything not cached fails. Hydra is an implementation detail of the nixos.org binary cache and it would strike me as odd if Nix had special code for it. More importantly though, it wouldn’t be feasible to query Hydra on every cache miss, as Hydra would probably not be able to handle the load of those requests.
You could possibly work around this, but then you are starting to add a lot of extra complexity for a pretty small target audience which would also be served by having a script locally that wraps hydra-check and nix-build.
OOM/failures don’t have to be reproducible
That’s a great point! I guess an even more skeptic way to say this would be “a failure is less likely reproducible than a success”.
But there is nix log which is able to fetch the log for a derivation, so fundamentally we can check in advance if something has failed before. Perhaps what we want for this use case is for nix build to have an interactive mode where it would query the cache before building, and warn the user: “this derivation already failed once, would you still like to try and build it? Y/n”
Presence of log might mean that build is in progress right now (on Hydra). At least I think so.
BuildStream & co do have features to cache failed builds for example, and my experience with those tools is that it causes quite a few headaches (especially for bazel, since it lacks the force rebuild switch).
Most of the time I know that my build failed when working locally, and only re-run it to check if the failure is spurious, so the use case is almost nonexistent locally (though it does provide a quick way to check whether my derivation changed meaningfully, and can speed up tests).
The issue comes in when building stuff that has been cached to fail remotely, which seems to be the main use case intended here - it in effect poisons the cache, because it makes it so that CI refuses to ever try again, which is very problematic (occasional successful artifacts from a broken build is better than no artifacts ever). Users can’t determine whether the remote build failure is spurious either, and either know about this quirk and force rebuilds anyway or end up confused. If the “spurious build failure” story isn’t thought out well in advance, it can be very annoying.
I like @SergeK’s suggestion, assuming it also means that non-interactive builds will always still rebuild. Giving a prompt also flags to a potential user who isn’t aware of this caching why the build is failing, though I’d like a more expressive message focused on new users.
As nobody has mentioned it: Hydra has a feature where it can mark a build as a “failed” build if it produces $out/nix-support/failed.
This can be used to keep around some more stuff than just the logs for later debugging.
I think the NixOS tests did make use of that at some point to generate a html report even on failed runs.
Caveat is that you need to modify your derivations such that this works.
Also nix itself will treat it as a successful build, so this is best left to “leaf” packages such as nixos-tests.
Nix used to have support for caching build failures, but this was removed because it was only really used by Hydra (which has its own negative caching now) and because it’s hard to distinguish between transient and permanent failures.