Hydra release 21.11 fails for 5 days now

Unsure what to do in this case. Hydra - Build 164685259 of job nixos:release-21.11:tested seems to time out on building Firefox.

Couldn’t really find a topic regarding what to do when release hangs for a long time, I’m having a bit of trouble finding the commit that causes it. I know we can’t create a topic for every hang, so looking for generic advice on unjamming the build.

1 Like

Naively, it seems to be the linking step. I wonder whether we can switch to mold.

Or a legit timeout. Closure changed to be bigger? Or upstream now includes more things?

I wish the logs included the # of seconds since start at the beginning of the line…

But anyway I think I figured it out: when I go to the tested job which I presume gates release, the first failing job is 164012629 which was propagated from 164013093, which is only the aarch64 build, which went on for 2h 35m before being marked as timed out.

So TIL that the entire channel can be blocked by a single architecture?

Is there a way to give more time to the aarch64 build?

1 Like

Reported at Release 21.11 blocked on timeout of FF build on aarch64 · Issue #1121 · NixOS/hydra · GitHub

I’m not super familiar with it, but I believe there is a way to increase the time until a build times out on Hydra.

The derivations for GHC use this feature:


I wonder how to specify “if on arm64, make it 4 hours, but otherwise leave at the default”?

I believe this is all something else: aarch64 build blocking release-channel for x86 users · Issue #150532 · NixOS/nixpkgs · GitHub

Anyway, I’m continuously watching lag of all our channels and do basic triage, leading to resolution or at least filing a channel-blocker issue which then shows at https://status.nixos.org/

1 Like