Hydra release 21.11 fails for 5 days now

wmertens · January 16, 2022, 12:27pm

Unsure what to do in this case. Hydra - Build 164685259 of job nixos:release-21.11:tested seems to time out on building Firefox.

Couldn’t really find a topic regarding what to do when release hangs for a long time, I’m having a bit of trouble finding the commit that causes it. I know we can’t create a topic for every hang, so looking for generic advice on unjamming the build.

L-as · January 16, 2022, 1:01pm

Naively, it seems to be the linking step. I wonder whether we can switch to mold.

tomberek · January 16, 2022, 1:31pm

Or a legit timeout. Closure changed to be bigger? Or upstream now includes more things?

wmertens · January 16, 2022, 3:53pm

I wish the logs included the # of seconds since start at the beginning of the line…

wmertens · January 16, 2022, 4:06pm

But anyway I think I figured it out: when I go to the tested job which I presume gates release, the first failing job is 164012629 which was propagated from 164013093, which is only the aarch64 build, which went on for 2h 35m before being marked as timed out.

So TIL that the entire channel can be blocked by a single architecture?

Is there a way to give more time to the aarch64 build?

wmertens · January 16, 2022, 4:13pm

Reported at Release 21.11 blocked on timeout of FF build on aarch64 · Issue #1121 · NixOS/hydra · GitHub

cdepillabout · January 17, 2022, 12:30am

I’m not super familiar with it, but I believe there is a way to increase the time until a build times out on Hydra.

The derivations for GHC use this feature:

github.com

NixOS/nixpkgs/blob/718a7207f8c911d2288da8ccc843e745b7242021/pkgs/development/compilers/ghc/9.0.2.nix#L339


      
              # Our Cabal compiler name
              haskellCompilerName = "ghc-${version}";
            };
          
          
  meta = {
              homepage = "http://haskell.org/ghc";
              description = "The Glasgow Haskell Compiler";
              maintainers = with lib.maintainers; [
                guibou
              ] ++ lib.teams.haskell.members;
              timeout = 24 * 3600;
              inherit (ghc.meta) license platforms;
            };
          
          
} // lib.optionalAttrs targetPlatform.useAndroidPrebuilt {
            dontStrip = true;
            dontPatchELF = true;
            noAuditTmpdir = true;
          })

wmertens · January 19, 2022, 11:04am

I wonder how to specify “if on arm64, make it 4 hours, but otherwise leave at the default”?

vcunat · January 20, 2022, 6:58pm

I believe this is all something else: aarch64 build blocking release-channel for x86 users · Issue #150532 · NixOS/nixpkgs · GitHub

Anyway, I’m continuously watching lag of all our channels and do basic triage, leading to resolution or at least filing a channel-blocker issue which then shows at https://status.nixos.org/