Hello, I am seeking advice for how to troubleshoot this NixOS build issue when service.hydra.enable is set true. (Hoping this is not a duplicate post, as finding related threads has been difficult. “Hydra build fail” or similar google searches rarely refer to hydra itself failing to build, usually more like ZHF, etc.)
My understanding was that a NixOS config defined in a flake would NOT depend on the ambient host environment, but I am seeing different results when building the same flake on different hosts.
This simplified flake on github builds successfully on one host, and fails to build on another:
Success on desktop host (running nixos-23.11.20240115.b8dd8be)
$ nixos-rebuild build --flake 'github:danjl1100/nixos-hydra-service-build-fail?rev=c246d20607910df5fb9f1ecfec451a67ce34c298#test-system'
building the system configuration...
## the command succeeds
Fails on server host (running nixos-23.05.20231216.b2566f4)
nixos-rebuild build --flake 'github:danjl1100/nixos-hydra-service-build-fail?rev=c246d20607910df5fb9f1ecfec451a67ce34c298#test-system'
### fails to build hydra-2023-11-17 after 5-10 minutes, log copied below
On the successful host, nixos-rebuild appears to use the binary cache for hydra-2023-11-17 (though I am unsure how to confirm this).
On the failing host, nixos-rebuild attempts to build the package hydra-2023-11-17 and fails ~5 minutes later in the checkPhase.
Shortened nix log for the "hydra-2023-11-17" build failure
[ sections omitted for discourse.nixos.org post ]
PASS: perlcritic.pl
test.pl: Defaulting $YATH_JOB_COUNT to $NIX_BUILD_CORES (16)
[ sections omitted for discourse.nixos.org post ]
The following jobs failed:
+------------------------------------+------------------------------------+
| Job ID | Test File |
+------------------------------------+------------------------------------+
| EEA44D68-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Plugin/RunCommand/errno.t |
| ED | |
| | |
| EEA4CB4E-B7E1-11EE-9DD0-A4D35A4663 | input-types/hg.t |
| ED | |
| | |
| EEA4E912-B7E1-11EE-9DD0-A4D35A4663 | queue-runner/direct-indirect-const |
| ED | ituents.t |
| | |
| EEA39FA8-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Jobset/evals.t |
| ED | |
| | |
| EEA39AF8-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Jobset/channel.t |
| ED | |
| | |
| EEA46B0E-B7E1-11EE-9DD0-A4D35A4663 | Hydra/PostgresListener.t |
| ED | |
| | |
| EEA3832E-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Build/cancel.t |
| ED | |
| | |
| EEA48CD8-B7E1-11EE-9DD0-A4D35A4663 | api-test.t |
| ED | |
| | |
| EEA3E364-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/projects.t |
| ED | |
| | |
| EEA44890-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Plugin/RunCommand/dynamic-en |
| ED | abled.t |
| | |
| EEA3B362-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/JobsetEval/fetch. |
| ED | t |
| | |
| EEA40E98-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Event/EvalFailed.t |
| ED | |
| | |
| EEA3D540-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/User/dashboard.t |
| ED | |
| | |
| EEA3F6EC-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Event/BuildStarted.t |
| ED | |
| | |
| EEA3CBC2-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Root/narinfo.t |
| ED | |
| | |
| EEA4B82A-B7E1-11EE-9DD0-A4D35A4663 | input-types/darcs.t |
| ED | |
| | |
| EEA37992-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Admin/clear-queue |
| ED | -non-current.t |
| | |
| EEA38CAC-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Build/restart.t |
| ED | |
| | |
| EEA3A9D0-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Jobset/type-const |
| ED | raints.t |
| | |
| EEA43E36-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Plugin/RunCommand/basic.t |
| ED | |
| | |
| EEA41366-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Event/EvalStarted.t |
| ED | |
| | |
| EEA41CF8-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Helper/BuildDiff.t |
| ED | |
| | |
| EEA3F214-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Event/BuildQueued.t |
| ED | |
| | |
| EEA49B42-B7E1-11EE-9DD0-A4D35A4663 | evaluator/evaluate-constituents-br |
| ED | oken.t |
| | |
| EEA409CA-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Event/EvalCached.t |
| ED | |
| | |
| EEA47E50-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Schema/Users.t |
| ED | |
| | |
| EEA43990-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Plugin/DeclarativeJobsets/ba |
| ED | sic.t |
| | |
| EEA4BCC6-B7E1-11EE-9DD0-A4D35A4663 | input-types/deepgit.t |
| ED | |
| | |
| EEA4D9CC-B7E1-11EE-9DD0-A4D35A4663 | queue-runner/build-locally-with-su |
| ED | bstitutable-path.t |
| | |
| EEA3C6AE-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Root/evals.t |
| ED | |
| | |
| EEA41848-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Event/StepFinished.t |
| ED | |
| | |
| EEA443A4-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Plugin/RunCommand/dynamic-di |
| ED | sabled.t |
| | |
| EEA421BC-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Helper/CatalystUtils.t |
| ED | |
| | |
| EEA4AE8E-B7E1-11EE-9DD0-A4D35A4663 | input-types/bzr-checkout.t |
| ED | |
| | |
| EEA3DEA0-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/metrics.t |
| ED | |
| | |
| EEA3AE94-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/JobsetEval/cancel |
| ED | .t |
| | |
| EEA3D9E6-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/User/ldap.t |
| ED | |
| | |
| EEA47982-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Schema/ResultSet/TaskRetries |
| ED | .t |
| | |
| EEA4966A-B7E1-11EE-9DD0-A4D35A4663 | evaluator/evaluate-basic.t |
| ED | |
| | |
| EEA4A010-B7E1-11EE-9DD0-A4D35A4663 | evaluator/evaluate-constituents-gc |
| ED | .t |
| | |
| EEA3D090-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Root/status.t |
| ED | |
| | |
| EEA387F2-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Build/evals.t |
| ED | |
| | |
| EEA40042-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Event/CachedBuildQueued.t |
| ED | |
| | |
| EEA45718-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Plugin/RunCommand/http.t |
| ED | |
| | |
| EEA3B83A-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/JobsetEval/restar |
| ED | t.t |
| | |
| EEA4EDFE-B7E1-11EE-9DD0-A4D35A4663 | queue-runner/notifications.t |
| ED | |
| | |
| EEA39166-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Job/builds.t |
| ED | |
| | |
| EEA3BCFE-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Project/builds.t |
| ED | |
| | |
| EEA374D8-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/API/checks.t |
| ED | |
| | |
| EEA4C19E-B7E1-11EE-9DD0-A4D35A4663 | input-types/git-rev.t |
| ED | |
| | |
| EEA4FC86-B7E1-11EE-9DD0-A4D35A4663 | scripts/hydra-init.t |
| ED | |
| | |
| EEA4DF4E-B7E1-11EE-9DD0-A4D35A4663 | queue-runner/constituents.t |
| ED | |
| | |
| EEA4B334-B7E1-11EE-9DD0-A4D35A4663 | input-types/bzr.t |
| ED | |
| | |
| EEA3ECF6-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Event/BuildFinished.t |
| ED | |
| | |
| EEA4D4EA-B7E1-11EE-9DD0-A4D35A4663 | input-types/svn.t |
| ED | |
| | |
| EEA4A4DE-B7E1-11EE-9DD0-A4D35A4663 | evaluator/evaluate-dependent-jobse |
| ED | ts.t |
| | |
| EEA4E426-B7E1-11EE-9DD0-A4D35A4663 | queue-runner/default-machine-file. |
| ED | t |
| | |
| EEA50168-B7E1-11EE-9DD0-A4D35A4663 | scripts/hydra-send-stats.t |
| ED | |
| | |
| EEA39634-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Jobset/builds.t |
| ED | |
| | |
| EEA3C1CC-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Controller/Project/delete.t |
| ED | |
| | |
| EEA45240-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Plugin/RunCommand/fanout.t |
| ED | |
| | |
| EEA45BF0-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Plugin/RunCommand/json.t |
| ED | |
| | |
| EEA474B4-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Schema/Result/TaskRetries.t |
| ED | |
| | |
| EEA49188-B7E1-11EE-9DD0-A4D35A4663 | build-products.t |
| ED | |
| | |
| EEA4F2E0-B7E1-11EE-9DD0-A4D35A4663 | scripts/hydra-create-user.t |
| ED | |
| | |
| EEA4662C-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Plugin/gitea.t |
| ED | |
| | |
| EEA4F7CC-B7E1-11EE-9DD0-A4D35A4663 | scripts/hydra-eval-jobset/notifica |
| ED | tions.t |
| | |
| EEA4D012-B7E1-11EE-9DD0-A4D35A4663 | input-types/svn-checkout.t |
| ED | |
| | |
| EEA4C66C-B7E1-11EE-9DD0-A4D35A4663 | input-types/git.t |
| ED | |
| | |
| EEA42676-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Helper/Nix.t |
| ED | |
| | |
| EEA46FE6-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Schema/Result/RunCommandLogs |
| ED | .t |
| | |
| EEA40506-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Event/EvalAdded.t |
| ED | |
| | |
| EEA4880A-B7E1-11EE-9DD0-A4D35A4663 | Hydra/View/TT.t |
| ED | |
| | |
| EEA3FB88-B7E1-11EE-9DD0-A4D35A4663 | Hydra/Event/CachedBuildFinished.t |
| ED | |
+------------------------------------+------------------------------------+
Yath Result Summary
-----------------------------------------------------------------------------------------
Fail Count: 74
File Count: 90
Assertion Count: 210
Wall Time: 339.40 seconds
CPU Time: 453.89 seconds (usr: 0.83s | sys: 0.12s | cusr: 323.55s | csys: 129.39s)
CPU Usage: 133%
--> Result: FAILED <--
1..1
not ok 1 - Passed tests when run by yath
yath exited with 256FAIL: test.pl
===================
1 of 2 tests failed
===================
make[2]: *** [Makefile:354: check-TESTS] Error 1
make[2]: Leaving directory '/build/source/t'
make[1]: *** [Makefile:480: check-am] Error 2
make[1]: Leaving directory '/build/source/t'
make: *** [Makefile:422: check-recursive] Error 1
Prior commit (hydra disabled)
This earlier commit passes on both hosts, since the hydra service is disabled:
nixos-rebuild build --flake 'github:danjl1100/nixos-hydra-service-build-fail?rev=283d3dbaeeadeb558f5de0fe31ff21c1f1b9bd1a#test-system'
## the command succeeds (on both hosts)
The build is failing on the host that previously had Hydra installed, so I am not sure if something is interfering there. I followed the postgresql upgrade instructions in the nixos manual, but surely the contents of /var/lib/postgresql wouldn’t affect the NixOS configuration build phase…(?)
Any suggestions for other things to try or possible explanations are welcome. Cheers 