I recently setup Hydra, I made some overrides that apply to a lot of packages from nixpkgs in the 10s of thousands and I wanted to deploy hydra in order to increase CPU utilization when building, have a tested framework for distributing builds across machines and use it as a robust CI solution. I have set up hydra itself i have maxjobs in nix at 32 I have tuned eval to be reasonably fast but in preliminary testing on my reasonably powerful machine Hydra is not utilizing the machine’s resources to the extent that I hoped most of the time the CPU is idling. I think its the queue runners fault or I have messed something up in configuration I expected hydra to pin my CPU constantly and improve my throughput. I am adding the queue runner’s status if its any help.
{
“avgStepBuildTime” : 13.2618560791016,
“avgStepTime” : 15.8762884140015,
“buildReadTimeAvgMs” : 6491.13916015625,
“buildReadTimeMs” : 5355190,
“bytesReceived” : 1238760,
“bytesSent” : 4148304,
“dispatchTimeAvgMs” : 0.00310752028599381,
“dispatchTimeMs” : 5,
“jobsets” : {
“test:test-build” : {
“seconds” : 12864,
“shareUsed” : 128.64
}
},
“machineTypes” : {
“i686-linux” : {
“runnable” : 0,
“running” : 0
},
“i686-linux:local” : {
“runnable” : 0,
“running” : 0
},
“x86_64-linux” : {
“runnable” : 0,
“running” : 0
},
“x86_64-linux:big-parallel” : {
“runnable” : 0,
“running” : 0
},
“x86_64-linux:local” : {
“runnable” : 1,
“running” : 0
}
},
“machines” : {
“ssh://localhost” : {
“avgStepBuildTime” : 13.2618560791016,
“avgStepTime” : 15.8762884140015,
“consecutiveFailures” : 0,
“currentJobs” : 0,
“disabledUntil” : 0,
“enabled” : true,
“idleSince” : 1773607877,
“lastFailure” : 0,
“mandatoryFeatures” : [ ],
“nrStepsDone” : 970,
“supportedFeatures” : \[
“benchmark”,
“big-parallel”,
“kvm”,
“nixos-test”
\],
“systemTypes” : \[
“x86_64-linux”
\],
“totalStepBuildTime” : 12864,
“totalStepTime” : 15400
}
},
“maxNrRetries” : 3,
“nrActiveDbUpdates” : 0,
“nrActiveSteps” : 0,
“nrBuildsDone” : 824,
“nrBuildsRead” : 825,
“nrDbConnections” : 37,
“nrDispatcherWakeups” : 1609,
“nrQueueWakeups” : 6489,
“nrQueuedBuilds” : 1,
“nrRetries” : 3,
“nrRunnableSteps” : 1,
“nrStepsBuilding” : 0,
“nrStepsCopyingFrom” : 0,
“nrStepsCopyingTo” : 0,
“nrStepsDone” : 970,
“nrStepsStarted” : 975,
“nrStepsWaiting” : 0,
“nrStepsWaitingForDownloadSlot” : 0,
“nrUnfinishedSteps” : 4,
“nrUnsupportedSteps” : 0,
“pid” : 2146,
“status” : “up”,
“store” : {
“narCompressionSavings” : 0,
“narCompressionSpeed” : 0,
“narInfoCacheSize” : 1155,
“narInfoMissing” : 0,
“narInfoRead” : 0,
“narInfoReadAverted” : 219328,
“narInfoWrite” : 0,
“narRead” : 0,
“narReadBytes” : 0,
“narReadCompressedBytes” : 0,
“narWrite” : 0,
“narWriteAverted” : 0,
“narWriteBytes” : 0,
“narWriteCompressedBytes” : 0,
“narWriteCompressionTimeMs” : 0
},
“time” : 1773607926,
“totalStepBuildTime” : 12864,
“totalStepTime” : 15400,
“uptime” : 70387
}
Note: the override I am testing influences stdenv