Are there concrete suggestions for balancing `cores` and `max-jobs`?

Help
1

Is there some optimal x, y, and z such that

zN = xJ * yC

where

N = number of (virtual?) cores
J = max-jobs
C = cores

?

In other words,

  1. What should maxJobs * numCores equal?
  2. Should maxJobs and numCores be equal to each other, or is there a better ratio?
  3. If the answer to these questions are really different from use case to use case, should we approach the whole thing in a different way?

In the manual, Introduction provides a nice overview of how the two options interact, but it stops short of making any recommendations.

NixOS options (nix.maxJobs and nix.buildCores) provide similar details, although I’m not sure those descriptions are even internally consistent. The description for nix.buildCores says the special value 0 should be “used with care”, yet 0 is also the default value. :thinking: Should we have a better default?

I gather that the defaults are bad, since they can potentially spawn N^2 processes on an N-core machine.

But what should we do?

3 Likes
2

From what I’ve observed, the manual doesn’t give the whole picture these days. For the regular stdenv make-based builds, Nixpkgs passes the value of cores to make as:

make -j${NIX_BUILD_CORES} -l${NIX_BUILD_CORES} ...

The -l/--load-average tells make to avoid pushing the system-wide load average over that value, so make will parallelize up to that much but won’t overload your system. On the one hand, this is great because you can set cores to the number of logical CPU cores in your system, and tune maxJobs independently depending on what you’re building. Things might take longer, but you won’t get crazy CPU or RAM usage. Edit: Also, if you set maxJobs and cores expecting the product of the two to be how many processes are running, that won’t be true, you’ll only see cores processes.

The downside is that a --load-average limit isn’t applied to all build systems; Haskell for example doesn’t, and for those builds you will get maxJobs * cores threads running. So you potentially need to pay attention to what it is that you’re building.

Personally I set maxJobs = 2, cores = <num logical CPUs> and override if I want something different for a specific build. Thanks for posting this question, I’ve been meaning to ask what the general advice is these days. Is the goal is to have the load average limit be applied for more build systems eventually?

2 Likes