Load average above thread count during nixos-rebuild

During nixos-rebuild, if multiple large packages have to built locally, the load average goes significantly beyond the number of threads. This, if I understand correctly, leads to sub optimal performance.

For example, building top of master with a modified version of mesa, the load average on my 32 thread system got up to 94: https://i.imgur.com/3jxraZG.png

This is also a problem in systems with disproportionately more threads than ram, like a dom0 NixOS.

Is there a way to properly limit this?

Not really, it’s messy.

You can limit max-jobs and cores (in nix config, also equivalents on command line). However it doesn’t play well if you’d like to have both higher based on current situation, i.e. to be safe-ish you’d aim for their product to be not much above $(nproc).

Then there are some packages with build systems that do not respect the cores settings and detect whatever you have, so those can derail it completely.

Not sure how that’d suit your use case, but there’s a way to dynamically constraint CPU usage and RAM usage using cgroups(7).

You should be able to define a new systemd.slice(5), give it constraints using a command such as systemctl set-property --runtime -- custom.slice AllowedCPUs=0-1, and launch a shell within this slice using systemd-run(1) (systemd-run --slice=custom.slice …).

FYI, cgroups attributes can be redefined at any time from anywhere using the systemctl(1) command above, and take effect immediately.