tl;dr let’s put something into stdenv that uses an external helper to act as though nix-daemon was also a make -jN instance and get ≥10% build time improvements for large parallel builds?
make -jN is a wonderful thing. it’ll create up to N jobs to fulfill its goals of building something, and no more than N jobs. this of course speeds up builds greatly if there’s much that can run in parallel, but it turns into a problem if you run many builds in parallel—each of them can now start N jobs, and the system can be overcommitted greatly.
in stdenv this is mitigated by not using -jN but -jN -lN, which inhibits new jobs when the system load average goes past N. on an idle system we still start out massively overutilized because each make instance can start N jobs immediately, but after a while loadavg climbs and make stops starting things until the system calms down again.
unfortunately this requires the system to go partially idle for a while, otherwise (with N = #CPU) loadavg can’t fall far enough for make to resume starting jobs. this leads to a somewhat sinewavy load curve, and overall the system oscialltes between being overcommitted and partially idle.
make can however just use an externally provided jobserver instead of starting its own, and we’ve put together a proof of concept for that in https://github.com/NixOS/nixpkgs/pull/143820. having a global limiter on concurrency that does not rely on loadavg immediately leads to better system utilization, in our tests of just compiling the nine llvm versions in nixpkgs we’ve seen 10% improvement over what stdenv currently does. similar effects can be had by turning the load limit way up (to -l4N or thereabouts), but that obviously comes at the cost of having six times as many jobs running at any given time, using as much extra memory and causing more scheduling overhead. currently this is implemented only for linux with the help of CUSE (which is an offshoot of FUSE), but it could also be done with plain FUSE and possibly ported to darwin (using macFUSE, of which we know little more than that it exists).
no idea whether this is worth going for, considering that it does require another daemon running on the machine and a sandbox break to give stdenv access to that daemon.
