RFC: make stdenv to build in parallel by default

Development
#1

Hi all!

You might know that stdenv.mkDerivation has support for parallel builds via enableParallelBuilding = true; attribute (NixOS - Nixpkgs 21.05 manual). For some build systems the attribute is already set by default. But it’s not set for autotools-based systems. Only a few large packages opt into parallel builds. I happened to notice that coreutils and bison didn’t and enabled them explicitly. I would say enableParallelBuilding = false; is a wrong default for today’s multicore systems.

What would be the path to enable enableParallelBuilding = true; by default in nixpkgs?

It’s a disruptive change. I expect up to 1% of packages to start failing the build once it’s enabled. I fixed all the packages I found that are part of my desktop NixOS system.

How that kind of changes is usually introduced in nixpkgs? Caveat: I plan to do something similar with gcc-11 and with -fno-common workaround removal. Both are guaranteed to produce comparable fallout once enabled by default.

I personally can’t realistically fix (or even find) 100s of newly failing packages myself. Ideally I would like package maintainers to help me. Does nixpkgs and/or Hydra have an infrastructure to do such kinds of change? Other distributions usually use bug trackers and bots to auto-file bugs and CC maintainers.

Want to try out parallel builds yourself (or maybe fix a package or two)?

4 Likes
#2

Would this improve the Nix experience for that many users? Personally I hardly ever build large autotools packages locally, I get them from hydra. And hydra already parallelizes at the package level, right? It even seems possible that this could slow down hydra’s overall throughput.

Of course, it’s great if you’re doing development on such a package, so maybe there should be an easier way to turn it on.

#3

I expect the change to shrink critical path for any rebuild chain. Be it local rebuild or hydra build. Shorter critical path makes for faster binaries an better package-level parallelism available for everyone.

Typical use case would be to build something from nixpkgs/staging branch. Bison’s update triggers 80 rebuilds if you try to update it. It can’t saturate parallelism of a single 16-core desktop.

$ nix edit -f. bison # add `FOO = "1";`
$ time nix build -f. bison
real    20m59,001s

Would you guess that binutils or coreutils are built sequentially? They are. Same change with parallel builds enabled by default:

$ nix edit -f. bison # add FOO = "parallel-1"
$ time nix build -f. bison
real    20m25,928s

Note that it’s a 2% improvement on an optimised path.

#4

How that kind of changes is usually introduced in nixpkgs ?

If we expect a lot of build failures we could do a jobset on hydra but if there are only a few hundred ones we can just throw it onto the next staging and see what happens.

I personally can’t realistically fix (or even find) 100s of newly failing packages myself. Ideally I would like package maintainers to help me. Does nixpkgs and/or Hydra have an infrastructure to do such kinds of change? Other distributions usually use bug trackers and bots to auto-file bugs and CC maintainers.

There is not really a system here. We can pin an issue for a while or create noise around it. Worst case there are 100 new build failures and they need to be fixed.

everyone profits from faster build times. Faster hydra means faster time until we get a cache entry and faster development. This will also speedup staging builds which is especially nice for core maintainers.

And hydra already parallelizes at the package level, right?

It is still very nice if hydra would be faster done and we can save build capacity.

It even seems possible that this could slow down hydra’s overall throughput.

I don’t think that can happen at all. This change is at a builder level.

3 Likes
#5

That’s true, I didn’t really consider the notion of a critical path. Still, I wonder if selectively turning it on for certain known-important packages could be a better use of effort than changing the default and dealing with lots of breakage.

I admit I don’t really know how hydra works and what the current load on the cluster(s) looks like. I’m assuming they’re pretty much running at capacity all the time (bottlenecked on either cpu or io bandwidth). If that’s the case, and you just start lots more gcc or whatever processes, you just increase contention without increasing throughput, and contention has a cost. To avoid this, you’d have to coordinate parallelism across package builds, and I don’t know if that happens.

(Also, even if there is a slight decrease in throughput, the improvement in latency could be worth it.)

#6

Still, I wonder if selectively turning it on for certain known-important packages could be a better use of effort than changing the default and dealing with lots of breakage.

If you plan to do it I’m all for it. In my opinion observing a parallel build failure and disabling parallelism is usually a lot more straightforward than finding out that package has useful parallelism to enable.

Is it really lots of breakage? So far selective turning on is enabled in 1188 packages in nixpkgs and it still misses trivial cases like coreutils. Selective turning it off is currently explicitly set in 94 packages. I think it tells us that selective-on is not practically maintainable as people just don’t think of enabling parallelism by default.

1 Like
#7

This is already being done and still a lot of packages miss it. I think the benefits outweigh the drawbacks. Worst case we just revert the commit if it turns out to be a disaster.

#8

https://github.com/NixOS/nixpkgs/issues/142338 was rejected as is. For the time being people would need to somehow else spot slow packages and enable parallelism on per-package basis.

Hosted by Flying Circus.