Benchmarking `stdenv.mkDerivation` vs `derivation` for trivial builds

Development Nix
1

I wanted to benchmark the performance difference between stdenv.mkDerivation and derivation for trivial builders; largely to find out how much time is spent processing setup.sh even for optimally tuned use of stdenv.mkDerivation.

For context this benchmarking is of interest to my work with Node.js packaging, where the vast majority of packages/modules only need to be unpacked and copied to the Nix store; with that in mind even if derivation were only slightly faster, there is a real impact at scale where many Node.js applications require thousands of dependencies to perform trivially simple tasks ( I’ll restrain myself from any commentary about web developers’ flippant use of modules ).

I cooked up this playground repo with a simple benchmarking hardness.
Below the link is an abbreviated form of the README with findings.

Goals

Benchmark performance of a trivial build using stdenv.mkDerivation vs. a plain derivation .

This build simply calls touch on $out . Setting the argument count controls the number of builds to be tested.

The outputs of this expression are two derivations which depend on count inputs using stdenv.mkDerivation and derivation respectively.

Findings

After running this benchmark over 1,000,000 builds, I found that for this trivial operation derivation was roughly twice as fast as stdenv.mkDerivation .

My personal suspicion is that the overhead of processing setup.sh for each build that uses stdenv.mkDerivation accounts for the difference in runtime. I want to be clear here and say “I do not think the use of stdenv.mkDerivation is bad.” On the contrary the routine is incredibly useful, and the structure of setup.sh and its phases are an essential tool. Rather I’m hoping to show that for very trivial operations such as copy, moving, or modifying permissions on files - the overhead of setup.sh may lead developers to use derivation directly in these cases.

Setup

Notably these tests disable any extraneous stdenv.mkDerivation phases, so we basically just run installPhase = "touch \"$out\";"; here. I went out of my way to be as “fair” as possible to stdenv.mkDerivation and tuned/optimized its settings as best as I could.

In practice the performance will be worse if people forget to disable things like patchPhase , configurePhase , and fixupPhase when they don’t really require them. If we’re being honest it’s uncommon for people to disable these, particularly fixupPhase which is the most costly.

You can experiment with the additional runtime spent when those phases are activated by modifying default.nix .

5 Likes
2

A potential nit WRT to whether it’s a fair comparison:

I wonder how much of this difference is just startup time for bash? The current accounting may be fair if you plan to directly invoke a single copy/install command as your builder, but it may be overstating the difference if you’re just intending to use a much smaller shell script?

Here’s a gut-check on relative timings from a fairly slow intel macbook air:

$ echo : > near-empty.sh
Thu Dec 29 2022 14:03:21 (2.121ms)

$ /run/current-system/sw/bin/bash -n near-empty.sh
Thu Dec 29 2022 14:03:34 (126ms)

$ /run/current-system/sw/bin/bash -n ./pkgs/stdenv/generic/setup.sh
Thu Dec 29 2022 14:03:38 (158ms)

$ /run/current-system/sw/bin/touch near-empty.sh
Thu Dec 29 2022 14:04:16 (26.7ms)
1 Like
3

would be interesting to see these tests repeated with draft: per-dso .so resolution cache on glibc by pennae · Pull Request #207893 · NixOS/nixpkgs · GitHub applied (which could reduce time spent in setup.sh, depending on how much it uses non-shell-builtins)

edit did the test, here the difference is not that large (with or without the patch, that doesn’t seem to matter). on 10k builds mkDerivations takes about 475 seconds while derivations takes 318. much of that seems to be spent waiting for nix to schedule stuff rather than actually running builds, average cpu load was about 10% across the entire system.

2 Likes