February 2022
While waiting for builds to test the hybrid open source releases and SDK frameworks approach, a particularly slow phase of the Darwin stdenv build stood out to me. Libsystem is mostly just a collection of headers from other open source releases. Simply copying some header files from other derivations seemed like it shouldn’t take a significant amount of time. Turns out this assumption was wrong. From my rigorous scientific testing, i.e., running the Libsystem build once with the time command (only Libsystem, not including dependencies), using cpio in the Libsystem build made it take ~250 minutes, or 4 hours and 10 minutes. That’s a lot of minutes.
The motivation for using cpio seems to be its -p flag for “pass-through” mode. This mode accepts paths on stdin and copies the files to a target directory, without creating or extracting an archive. What’s special about this is that it can preserve the relative path when doing so, whereas cp A/B C would copy B to C, echo A/B | cpio -pd C would result in the copy being at C/A/B. As I found out, doing this with other tools is not straightforward. It seems like rsync is the only other copying tool with close to drop-in compatible behavior. However, that’s a more complicated program than we want in the bootstrap-tools.
So I had to roll my own shell function based on cp --parents, which has a similar function but preserves the source’s read-only permissions making it impossible to copy multiple files into the same subdirectory. With the current implementation of CopyHierarchy build times went down to ~75 seconds, this time I re-ran nix-build --check many times because I couldn’t believe the difference. Yes, on my aging machine with a 7200 RPM HDD this is a 200x speedup.
Now that Nix 2.4+ has been released and the broad changes to the manual have made their way to the documentation it has finally become practical to incorporate the uninstallation instructions I wrote up in June. During the discussion of these changes expanding the installer to include uninstallation capabilities came up again. @abathur’s call for contribution to the installer, in particular adding uninstall capabilities, hasn’t been fullfilled yet so it’s still outstanding.