There’s some value in the straightforward test, though idk how many users the detsys installer is adding. I think cachix adds 8? AFAIR user creation is a lot if not most of the time taken up by any install.
Thanks, I updated the main post to include the nixbuild one. Also, the DeterminateSystems one did a lot better on macos this time. I’m not sure if that is variability in the GitHub Actions system or the installer itself. More testing is needed.
There’s quite a bit of play in the performance of the runners IME. IDK if it’s just a heterogeneous set of hardware or if it’s just busy v. quiet devices.
Could do something like take the best of 10 for each as a floor?
Thanks for evaluating this! It’s great to see this comparison, and that our new installer doesn’t stack up too badly even though performance was a secondary concern during its development. It’s also a great poke for us to look into improving performance (though we will always choose robustness if we have to choose between the two!), especially in scenarios like CI where going fast makes a huge difference for how using Nix for development feels.
Yes, because you’re using everything without further configuration which is likely a good representation of how a “typical user” might use it. Defaults are important!
No, because the resulting installations are quite different — the choice of action and OS affects whether you get a daemon/multi-user installation or a non-daemon/single-user installation — the latter requires far fewer steps.
No, because GHA is quite a black box, and there are a lot of factors at play in the execution time which we can neither control nor observe (as @abathur points out).
Adding cases for specific installation types, so that similar installations can be compared in addition to the defaults. Setting install_options to --no-daemon--daemon for the Cachix action will make the Cachix installer action produce a single-user multi-user installation on both OSes instead of only on darwin. The Determinate Nix installer will (as of today) always* produce a multi-user installation.
* if the Determinate Nix installer is running on Linux and does not detect systemd, or if it is run with --init none, it will produce an installation resembling a multi-user installation but without a daemon service configured. This means that only root will be able to access the store, though any user can run software already present in the Nix store.
Without looking in on what the Cachix action is doing, I’m a little skeptical of this. Maybe the inverse? Single-user installs are disabled for macOS, so cachix would need to be patching out that guard and maybe taking some other custom steps to get a single-user macOS install.
Yes, I got things confused while writing that — I’ve adjusted my post accordingly. Thanks!
To elaborate on the difference: multi-user provides more solid sandboxing, but also requires more setup. So if installing Nix has to happen as part of every CI job and a reusable base image isn’t an option (as is currently the case in GHA, as far as I understand), the speed of single-user installation is a strong argument in its favour.
I don’t think there’s any technical reason that some consumers couldn’t set up single-user macOS installs if they have an opinionated use-case.
(But maybe there are some others. IIRC @domenkozar made a public argument for simplifying the install base. I’m not sure, but if the relaxed permission model of single-user mode makes it easier to build derivations that would fail on a multi-user install, there’s probably an argument for CI defaulting to the most-restrictive model–not getting bit by such an error will almost certainly save most projects more than the extra install time.)
The majority of the feedback when I was working on handling FileVault cleanly was that there it used sudo too much and the install left too many artifacts laying around to call it single-user with a straight face. I ~disabled the option to satisfy those concerns.
I suspect these concerns would still argue against “supporting” single-user installs, but perhaps not against documenting and/or maybe making it a little easier for people to DIY it?
(I’m of two minds here, though. As the person who currently provides most of the “support” for people having macOS problems, I’d be annoyed if this was done in such a way that there’s a noticeable uptick in the number of clueless people using said documentation to get their system into unexpected states and then expecting “us” to help them figure out resulting obscure errors that no one else can ever repro.)
Coincidentally, I sketched out some notes on what cobbling together a single-user install on macOS might look like this morning on Matrix for an unrelated reason. (Edit: but note that the long-term feasibility of this specific approach would depend on whether the detsys installer eventually becomes official upstream; the equivalent result might need its own fork or install plan or whatever.)
nixbuild/nix-quick-install-action uses single user installation, that’s why it can be so fast.
cachix/nix-install-action uses multi-user installation and installs as many users as there are cores, but it’s using multi-user installation.
While multi-user installation does take longer, it was always my goal to support sandboxing on Darwin, which as far as I’m aware isn’t possible without nix-daemon.