@domenkozar landed work in March that establishes basic Nix installer tests on Ubuntu and macOS (how to set them on your fork).
This is already a Big Deal in my eyes, but there’s still a lot of low-hanging fruit here.
If you know anyone interested in working on (or promoting/boosting) a small high-leverage project, I think this is a good candidate.
It has a few distinct bits, so I think a few people could make short work of it without tons of coordination:
-
Add single-user install tests.
- Install tests currently use cachix/install-nix-action, which may limit it to multi-user installs (I know this is the default, at least). In order to add single-user test cases, we need to verify whether it supports single-user installs and if not work toward support in install-nix-action or build a replacement mechanism dedicated to testing?
- The single-user macOS test should be allowed to fail (macOS single-user installs no longer supported in master after #4289).
-
Develop the organization / logistics scaffold for building and running tests.
I haven’t thought deeply about this, but superficially:
- xfail support (so we can convert installer bug reports into tests)
- modular components for setting up a given pre-condition (i.e., pre-create nixbld group/users as Nix will expect; pre-create them but with different IDs; pre-create non-nixbld group/users in the ID space Nix wants to use, etc.)
- some way to ~declaratively compose them into a full set of test preconditions
- if there’s an obvious approach, stub out how to handle having more things we’d like to test than we can reasonably throw at github; if there isn’t, this is probably a separate discussion
- some way to avoid combinations we know won’t work (i.e., don’t try to run a slate of precondition tests for single-user macOS installs).
-
Census existing installer issues, suss out a few of the most-common, and implement regular/xfail tests for them (and maybe doc/comment/stub the unimplemented?). My gut says this will be mostly shell/profile/rc issues and some user/groups.
If extra razors help: I think it makes sense to prioritize plausible first-install conditions over artifacts of old Nix installs. Previous install problems are common, but most ~newbies run into them because the first install messed up.
Putting us on track to a well-covered installer will help Nix make more good first-impressions (or at least fewer bad ones). It’s a good first step toward documenting what shells/configurations/environments do/don’t and should/shouldn’t work. It’ll improve velocity/confidence for basically every installer fix/improvement.
