Installer test suite? :) small project(s), high-leverage, help wanted

@domenkozar landed work in March that establishes basic Nix installer tests on Ubuntu and macOS (how to set them on your fork).

This is already a Big Deal in my eyes, but there’s still a lot of low-hanging fruit here.

If you know anyone interested in working on (or promoting/boosting) a small high-leverage project, I think this is a good candidate.

It has a few distinct bits, so I think a few people could make short work of it without tons of coordination:

  1. Add single-user install tests.

    • Install tests currently use cachix/install-nix-action, which may limit it to multi-user installs (I know this is the default, at least). In order to add single-user test cases, we need to verify whether it supports single-user installs and if not work toward support in install-nix-action or build a replacement mechanism dedicated to testing?
    • The single-user macOS test should be allowed to fail (macOS single-user installs no longer supported in master after #4289).
  2. Develop the organization / logistics scaffold for building and running tests.

    I haven’t thought deeply about this, but superficially:

    • xfail support (so we can convert installer bug reports into tests)
    • modular components for setting up a given pre-condition (i.e., pre-create nixbld group/users as Nix will expect; pre-create them but with different IDs; pre-create non-nixbld group/users in the ID space Nix wants to use, etc.)
    • some way to ~declaratively compose them into a full set of test preconditions
    • if there’s an obvious approach, stub out how to handle having more things we’d like to test than we can reasonably throw at github; if there isn’t, this is probably a separate discussion
    • some way to avoid combinations we know won’t work (i.e., don’t try to run a slate of precondition tests for single-user macOS installs).
  3. Census existing installer issues, suss out a few of the most-common, and implement regular/xfail tests for them (and maybe doc/comment/stub the unimplemented?). My gut says this will be mostly shell/profile/rc issues and some user/groups.

    If extra razors help: I think it makes sense to prioritize plausible first-install conditions over artifacts of old Nix installs. Previous install problems are common, but most ~newbies run into them because the first install messed up.

Putting us on track to a well-covered installer will help Nix make more good first-impressions (or at least fewer bad ones). It’s a good first step toward documenting what shells/configurations/environments do/don’t and should/shouldn’t work. It’ll improve velocity/confidence for basically every installer fix/improvement.

3 Likes

There’s prior art by @grahamc at GitHub - grahamc/nix-install-matrix: A matrix of install tests for Nix.

2 Likes

Remove curl deps in install script by AnatoleLucet · Pull Request #5096 · NixOS/nix · GitHub surfaces a specific case: testing with presence/absence of wget/curl

Shameless bump :slight_smile:

This could pay big dividends when it comes to ensuring a good first-install experience by helping us improve and maintain the quality of the installer without being afraid to touch it.

2 Likes