Installer test suite? :) small project(s), high-leverage, help wanted

@domenkozar landed work in March that establishes basic Nix installer tests on Ubuntu and macOS (how to set them on your fork).

This is already a Big Deal in my eyes, but there’s still a lot of low-hanging fruit here.

If you know anyone interested in working on (or promoting/boosting) a small high-leverage project, I think this is a good candidate.

It has a few distinct bits, so I think a few people could make short work of it without tons of coordination:

  1. Add single-user install tests.

    • Install tests currently use cachix/install-nix-action, which may limit it to multi-user installs (I know this is the default, at least). In order to add single-user test cases, we need to verify whether it supports single-user installs and if not work toward support in install-nix-action or build a replacement mechanism dedicated to testing?
    • The single-user macOS test should be allowed to fail (macOS single-user installs no longer supported in master after #4289).
  2. Develop the organization / logistics scaffold for building and running tests.

    I haven’t thought deeply about this, but superficially:

    • xfail support (so we can convert installer bug reports into tests)
    • modular components for setting up a given pre-condition (i.e., pre-create nixbld group/users as Nix will expect; pre-create them but with different IDs; pre-create non-nixbld group/users in the ID space Nix wants to use, etc.)
    • some way to ~declaratively compose them into a full set of test preconditions
    • if there’s an obvious approach, stub out how to handle having more things we’d like to test than we can reasonably throw at github; if there isn’t, this is probably a separate discussion
    • some way to avoid combinations we know won’t work (i.e., don’t try to run a slate of precondition tests for single-user macOS installs).
  3. Census existing installer issues, suss out a few of the most-common, and implement regular/xfail tests for them (and maybe doc/comment/stub the unimplemented?). My gut says this will be mostly shell/profile/rc issues and some user/groups.

    If extra razors help: I think it makes sense to prioritize plausible first-install conditions over artifacts of old Nix installs. Previous install problems are common, but most ~newbies run into them because the first install messed up.

Putting us on track to a well-covered installer will help Nix make more good first-impressions (or at least fewer bad ones). It’s a good first step toward documenting what shells/configurations/environments do/don’t and should/shouldn’t work. It’ll improve velocity/confidence for basically every installer fix/improvement.

5 Likes

There’s prior art by @grahamc at GitHub - grahamc/nix-install-matrix: A matrix of install tests for Nix.

3 Likes

Remove curl deps in install script by AnatoleLucet · Pull Request #5096 · NixOS/nix · GitHub surfaces a specific case: testing with presence/absence of wget/curl

Shameless bump :slight_smile:

This could pay big dividends when it comes to ensuring a good first-install experience by helping us improve and maintain the quality of the installer without being afraid to touch it.

4 Likes

With nix 2.4 released and work to establish a regular release schedule ongoing, a test suite should make it much easier to iteratively improve all kinds of install/reinstall/uninstall issues. :eyes:

3 Likes

Hi @abarthur. What would you like to see accomplished first for a darwin installer test matrix?

What scope do you have in mind? I’m a little hesitant to go into a lot of detail. (I don’t want the perfect to be the enemy of the good, I have no authority to dictate/gatekeep, and I want to avoid sucking any ~discovery/ownership/fun out of it.)

In terms of higher-level ~end goals, I’d like to see Nix end up with a way to run an install-test suite against every OS/platform/version it intends to support such that the result is visible and specific combinations can block release.

To that end:

  1. I think I would personally start with hunting down infrastructure people to get a sense of whether there are any paths to that goal.
  2. Stub one or more out (depending on the difficulty) with some basic tests (maybe: Nix is available in every Shell we try to set it up for, and a 2nd install attempt fails as expected?)

But, it’d also be great if someone “just” expanded on the existing Ubuntu + macOS jobs in Github Actions by hammering out the logistics for specifying, collecting, and running multiple distinct tests.

1 Like
Hosted by Flying Circus.