nixosTest with flake configurations

1

I am trying to write some integration tests for my machines in a flake.nix, however I seem to have run into an issue. The nixosTest function and underlying make-test-python.nix seem to expect a nixos module, but the flakes nixosConfigurations output wants a system that has already been evaluated by eval-config.nix.

I’ve tried:

nixosTest { machine = self.nixosConfigurations.NixOS; }
nixosTest { machine = self.nixosConfigurations.NixOS.config; }
nixosTest { machine = self.nixosConfigurations.NixOS.config.system.build.toplevel; }
nixosTest { machine = self.nixosConfigurations.NixOS.config.system.build.vm; }

to no avail.

2 Likes
3

I don’t think you can do it at this level. However you can have your nixos modules list in variable and pass it to both nixosSystem { modules = myModules; } and nixosTest { nodes.machine = {...}: { imports = myModules;};}

2 Likes
4

Indeed, I already knew I had to generalize hosts. Dumb question but I’m glad it has an answer for anyone as stupid as me :+1:

5

If anyone is interested I created an issue here Use different versions of nixpkgs for the nodes than the one used for the nixos test · Issue #195600 · NixOS/nixpkgs · GitHub.

6

I tried following your advice, but I run into an infinite recursion which I don’t fully understand, which I believe is related to access to config._module.args.

Any suggestions for how to test a nixosConfiguration with the testing framework?

7

Ive been down this rabbit hole myself and recently found a solution that fits my needs similar to you. Below is a link to a function that builds a nixosTest. I hope it helps and let me know if you are confused as to what is going on below.

function:

toplevels required:

Usage:

8

This thread shows up on Google, so responding for future reference: the nixosTest function is being deprecated as of 2024: testers.nixosTest: Deprecate by roberth · Pull Request #293891 · NixOS/nixpkgs · GitHub

Instead, runNixOSTest should be used, which coincidentally makes working with nixosConfigurations easier. This tester specifically helps you with the pkgs argument, which it tries to derive automatically. And, if you have overlay applied in your flake, you can use node.pkgsReadOnly = false; to pass the overlayed pkgs definition into your test.

A full example here with flake.parts: flake: add nixos tests by jhvst · Pull Request #92 · ponkila/homestaking-infra · GitHub
Moreover, in the diffset above, it’s possible to test a nixosConfiguration by doing a let binding for a nixosSystem before opening the flake schema. Take a look at the rest of the PR flake.nix for an example ponkila-ephemeral-sigma. You can test the full host by defining the imports = ponkila-ephemeral-sigma.modules; inside of the testable machine.

9

Anecdotally, I’ve had luck creating test machines that mirror flake nixosConfigurations (without a let-binding / post-hoc) by simply setting:
nodes.machine.imports = self.nixosConfigurations.${NAME}._module.args.modules;

Using flake-parts, in a flake module, e.g.

{
  inputs,
  self,
  withSystem,
  ...
}:
{
  perSystem = {pkgs, system, lib, ...}: {
    checks.machine-test = pkgs.testers.runNixOSTest {
      name = "machine-test";
      nodes.machine.imports = self.nixosConfigurations.machine._module.args.modules;
      hostPkgs = lib.mkForce pkgs;
      node = withSystem "x86_64-linux" ({pkgs, inputs', config, ...}: {
        pkgs = lib.mkForce pkgs;
        specialArgs = lib.mkForce {
          inherit inputs inputs' self;
          inherit (config) packages;
        };
      });
      testScript = ''
        ...
      '';
    };
  };
}

Are there any downsides to this approach that y’all can see?

One might be able to do the same for node.{specialArgs,pkgs} (inheriting them from nixosConfigurations.*._module); yet to try.