Hardware-dependent NixOS tests

Development
1

Hi,

I’m working on adding usbrelay to NixOS (see this the draft PR). The package contains a simple CLI tool as well as a MQTT service to control USB-connected relays. To be sure that everything works as needed, I’d like to have an automated test of the whole setup, but it is a bit problematic right now. Below, I describe my current setup and at the end I’ve a few questions.

NixOS tests would be perfect fit for such automated testing, but they don’t see host-attached hardware devices and my tests require the hardware to be present. I’ve ended up with the following test (stored in usbrelayd.nix):

import ./make-test-python.nix ({ pkgs, ... }: {
  name = "usbrelayd";

  nodes.machine = {
    virtualisation.qemu.options = [
      "-device qemu-xhci"
      "-device usb-host,vendorid=0x16c0,productid=0x05df"
    ];
    services.usbrelayd.enable = true;
    systemd.services.usbrelayd = {
      after = [ "mosquitto.service" ];
    };
    services.mosquitto = {
      enable = true;
      listeners = [{
        acl = [ "pattern readwrite #" ];
        omitPasswordAuth = true;
        settings.allow_anonymous = true;
      }];
    };
    documentation.nixos.enable = false; # building nixos manual takes long time
  };

  testScript = ''
    machine.start()
    machine.wait_for_unit("usbrelayd.service")
  '';
})

Note that the test passes a specific USB device (the relay) from host to the guest system via virtualisation.qemu.options.

The test can be run in several ways, with different in level of success:

  • nix-build usbrelayd.nix -A driverInteractive && ./result/bin/nixos-test-driver --no-interactive

    This way, the test works out of the box and succeeds when the relay is connected to the host computer, but the command line is a bit cumbersome.

  • nix-build usbrelayd.nix doesn’t work for two reasons:

    • The non-interactive test driver uses qemu without support for USB redirection. This can be resolved by the following nixpkgs patch:

      --- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -23209,1 +23209,1 @@ with pkgs;
-  qemu_test = lowPrio (qemu.override { hostCpuOnly = true; nixosTestRunner = true; });
+  qemu_test = lowPrio (qemu.override { hostCpuOnly = true; nixosTestRunner = true; usbredirSupport = true; });

    • The test probably runs in a sandbox, which makes the host devices invisible even with the above patch. The --no-sandbox option to nix-build didn’t help. I guess that the sandbox is somehow hardcoded in nix-daemon.

  • nix-build usbrelayd.nix -A driver && ./result/bin/nixos-test-driver

    This works with the above patch to qemu_test and it’s less cumbersome than the first way.

Questions:

  • Would it be acceptable or desirable to have hardware-dependent tests like the above in nixpkgs? They would be useless for hydra, but maintainers can use them to test updates or run them in their CI equipped with proper hardware.

  • If yes, should the test report that it cannot be executed when the hardware is not available, perhaps by doing something like this?

    testScript = ''
  if os.waitstatus_to_exitcode(os.system("lsusb -d 16c0:05df")) != 0:
      print("No USB relay detected, skipping test")
      import sys
      sys.exit(2)
  # ...
'';

  • Shall we try to make such tests runnable by simple nix-build usbrelayd.nix? Would it be possible without loss of reproducibility for other (non-hw-dependent) tests?

4 Likes
2

I’d very much like to see hardware-dependent tests! For a long time, I wanted to automate tests of the Vulkan driver, which needs a (fitting) GPU in the system.
GPU passthrough into qemu is not really reliable though, due to hardware limitations and bugs, so testing on GPUs would only work well in a container or on the host system (I assume usb passthrough is more reliable, so qemu may be a good choice there).

4 Likes
3

I haven’t delved deep into this, but nix has support for custom system-features, which can be:

  • set for a system (system-features in nix.conf)
  • set for a derivation (requiredSystemFeatures array in runCommand, mkDerivation, etc)

Video demonstration: https://www.youtube.com/watch?v=RgKl8Jue4qM&t=3008s.

Maybe such mechanism can be used to specify custom hardware facilities, required for tests.

1 Like
4

You’d then have to pass through certain devices to the Nix sandbox on those machines but doesn’t sound impossible.

5

Perhaps, my main mistake here is that system-features affect build time, but NixOS tests need hardware at runtime. And moving tests into build phase is not a good idea.

6

There was a talk about that at nixcon Automating testing of NixOS on physical machines - media.ccc.de

7

There is makeImpureTest in nixpkgs now, which can be used to pass devices to the nix sandbox: https://github.com/NixOS/nixpkgs/blob/a5be212e00b6e2b0dff97ffd781a3f7026abd870/pkgs/build-support/make-impure-test.nix
It’s used e.g. here to run tests with a GPU: https://github.com/NixOS/nixpkgs/blob/a5be212e00b6e2b0dff97ffd781a3f7026abd870/pkgs/development/rocm-modules/5/clr/test-rocm-smi.nix

1 Like