nixosTest vs runTest

Help
1

I am having trouble understanding the big differences between runTest and nixosTest. The diffrences I know of are…

  • runTest allows including multiple tests and nixosTest can only be passed one test.
  • The various docs seem to learn towards runTest for any nixos related stuff (like modules) and nixosTest for packages. And the file paths also align with this (runTest is defined under nixos/lib/testing/default.nix and nixosTest at pkgs/build-support/testers/default.nix

But even with that, it seems like it would easy enough to use either function to run a single or multiple test (I am making the assumption that the code to use nixosTest to run multiple tests would be simple, someone correct me if I am wrong, especially if the purpose of runTest is to easily run multiple tests).

Also when trying to read threw the code for both, they seem to have significant amounts of different code (even if they do share some code). While I could not quite figure what the differences are, the code leads me to assume that they are likely to have significant differences.

I want to know the significant differences because I am looking to set up testing for my nixos system configs (the “Testing outside the NixOS project” use case from manual), and I can’t decided what I should use, or why to use one over the other.

5 Likes
3

Bumping this because I am interested in an answer as well.

4

Bump! It’s quite confusing to have them both at the same time.

nix.dev seems to recommend nixosTest, but tests built by nixosTest fails to run on Darwin even after PR NixOS/nixpkgs#282401.

5

nixosTest is supported legacy.

With the introduction of the module system for the NixOS test wiring, nixosTest became an awkward interface because when you pass it a function, it used to be callPackage-ed (and still is), whereas it is now more natural to pass the module arguments.
Since the darwin support, the callPackage behavior makes even less sense, because we’re dealing with two Nixpkgs sets, and the choice is arbitrary. Maybe you want host packages, or maybe you want packages that work in the VMs.

I think this is reason to really deprecate it now. runTest has existed for quite some time now, so there’s little reason not to switch.

Side note: imports into runTest don’t cause multiple tests to be created; they’re just modules that contribute to a single test. You can use it for things like common test setup. Here’s an example in hercules-ci-effects, for adding a real DNS server to the network.

3 Likes
6
2 Likes