Improvements to the Python test runner

Development
1

Switching to Python for the NixOS tests has been great for us!

I’d like to suggest moving a bit further with a number of improvements I noticed are making life less than ideal at the moment:

  1. I’d like to place the Python scripts in separate files so that I can properly use the traceback info for line numbers with my editor.
    This has a number of follow-up problems like either ensuring antiquotation (which is not a function currently) or passing environment info to the module with a structure. Also, using proper files instead of readFile would help with the traceback output being more helpful.

  2. I’d like to add pytest’s assertion magic so that writing tests like this would give proper exception info instead of just empty assertion errors.

    assert rm['auditd']['summary']['actor']['primary'] == 'customer'
  1. I’d like to be able to add helper functions to the test runner both on a library level (like helpers for calling code that might require “waitfor/timeouts/retry” in a more flexible Pythonic fashion) and add helpers for calling things on the machines that ensure output can be dealt with more flexibly. I often find myself that I want the output of a test to be potentially helpful with debugging if it fails in hydra which means I’d really like to SEE what commands where run and what the output was. This is relatively easy to build, but I think we would profit from having a better standard library of those and establish common patterns that we can document.

Any more ideas?

2 Likes
2

Ad 1. One tiny annoyance with a separate file is that when viewed as a separate script the machine variables are undefined. Maybe we should use something like import machines; machines.hostname1 or alternatively use something like a pytest fixture notation where the machines are test parameters def test1(machines):

Something that would be mind-blowing would also be if one could invert the caller logic such that we could write a pytest plugin that would spin up the NixOS machines and then run the same script as a normal pytest test

1 Like
3

I’ve been thinking about the same, that would be great!

4

I filed Provide a way to customize the python used for the nixos test-driver. · Issue #138318 · NixOS/nixpkgs · GitHub regarding 3.

5

Right. That’s one of the annoyances I got right out of the box … :wink:

Integrating with the whole setup driving into the fixture machinery would be quite nice. I guess we would want to think twice which layer a dependency injection approach would make the most sense.

6

Actually, you can write assertions with error messages like this:

λ ma27 [~] → python
Python 3.8.11 (default, Jun 28 2021, 10:57:31)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> assert False, "foo"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: foo
7

Well. That’s like stone-age tools compared to the amount of work needed and the quality given when you look at the assertion magic from pytest :wink: