Switching to Python for the NixOS tests has been great for us!
I’d like to suggest moving a bit further with a number of improvements I noticed are making life less than ideal at the moment:
I’d like to place the Python scripts in separate files so that I can properly use the traceback info for line numbers with my editor.
This has a number of follow-up problems like either ensuring antiquotation (which is not a function currently) or passing environment info to the module with a structure. Also, using proper files instead of readFile would help with the traceback output being more helpful.
I’d like to add pytest’s assertion magic so that writing tests like this would give proper exception info instead of just empty assertion errors.
assert rm['auditd']['summary']['actor']['primary'] == 'customer'
- I’d like to be able to add helper functions to the test runner both on a library level (like helpers for calling code that might require “waitfor/timeouts/retry” in a more flexible Pythonic fashion) and add helpers for calling things on the machines that ensure output can be dealt with more flexibly. I often find myself that I want the output of a test to be potentially helpful with debugging if it fails in hydra which means I’d really like to SEE what commands where run and what the output was. This is relatively easy to build, but I think we would profit from having a better standard library of those and establish common patterns that we can document.
Any more ideas?