Brainstorming: Improve nixos test VM logging

I recently implemented a slight impovement in visual UX for the nixos vm testing, so I’m a tiny bit familiar with the topic.

Problem

In the context of debugging nixos vm tests, the operator’s consciousness is quickly brainflooded with kilometers of (mostly non-relevant) log rolls, such as:

...
machine1 # [  733.667041] kube-addons[1257]: INFO: == Reconciling with addon-manager label ==
machine1 # [  734.124000] kube-addons[3042]: Warning: batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
...

Solution

Brainstorming. Here, in public. Hopefully.


I’ll make a start: I’d wish for a way to attach different terminals to different log filters so that I can monitor some important logs in real time and in different windows for different systemd units.

I think first I’d need a friendly pointer how I can teach systemd in the nixos-vm to output the logs in a more structured way. Even if that’s probably not necessary in the end as a regex filter can do, it might be just good to know.

Second, I’d need to have a sense of where the log stream, which is passed over a pipe to the python testing instrumentation, should be multiplexed into different terminals. Currently the testing instrumentation launches an embedded ptpython repl, hence I don’t see any hinderances why not launch tmux et al. somewhere around the same lines.

Possible UX:

./result/bin/nixos-test-driver \
  --suppress-unit-logs "kube,dmsq" \
  --split1 "kube-addons,apiserver" \
  --split2 "..." \
  --split3 "..."
# would result in 4 windows:
# 1 "managing" ptpython repl and 3 splits for log output only