Hello, I’m currently working on the automation of my code coverage directly in github CI actions.
Using nix makes a lot of things easier so I tried to use nixpkgs test-driver to automate some normal usage actions as this project is a GUI application. But I soon realized that the test-driver shouldn’t be used outside of nixpkgs and so I couldn’t make it to run on my repo.
I’m asking you those two questions :
- Is it possible to use the test-driver outside of nixpkgs ?
- If not, what are my other nix options ?
1 Like
Thanks to the IRC community on #nixos-dev:freenod.org, I found this excellent blog post from Gabriel Gonzalez that should answer this question Haskell for all: How to use NixOS for lightweight integration tests
NB : this post doesn’t really have anything to do with haskell if you’d ask me; I guess it’s there because postgrest is written in haskell.
I’m now having a critical hardware issue. The qemu test driver needs KVM kernel support which isn’t the case of most linux CI machines, github actions and travis to name only those.
I found this post Running Android Instrumented Tests on CI - from Bitrise.io to GitHub Actions - DEV Community that found a solution using github actions on macos machine because they’re delivered with HAXM support.
But nixos qemu test driver doesn’t support macos. So for the moment, my last solution is to move to https://bitrise.io even though I’m not developping mobile apps as they bring kvm support.
Using bitrise doesn’t quite work for the moment. The test-driver spawns and does detect kvm, but I get KVM errors
machine # Could not access KVM kernel module: Permission denied
machine # qemu-system-x86_64: failed to initialize kvm: Permission denied
Even though the kvm-ok from ubuntu ouputs :
INFO: /dev/kvm exists
KVM acceleration can be used
The full logs are :
INFO: /dev/kvm exists
KVM acceleration can be used
All done! ✨ 🍰 ✨
1 file would be left unchanged.
building '/nix/store/lal401x2pmwskcv3ar0bd07hilym1i6m-vm-test-run-unnamed.drv'...
starting VDE switch for network 1
/nix/store/xibghivp12jgk2xrwykpxxhy8wbmr5zi-python3-3.8.8/lib/python3.8/subprocess.py:848: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
self.stdout = io.open(c2pread, 'rb', bufsize)
/nix/store/xibghivp12jgk2xrwykpxxhy8wbmr5zi-python3-3.8.8/lib/python3.8/subprocess.py:853: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
self.stderr = io.open(errread, 'rb', bufsize)
running the VM test script
starting all VMs
machine: starting vm
/nix/store/xibghivp12jgk2xrwykpxxhy8wbmr5zi-python3-3.8.8/lib/python3.8/subprocess.py:848: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
self.stdout = io.open(c2pread, 'rb', bufsize)
machine # Formatting '/tmp/nix-build-vm-test-run-unnamed.drv-0/vm-state-machine/machine.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=536870912 lazy_refcounts=off refcount_bits=16
machine # Could not access KVM kernel module: Permission denied
machine # qemu-system-x86_64: failed to initialize kvm: Permission denied
machine: QEMU running (pid 3146)
(0.70 seconds)
machine: must succeed: cp -r /nix/store/761xslwc9gfrfvplwk1vwwa2g9pf4zl1-source tabbed
machine: waiting for the VM to finish booting
machine: connected to guest root shell
machine: (connecting took 0.00 seconds)
(0.00 seconds)
error:
Traceback (most recent call last):
File "/nix/store/kiwv96511p8jcld8cfbgl6kaf44ph20x-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 901, in run_tests
exec(tests, globals())
File "<string>", line 1, in <module>
File "<string>", line 6, in <module>
File "/nix/store/kiwv96511p8jcld8cfbgl6kaf44ph20x-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 420, in succeed
(status, out) = self.execute(command)
File "/nix/store/kiwv96511p8jcld8cfbgl6kaf44ph20x-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 401, in execute
self.shell.send(out_command.encode())
BrokenPipeError: [Errno 32] Broken pipe
cleaning up
killing machine (pid 3146)
(0.00 seconds)
builder for '/nix/store/lal401x2pmwskcv3ar0bd07hilym1i6m-vm-test-run-unnamed.drv' failed with exit code 1
error: build of '/nix/store/lal401x2pmwskcv3ar0bd07hilym1i6m-vm-test-run-unnamed.drv' failed
This fixed the first issue. But I got a new issue. The test runs fine on my NixOS Laptop with kvm enabled. But on a github workflow without kvm, I get a strange error. It seems like the files installed by environment.systemPackages aren’t accessible within the testScript, here is the corresponding CI run : https://github.com/SCOTT-HAMILTON/tabbed/runs/2361085659 and the test.nix file :
let
# For extra determinism
nixpkgs = builtins.fetchTarball {
url = "http://github.com/NixOS/nixpkgs/archive/d395190b24b27a65588f4539c423d9807ad8d4e7.tar.gz";
sha256 = "0r1kj8gf97z9ydh36vmgrar1q4l9ggaqiygxjvp8jmr1948y0nh2";
};
pkgs = import nixpkgs {};
# Single source of truth for all tutorial constants
patched-alacritty = with pkgs; lib.traceValFn
(x: "Nixpkgs version ${lib.version}")
(import ../alacritty.nix { inherit pkgs; });
instrumented-tabbed = with pkgs; callPackage ../tabbed.nix {
buildInstrumentedCoverage = true;
inherit (nix-gitignore) gitignoreSource;
};
source = ../.;
runTabbedAlacritty = pkgs.writeScriptBin "tabbed-alacritty" ''
#!${pkgs.stdenv.shell}
export TABBED_XEMBED_PORT_OPTION='--xembed-tcp-port'
export TABBED_WORKING_DIR_OPTION='--working-directory'
'';
in
import "${nixpkgs}/nixos/tests/make-test-python.nix" ({ pkgs, ...}: {
system = "x86_64-linux";
nodes.machine = { nodes, config, pkgs, ... }:
# let user = nodes.machine.config.users.users.alice;
# in
{
imports = [
"${nixpkgs}/nixos/tests/common/user-account.nix"
"${nixpkgs}/nixos/tests/common/x11.nix"
];
environment.systemPackages = with pkgs; [
patched-alacritty
llvmPackages_11.bintools
instrumented-tabbed
runTabbedAlacritty
less
coreutils
];
# test-support.displayManager.auto.user = user.name;
};
enableOCR = true;
testScript = ''
import os
start_all()
# Copy sources to tabbed directory
machine.succeed("cp -r ${source} tabbed")
machine.wait_for_x()
machine.succeed(
"export LLVM_PROFILE_FILE='tabbed-alacritty-%p.profraw'",
"export TABBED_XEMBED_PORT_OPTION='--xembed-tcp-port'",
"export TABBED_WORKING_DIR_OPTION='--working-directory'",
'DISPLAY=:0.0 tabbed -cr 2 alacritty --embed "" &',
)
machine.sleep(10)
machine.screenshot("window1")
# machine.wait_for_text("alice@machine")
#### Normal Use case sequences
### Goto /tmp
machine.send_chars("cd /tmp")
machine.send_key("ret")
machine.sleep(10)
machine.screenshot("tabbedtmp")
### Open a new tab
machine.send_key("ctrl-shift-ret")
machine.sleep(10)
machine.screenshot("tabbedtmptab")
### Goto /proc
machine.send_chars("cd /proc")
machine.send_key("ret")
machine.sleep(10)
machine.screenshot("tabbedproc")
### Open a new tab
machine.send_key("ctrl-shift-ret")
machine.sleep(10)
machine.screenshot("tabbedproctab")
### Goto ~ and exit proc tab
machine.send_chars("cd ~")
machine.send_key("ret")
machine.send_chars("exit")
machine.send_key("ret")
machine.sleep(10)
machine.screenshot("tabbedproctabexit")
### Goto ~ and exit tmp tab
machine.send_chars("cd ~")
machine.send_key("ret")
machine.send_chars("exit")
machine.send_key("ret")
machine.sleep(10)
machine.succeed(
"llvm-profdata merge -sparse *.profraw -o tabbed-alacritty.profdata",
"llvm-cov export ${instrumented-tabbed}/bin/tabbed -format=lcov -instr-profile=tabbed-alacritty.profdata > tabbed-alacritty.lcov",
)
machine.copy_from_vm("tabbed-alacritty.lcov", "coverage_data")
machine.copy_from_vm("tabbed-alacritty.profdata", "coverage_data")
out_dir = os.environ.get("out", os.getcwd())
eprint(
'Coverage data written to "{}/coverage_data/tabbed-alacritty.lcov"'.format(out_dir)
)
machine.screenshot("window2")
'';
})```
1 Like
So, with an autoPatchelfHook, I managed to get the binary running but despite building with -fprofile-instr-generate -fcoverage-mapping, I don’t get any profraw file (while it works locally). I ran a readelf -a binary | grep llvm and no coverage symbols showed up.
I doubt that the LLVM guys will have an idea of what could go wrong, it must be something with nix.
Ok, I finally got a working coverage CI !
3 Likes