Code coverage using test driver

Hello, I’m currently working on the automation of my code coverage directly in github CI actions.
Using nix makes a lot of things easier so I tried to use nixpkgs test-driver to automate some normal usage actions as this project is a GUI application. But I soon realized that the test-driver shouldn’t be used outside of nixpkgs and so I couldn’t make it to run on my repo.

I’m asking you those two questions :

  • Is it possible to use the test-driver outside of nixpkgs ?
  • If not, what are my other nix options ?

Thanks to the IRC community on #nixos-dev:freenod.org, I found this excellent blog post from Gabriel Gonzalez that should answer this question Haskell for all: How to use NixOS for lightweight integration tests

NB : this post doesn’t really have anything to do with haskell if you’d ask me; I guess it’s there because postgrest is written in haskell.

I’m now having a critical hardware issue. The qemu test driver needs KVM kernel support which isn’t the case of most linux CI machines, github actions and travis to name only those.

I found this post Running Android Instrumented Tests on CI - from Bitrise.io to GitHub Actions - DEV Community that found a solution using github actions on macos machine because they’re delivered with HAXM support.

But nixos qemu test driver doesn’t support macos. So for the moment, my last solution is to move to https://bitrise.io even though I’m not developping mobile apps as they bring kvm support.

Using bitrise doesn’t quite work for the moment. The test-driver spawns and does detect kvm, but I get KVM errors

machine # Could not access KVM kernel module: Permission denied
machine # qemu-system-x86_64: failed to initialize kvm: Permission denied

Even though the kvm-ok from ubuntu ouputs :

INFO: /dev/kvm exists
KVM acceleration can be used

The full logs are :

INFO: /dev/kvm exists
KVM acceleration can be used
All done! ✨ 🍰 ✨
1 file would be left unchanged.
building '/nix/store/lal401x2pmwskcv3ar0bd07hilym1i6m-vm-test-run-unnamed.drv'...
starting VDE switch for network 1
/nix/store/xibghivp12jgk2xrwykpxxhy8wbmr5zi-python3-3.8.8/lib/python3.8/subprocess.py:848: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stdout = io.open(c2pread, 'rb', bufsize)
/nix/store/xibghivp12jgk2xrwykpxxhy8wbmr5zi-python3-3.8.8/lib/python3.8/subprocess.py:853: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stderr = io.open(errread, 'rb', bufsize)
running the VM test script
starting all VMs
machine: starting vm
/nix/store/xibghivp12jgk2xrwykpxxhy8wbmr5zi-python3-3.8.8/lib/python3.8/subprocess.py:848: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stdout = io.open(c2pread, 'rb', bufsize)
machine # Formatting '/tmp/nix-build-vm-test-run-unnamed.drv-0/vm-state-machine/machine.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=536870912 lazy_refcounts=off refcount_bits=16
machine # Could not access KVM kernel module: Permission denied
machine # qemu-system-x86_64: failed to initialize kvm: Permission denied
machine: QEMU running (pid 3146)
(0.70 seconds)
machine: must succeed: cp -r /nix/store/761xslwc9gfrfvplwk1vwwa2g9pf4zl1-source tabbed
machine: waiting for the VM to finish booting
machine: connected to guest root shell
machine: (connecting took 0.00 seconds)
(0.00 seconds)
error: 
Traceback (most recent call last):
  File "/nix/store/kiwv96511p8jcld8cfbgl6kaf44ph20x-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 901, in run_tests
    exec(tests, globals())
  File "<string>", line 1, in <module>
  File "<string>", line 6, in <module>
  File "/nix/store/kiwv96511p8jcld8cfbgl6kaf44ph20x-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 420, in succeed
    (status, out) = self.execute(command)
  File "/nix/store/kiwv96511p8jcld8cfbgl6kaf44ph20x-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 401, in execute
    self.shell.send(out_command.encode())
BrokenPipeError: [Errno 32] Broken pipe
cleaning up
killing machine (pid 3146)
(0.00 seconds)
builder for '/nix/store/lal401x2pmwskcv3ar0bd07hilym1i6m-vm-test-run-unnamed.drv' failed with exit code 1
error: build of '/nix/store/lal401x2pmwskcv3ar0bd07hilym1i6m-vm-test-run-unnamed.drv' failed

See GitHub - cachix/install-nix-action: Installs Nix on GitHub Actions for the supported platforms: Linux and macOS.

This fixed the first issue. But I got a new issue. The test runs fine on my NixOS Laptop with kvm enabled. But on a github workflow without kvm, I get a strange error. It seems like the files installed by environment.systemPackages aren’t accessible within the testScript, here is the corresponding CI run : Fix source gitignore · SCOTT-HAMILTON/tabbed@53ff518 · GitHub and the test.nix file :

let
  # For extra determinism
  nixpkgs = builtins.fetchTarball {
    url = "http://github.com/NixOS/nixpkgs/archive/d395190b24b27a65588f4539c423d9807ad8d4e7.tar.gz";
    sha256 = "0r1kj8gf97z9ydh36vmgrar1q4l9ggaqiygxjvp8jmr1948y0nh2";
  };
  pkgs = import nixpkgs {};
  # Single source of truth for all tutorial constants
  patched-alacritty = with pkgs; lib.traceValFn 
              (x: "Nixpkgs version ${lib.version}")
              (import ../alacritty.nix { inherit pkgs; });
  instrumented-tabbed = with pkgs; callPackage ../tabbed.nix {
    buildInstrumentedCoverage = true;
    inherit (nix-gitignore) gitignoreSource;
  };
  source = ../.;

  runTabbedAlacritty = pkgs.writeScriptBin "tabbed-alacritty" ''
    #!${pkgs.stdenv.shell}
    export TABBED_XEMBED_PORT_OPTION='--xembed-tcp-port'
    export TABBED_WORKING_DIR_OPTION='--working-directory'
  '';
in
  import "${nixpkgs}/nixos/tests/make-test-python.nix" ({ pkgs, ...}: {
    system = "x86_64-linux";

    nodes.machine = { nodes, config, pkgs, ... }:
      # let user = nodes.machine.config.users.users.alice;
      # in 
      {
      imports = [
        "${nixpkgs}/nixos/tests/common/user-account.nix"
        "${nixpkgs}/nixos/tests/common/x11.nix"
      ];
      environment.systemPackages = with pkgs; [
        patched-alacritty
        llvmPackages_11.bintools
        instrumented-tabbed
        runTabbedAlacritty
        less
        coreutils
      ];
      # test-support.displayManager.auto.user = user.name;
    };

    enableOCR = true;

    testScript = ''
      import os
      start_all()
      
      # Copy sources to tabbed directory
      machine.succeed("cp -r ${source} tabbed")
      machine.wait_for_x()
      machine.succeed(
          "export LLVM_PROFILE_FILE='tabbed-alacritty-%p.profraw'",
          "export TABBED_XEMBED_PORT_OPTION='--xembed-tcp-port'",
          "export TABBED_WORKING_DIR_OPTION='--working-directory'",
          'DISPLAY=:0.0 tabbed -cr 2 alacritty --embed "" &',
      )
      machine.sleep(10)
      machine.screenshot("window1")
      # machine.wait_for_text("alice@machine")
      
      #### Normal Use case sequences
      ### Goto /tmp
      machine.send_chars("cd /tmp")
      machine.send_key("ret")
      machine.sleep(10)
      machine.screenshot("tabbedtmp")
      ### Open a new tab
      machine.send_key("ctrl-shift-ret")
      machine.sleep(10)
      machine.screenshot("tabbedtmptab")
      ### Goto /proc
      machine.send_chars("cd /proc")
      machine.send_key("ret")
      machine.sleep(10)
      machine.screenshot("tabbedproc")
      ### Open a new tab
      machine.send_key("ctrl-shift-ret")
      machine.sleep(10)
      machine.screenshot("tabbedproctab")
      ### Goto ~ and exit proc tab
      machine.send_chars("cd ~")
      machine.send_key("ret")
      machine.send_chars("exit")
      machine.send_key("ret")
      machine.sleep(10)
      machine.screenshot("tabbedproctabexit")
      ### Goto ~ and exit tmp tab
      machine.send_chars("cd ~")
      machine.send_key("ret")
      machine.send_chars("exit")
      machine.send_key("ret")
      machine.sleep(10)
      machine.succeed(
          "llvm-profdata merge -sparse *.profraw -o tabbed-alacritty.profdata",
          "llvm-cov export ${instrumented-tabbed}/bin/tabbed -format=lcov -instr-profile=tabbed-alacritty.profdata > tabbed-alacritty.lcov",
      )
      machine.copy_from_vm("tabbed-alacritty.lcov", "coverage_data")
      machine.copy_from_vm("tabbed-alacritty.profdata", "coverage_data")
      out_dir = os.environ.get("out", os.getcwd())
      eprint(
          'Coverage data written to "{}/coverage_data/tabbed-alacritty.lcov"'.format(out_dir)
      )
      machine.screenshot("window2")
    '';
})```

So, with an autoPatchelfHook, I managed to get the binary running but despite building with -fprofile-instr-generate -fcoverage-mapping, I don’t get any profraw file (while it works locally). I ran a readelf -a binary | grep llvm and no coverage symbols showed up.

I doubt that the LLVM guys will have an idea of what could go wrong, it must be something with nix.

Ok, I finally got a working coverage CI !

2 Likes