Running Tests for Custom Packages

Help
1

What’s the correct way to run my tests when I build?

I wrote a script and put it in a container, and it wasn’t quite doing what it was supposed to, so I added some tests. At the time, I created a shell.nix and just put a call to bats in there that invokes bats on the script. And I embedded all my tests in the script.

It worked great, found and fixed the bugs, and added some regression tests.

The problem now – what’s really the best way to run these tests? I tried adding a checkPhase but it doesn’t get executed during build. I tried adding doCheck=true; but they still don’t run.

default.nix:

let nixpkgs = <nixpkgs>; pkgs = import nixpkgs { config = {}; overlays = []; };
in
{
  package = pkgs.callPackage ./package.nix {};
  container = pkgs.callPackage ./container.nix {};
}

package.nix

{ lib, stdenv, config, }:                                                                                                               
let pkgs = import <nixpkgs> { config = {}; overlays = []; }; in
stdenv.mkDerivation {                                    
    name = "package";                                             
    version = "0";                                                
    src = ../scripts;                                             
    doCheck = true;   # edit: <- typo here was the main issue! doChek != doCheck                                               
                                                                  
    buildInputs = with pkgs; [
        bash       #runtime   
        coreutils  #runtime   
        gnugrep    #runtime   
        bats       #build/test time
    ];                    
                              
    nativeBuildInputs = [ pkgs.makeWrapper ];
                              
    installPhase = ''         
      mkdir -p $out/bin
      install -m755 script.sh $out/bin/entry-point.sh
      wrapProgram $out/bin/entry-point.sh \
        --prefix PATH : ${lib.makeBinPath [
            pkgs.bash         
            pkgs.coreutils    
            pkgs.gnugrep      
        ]}                    
    '';                       
                              
    checkPhase = ''           
      bats test-entry-point.sh
    '';                       
}

container.nix:

{                                                                                                                                       
  lib,                                                                                                                                  
  stdenv,                                                                                                                               
  }:                                                                                                                                    
let                                                                                                                                     
  pkgs = import <nixpkgs> { config = {}; overlays = []; };                                                                              
  package = pkgs.callPackage ./package.nix { inherit lib; inherit stdenv; };                                                            
in pkgs.dockerTools.buildImage {                                                                                                        
  name = "helper";                                                                                                                      
  tag = "nix";                                                                                                                          
  copyToRoot = pkgs.buildEnv {                                                                                                          
    name = "image-root";                                                                                                                
    paths = [ package pkgs.bash ];                                                                                                      
    pathsToLink = [ "/bin" "${package}" ];                                                                                              
  };                                                                                                                                    
  config = {                                                                                                                            
    User = "root";                                                                                                                      
    EntryPoint = [ "${package}/bin/entry-point.sh" ];                                                                                   
    cmd = [ "${pkgs.bash}/bin/bash" ];                                                                                                  
  };                                                                                                                                    
  runAsRoot = ''                                                                                                                        
    ${pkgs.dockerTools.shadowSetup}                                                                                                     
  '';                                                                                                                                   
}

File hierarchy looks like this:

root/
├── nix
│   ├── container.nix
│   ├── default.nix
│   ├── package.nix
│   └── shell.nix
└── scripts
    └── script.sh

I pull it all together thusly:

docker image load -i "$(nix-build nix -A container)"

The shell.nix has a shellHook which calls bats on ${package}/scripts/test-entry-point.sh and then does an exit $? to propogate the error code. All the pieces work, if the above doesn’t work because of a syntax or other issue, it’s probably just a typo on my part while making a simpler version of the real stuff for sharing. I just don’t know how to make the package run tests when I build it. All I have been able to find on google about it says rather nebulously to set doCheck=true but doesn’t say where that needs to be done. Also something about running nix develop but that looked interactive and I just want it to run when I build. I’ll probably resort to a justfile or a makefile to call the things in order, but it would be really nice if the nix-build would run it.

2

This typo won’t generate any errors, but it also won’t do what the properly spelled version would do…

3

Thank you so very much! It may seem simple to you, but I very much appreciate it.

Additionally, the buildPhase runs before the installPhase, so I removed test-specific setup from my original nix source above.

Also, my src is set to the scripts directory that contains the script to be tested, so I had to change the path on what bats takes as an argument.

Now, the bonus round:

There are test artifacts produced by running bats. They’re not obvious from the simplified version of my package above, but I tell it to place output in a test-output directory. How do I get those artifacts after the tests have finished running?

I thought I could define a path and checkPhase would be able to use it, but the following fails:

    artifacts = ../test-output;
    checkPhase = ''
        mkdir -p "${artifacts}"
        bats -F pretty -o "${artifacts}" --report-formatter junit script.sh
    '';

And the reason it fails is undefined variable: artifacts in the mkdir call within the checkPhase.

4

A derivation cannot write anything except its designated output store paths. (and the temporary build directory, which is deleted after the build completes)

I suspect the intention of what you wrote there was incompatible with that restriction.

A couple of things to keep in mind:

  • foo = bar; is not an assignment statement, it’s a key-value pair in a literal attrset. Except in an attrset defined with rec (which I recommend avoiding where feasible), this line does not put a variable named foo into scope, as you might have been thinking. This is the cause of the immediate error you ran into.
  • ${...} is a nix antiquote, so if you want bash variable substitution, you need to escape the $ like ''${...}.
  • If you wanted nix antiquoting, make sure you properly quote the result. It’s fine for a store path since there are strong restrictions on the characters that can appear in those anyway, but generally, you probably want to use lib.escapeShellArg for such things to be safe.
  • Nix antiquoting applied to a path type like ../test-output copies the path the the store, and puts the resulting store path into the string. As the store is immutable, this absolutely cannot be used to capture output of any sort.
5

Thank you for the additional clarification.

Sounds like it would be inappropriate all around.

I could write the test output into a build accessible path from the build phase instead of the test phase, then copy that into the final package, provided I dont make the build fail as a result of a failing test. So either I create a package for a build that should have failed during the check phase, or I split my tests up and have some run when creating the package and the others run in some sort of pipeline.