Getting benchmark results when packaging a Cabal project

Help
1

I have an example Haskell project with the following files:

mypkg.cabal:

name:                mypkg
version:             1.0.0.0
build-type:          Simple
cabal-version:       >=1.10

library my-lib
  exposed-modules: MyLib
  -- other-modules:
  build-depends:       base
  default-language:    Haskell2010

Benchmark benchmark
  type:           exitcode-stdio-1.0
  main-is:        benchmark.hs
  build-depends:  base, my-lib

MyLib.hs:

module MyLib where

f x = x + 1

benchmark.hs:

module Main where

main = do
  writeFile "results.csv" "instance, time"
  putStrLn "Benchmark run successfully."

I want to create a derivation which contains the file results.csv, generated by benchmark.hs.

I tried using cabal2nix --benchmark ., and obtained the following:

{ mkDerivation, base, stdenv }:
mkDerivation {
  pname = "mypkg";
  version = "1.0.0.0";
  src = ./.;
  isLibrary = false;
  isExecutable = false;
  libraryHaskellDepends = [ base ];
  benchmarkHaskellDepends = [ base ];
  doHaddock = false;
  doBenchmark = true;
  license = "unknown";
  hydraPlatforms = stdenv.lib.platforms.none;
}

The issue is, if I now run nix-build -E "with import <nixpkgs> {}; haskellPackages.callPackage ./default.nix {}", I obtain a derivation with only the following files:

$ find /nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0/lib
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0/lib/ghc-8.10.2
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0/lib/ghc-8.10.2/x86_64-linux-ghc-8.10.2
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0/lib/ghc-8.10.2/x86_64-linux-ghc-8.10.2/mypkg-1.0.0.0-HTw6DKrKRbGAoH2PO8YOES-my-lib
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0/lib/ghc-8.10.2/x86_64-linux-ghc-8.10.2/mypkg-1.0.0.0-HTw6DKrKRbGAoH2PO8YOES-my-lib/libHSmypkg-1.0.0.0-HTw6DKrKRbGAoH2PO8YOES-my-lib.a
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0/lib/ghc-8.10.2/x86_64-linux-ghc-8.10.2/mypkg-1.0.0.0-HTw6DKrKRbGAoH2PO8YOES-my-lib/libHSmypkg-1.0.0.0-HTw6DKrKRbGAoH2PO8YOES-my-lib_p.a
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0/lib/ghc-8.10.2/x86_64-linux-ghc-8.10.2/mypkg-1.0.0.0-HTw6DKrKRbGAoH2PO8YOES-my-lib/MyLib.dyn_hi
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0/lib/ghc-8.10.2/x86_64-linux-ghc-8.10.2/mypkg-1.0.0.0-HTw6DKrKRbGAoH2PO8YOES-my-lib/MyLib.hi
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0/lib/ghc-8.10.2/x86_64-linux-ghc-8.10.2/mypkg-1.0.0.0-HTw6DKrKRbGAoH2PO8YOES-my-lib/MyLib.p_hi
/nix/store/72wv9yf0rhfb83qxhm0sms31ksgjyqk6-mypkg-1.0.0.0/lib/ghc-8.10.2/x86_64-linux-ghc-8.10.2/libHSmypkg-1.0.0.0-HTw6DKrKRbGAoH2PO8YOES-my-lib-ghc8.10.2.so

I need somehow to execute cabal bench and cp results.csv $out, but cabal by default looks at the users home directory, which of course breaks the build process.

How can I change default.nix to run the benchmark and install results.csv? It would also be nice if I could avoid installing the library alongside the result, as I only need that one file.

Installing only the benchmark as a binary that I can run later would also be fine.

Many thanks in advance!

And in case you’re wondering what I’m trying to achieve: I want to have a self-documented way of benchmarking different versions of my code against the same set of benchmarks in order to know how specific changes affect the performance of the program. The goal in the end is to have another derivation which produces a plot comparing all different versions, which is the reason why I only need the .csv file instead of the whole library.

2

Off the top of my head, I don’t know how to do what you’re asking, but if I was going to try to figure this out, I’d start with using overrideCabal to override the postBuild hook. Or possibly something like postCheck or postFixup or postInstall. I’m not sure exactly what phase Cabal runs the benchmarks:

https://github.com/NixOS/nixpkgs/blob/80418f65eb91c808544a092c9713cc2c9ce913b5/pkgs/development/haskell-modules/generic-builder.nix#L66

https://github.com/NixOS/nixpkgs/blob/80418f65eb91c808544a092c9713cc2c9ce913b5/pkgs/development/haskell-modules/generic-builder.nix#L437-L441

You could start by just doing an ls and seeing what the file structure looks like. You could then start start drilling down to directories that look interesting and try to figure out where the results.csv file is created.

When you find the results.csv file, you could copy it to somewhere in the $out directory, similar to a normal derivation.

Another possibility would be to create a new output called something like benchmark-results, and copy your results to there. That way, you’d have a single result derivation that contained the benchmark results.

If you’d like any more specific help about this, I’d recommend creating an example repo that shows the problem you’re having. For someone that is trying to help you, it’s much easier to git clone a repo and then just run nix-build, rather than to copy all of the required files from your post.

1 Like
3

Thanks for the help and the links. I managed to solve the problem in the end by using the following default.nix:

{ mkDerivation, base, stdenv }:

mkDerivation {
  pname = "mypkg";
  version = "1.0.0.0";
  src = ./.;
  isLibrary = false;
  isExecutable = false;
  libraryHaskellDepends = [ base ];
  benchmarkHaskellDepends = [ base ];
  doHaddock = false;
  doBenchmark = true;
  license = "unknown";
  hydraPlatforms = stdenv.lib.platforms.none;
  postBuild = ''
    # Run benchmarks
    ./Setup bench
  '';
  installPhase = ''
    mkdir -p $out/results
    mkdir -p $out/bin
    # Install benchmark results
    cp results.csv $out/results/
    # Install benchmark binary
    cp dist/build/benchmark/benchmark $out/bin/
  '';
}

This creates a derivation with the benchmark binary and the results of the benchmarks, but without the library. Which is precisely what I needed.

1 Like