Build matrix derivation for benchmarking and testing?

I would like to build a package with a set of compiler options and several different packages. Is there any work that has been done on a “build matrix” derivation. I’m looking to run a test suite for each build configuration and benchmark the performance with some plots.

In the end it would be just some sort of map over a list of c compilers and a list of benchmarks I think.
You can use the overrideCC function to set a different stdenv compiler and then use this stdenv to build the benchmarks in question.

In case you need to run SPEC, I have used the following: GitHub - nix-community/nix-environments: Repository to maintain out-of-tree shell.nix files (maintainer=@mic92)

Thanks @Mic92. Yes the map over overrideCC could just be some simple shell script that calls a build derivation with different options or a nix map with a derivation that depends on all of the derivations created in the map. Was wondering if anyone had done anything more complex than this. I was unaware of SPEC but thanks for sharing that.

So it looks like there is no magic derivation it is simple enough to create the build matrix and this is the recommended approach.

Don’t use SPEC unless you have to :slight_smile: It costs money and has a terrible installer.

I’ve put together a few ‘build matrix’-style expressions
but don’t think any are sufficiently general to make integration
less painful than the map-over-desired-envs approach described above.

That said, I would be interested in helping create such a common
abstraction-- it’s a common requirement and a great demonstration
of the sort of thing Nix makes easier.

Except right now it requires understanding all the pieces enough
to roll your own-- which means it’s much less accessible than
a list of compilers in a CI yaml file or w/e.

And maybe pushing this further might help refine the underlying
abstractions as we’ve found often to happen with these things :).

FWIW, some examples I’ve put together for CI purposes
might be helpful for ideas even if they’re not necessarily
the best way to accomplish precisely what you’re asking :slight_smile:.

Anyway, linky:

1 Like