HPC experiments with multiple architectures

Help
1

Hi, at our supercomputing center we are using nix for some experiments where we want to have a very fine control over all the dependencies of several benchmarks. We have implemented an overlay which extends nixpkgs with common HPC tools, libs and compilers along with a set of parametric scripts to define experiments where we can vary the compiler or the MPI implementation (recursively in all dependencies).

The benchmarks require details about the CPU to be able to use optimized features such as AVX512 or FMA. Additionally, we need to store more details about the hardware such as the cache sizes, which are typically used to optimize data structures at build time. Until now we have those details outside the nixpkgs platform attribute, but I was wondering whether it would make sense to extend the stdenv.hostPlatform with those details to be used by the benchmark derivations in order to setup the build flags. If so, where should they live? I saw the avx512Support attribute, but I’m not sure how to use it. Also, what is the purpose of platform.gcc.arch ?

Rodrigo.

5 Likes
2

Hi,

I’m also using Nix for this use-case. I’ll not answer directly to your question but I’m willing to hear how you have implemented your overlay.

Here is how I solved the benchmarks customisation (from years now) by using a function customFlags to set the cflags, flags or ldflags per derivation. As an example, you could see withOpenMP to add openmp to a derivation. I define stackable layers into nixpkgs (in my private envs folder) for compilers/mpi/architecture flags/…

My nur-packages repo also provides some overlays for various compilers: AOCC, Arm, PGI, Intel Parallel Studio (old), Intel oneAPI, flang, gcc with ptx offloading, nvhpc (not pushed to github yet).