Approaches for building packages with proprietary compilers and libraries with nix

So I am looking to use nix and nixos in some HPC environments and a common question that comes up is if the software is optimized for the architecture. Often times this involves using a proprietary compiler (portland group compiler, intel compiler, cray compiler etc.) and numerical libraries such as intel MKL. As far as I am aware nix by default uses stdenv with gcc as a compiler. I am not so much asking for a full fledged solutions but rather if others have tried to do this before and what a good approach would be? Is this a big undertaking if I what to change the compiler for a small subset of packages?

I still consider myself a beginner with Nix but I am comfortable with the programming language itself and have contributed several packages to nix. What would a sane approach be? Would an overlay be used to change the default compiler and replace openblas with intel mkl? This would only require that a fraction of nix packages to be rebuilt. I see nix as a HUGE win for HPC systems especially now that nix allows the store to be in an arbitrary locations, not require root, and can use the binary cache.

1 Like

I would package the compiler/libraries in question as normal Nix binary packages (like Skype is packaged, or, unfortunately, SciLab).

Then try applying wrapCC and see if the result works.

Then I would try overrideCC in the same manner as it is used in Nixpkgs.

As for non-root non-/nix installation with binary cache use: if you want all three, you need user namespaces, and this feature is not always enabled, especially on servers (and there are some specific security risks related to this feature). So you might not be as lucky as you hoped.

Thank you for the feedback. So I may not be lucky enough to be able to use the binary cache (which is a bummer for build times). Maybe multiuser build times could be reduced by using a common directory such as the scratch directory that everyone can access.

I appreciate you showing wrapCC and overrideCC I see several examples of it used for clang in builds.

FWIW, I’m working on adding Intel Parallel Studio. Slow progress, but you can track it here: Add Intel Parallel Studio stdenv · Issue #32434 · NixOS/nixpkgs · GitHub

1 Like

The derivations for MKL are available from the NixOS-QChem project (I am not affiliated with them):

https://github.com/markuskowa/NixOS-QChem/blob/master/mkl/default.nix

I totally agree that Nix is a lifesaver for HPC clusters.

1 Like

Awesome. I think seeing all of this shows that people are making progress in this space. Happy to see some complicated codes being built and that working is being done on integrating the intel compiler.

Here is one more repository. It can be added as a nix channel. In particular, it has a derivation for Intel’s ICC compiler (it needs the Intel Parallel Studio to be installed using the official installer first):

An Intel stdenv stdenvIntel is implemented here: https://github.com/dguibert/nixpkgs-overlays/blob/master/intel-compilers-overlay.nix, it also provides the Intel MPI library.

It needs the Intel Compilers to be already installed (as in the nix-ciment-channel repository).

The derivation needs the cc-wrapper function to be modified to handle the Intel compilers (icc, icpc and ifort) (This is done in the set intel-compilers). This is also the key point to support other compilers like flang.

I’m currently giving packaging the intel compilers a try without them having to be preinstalled (See: GitHub - knedlsepp/nixpkgs at add-intel-parallel-studio-xe). It seems in terms of stdenvIntel you’ve got already a solution going. I noticed some duplication between nixpkgs/cc-wrapper and your overlay’s cc-wrapper. Could you give a quick outline what the differences are and why it was necessary to modify the existing wrapper. Could wrapCCWith achieve any of that?

I’ve a working derivation for both: preinstalled or from the archive
(for now from the parallel studio xe archive via the Open Source Initiative [1])
I aim to update it to download the cluster studio like your derivation,
to provide the support of Intel mpi (without having it preinstalled).

See in my (new) nur-packages repository (instead of my “old”
nixpkgs-overlays repo):

https://github.com/dguibert/nur-packages/blob/1c88c67fa6076a74ef8f6c3f1b44f66cc095fa2d/overlays/intel-compilers-overlay/default.nix#L77).

To sum up what is required for stdenvIntel:

  • The executables and the libraries are patched with patchelf. No need
    to source compilervars.sh anymore.
  • Compiler config files (icc.cfg, icpc.cfg and ifort.cfg) need to be
    created, to define the default libc include directory and the default
    linker.
  • “stackproctector” flag should not be added.

To answer your question about wrapCCWith, it need to be extended to
support intel compilers (icc, icpc and ifort)

With those modifications, the hello derivation compiles just fine with
stdenvIntel
(https://github.com/dguibert/nur-packages/blob/master/overlays/intel-compilers-overlay/default.nix#L77).

Hope this help :wink:

[1] Priority Support for Intel® oneAPI Base & HPC Toolkit

2 Likes

Has there been any more progress to fully packaging the Intel compilers? They are freeware nowadays.

1 Like