Writing a derivation to support optimized builds

Help
1

I’m trying to package some HPC software which would benefit greatly from build optimizations like enabling AVX where available. I was advised that using derivation arguments to manually specify build flags is to be avoided in Nixpkgs. I’m trying to follow wiki which is unfortunately confusing and contains errors. Also I’ve read the cross-compiling chapter in Nixpkgs manual, but it didn’t make things much clearer.

My test environment is following:

avx-test.nix:

{ stdenv
, lib
}:

let
  buildFlags = lib.optionalString stdenv.hostPlatform.avxSupport "-mavx";
in stdenv.mkDerivation {
  pname = "build-flag-tester";
  version = "0.0.1";

  src = ./avx_test;

  buildPhase = ''
    ${stdenv.cc}/bin/g++ ${buildFlags} avx_test.cc -o avx_test
  '';

  installPhase = ''
    install -Dm755 avx_test $out/avx_test
  '';
}

pkgs.nix:

let
  avxTestOverlay = self: super: {
    avx_test = super.callPackage ./avx_test.nix {};
  };
  pkgs = import <nixpkgs> {
    overlays = [ avxTestOverlay ];
    localSystem = {
      gcc.arch = "skylake";
      gcc.tune = "skylake";
      system = "x86_64-linux";
    };
  };
in pkgs

skylake feature is enabled via nix.extraOptions in system-features, I’m building with nix-build pkgs.nix -A avx-test.

Now the questions:

  1. Specifying localSystem as shown above makes Nix rebuild the whole stdenv. Is this the expected behavior?
  2. Using crossSystem instead makes Nix rebuild glibc and gcc. Why should the build tools be rebuild? Also when adding e.g. perl as a nativeBuildInput it tries to cross-build it and fails with “could not find a working compiler”.
  3. What is the best way to handle this situation? Am I doing something wrong? I’m not trying a classic cross-build (the software is basically x86_64 only), but only to supply custom (optimized) build flags in the correct way.
1 Like
2

  1. By changing localSystem you modified stdenv. This means that all packages that use stdenv will be rebuilt (since they will likely not reside in the Nix Binary Cache). Which is pretty much every package. I would assume if you want to go on such an endeavor, especially if you plan to move this to multiple machines, it would be beneficial if you have build and binary cache infrastructure in place.

  2. Haven’t used crossSystem alot, but so far haven’t seen such behavior. But your description is very vague so it’s hard to diagnose anything.

  3. If this is just to test with a single package, I’d probably make a derivation argument that enables these optimizations. Now you said in your post that you were advised against it, but since it seems to be the simplest way to get started, I’d like to know the arguments behind this advice. Having this as an argument makes the version with and without optimization different derivations which can be kept apart neatly.
    What is really dangerous are “clever” build scripts that detect processor features and then enable/disable these flags themselves. These may produce different outputs for the same derivation depending on the features of the build machine and make the builds non-reproducible.
    Note however, if you do this, the optimizations will only be enabled for the derivation where you specify this option. If the program uses some math library that does all the computational heavy lifting, you may not see much of an effect, because this library is still compiled without these options (though, if they are well maintained, they may be built as a “fat binary” and detect processor features at runtime).
    So the answer for the best way is unfortunately “It depends”.

3

That’s what I was using to initially test the derivation. I was specifying the flags via CMake arguments, and it worked fine (correct flags etc). Now I’d like to add it to Nixpkgs.

I’d like for the optimizations to have an effect on runtime dependencies, not build-time ones. In this regard specifying crossSystem seems more logical and prettier, but for some reason it still tries to rebuild derivations from nativeBuildInputs.

The relevant discussion: dealii: init at 9.4.0 by yl3dy · Pull Request #181948 · NixOS/nixpkgs · GitHub

avx_test.nix:

{ stdenv
, lib
, perl
}:

let
  buildFlags = lib.optionalString stdenv.hostPlatform.avxSupport "-mavx";
in stdenv.mkDerivation {
  pname = "build-flag-tester";
  version = "0.0.1";

  src = ./avx_test;

  nativeBuildInputs = [ perl ];

  buildPhase = ''
    ${stdenv.cc}/bin/x86_64-unknown-linux-gnu-g++ ${buildFlags} avx_test.cc -o avx_test
  '';

  installPhase = ''
    install -Dm755 avx_test $out/avx_test
    ${perl}/bin/perl -e 'print "Hello world\n";'
  '';
}

pkgs.nix:

let
  avxTestOverlay = self: super: {
    avx_test = super.callPackage ./avx_test.nix {};
  };
  pkgs = import <nixpkgs> {
    overlays = [ avxTestOverlay ];
    crossSystem = {
      gcc.arch = "skylake";
      gcc.tune = "skylake";
      system = "x86_64-linux";
    };
  };
in pkgs

The stdenv here is different from the default one:

% nix-build pkgs.nix -A stdenv.cc --no-out-link
/nix/store/9gmrlkwjgjyqg6p39hkb2xncmvx7v1l1-x86_64-unknown-linux-gnu-stage-final-gcc-wrapper-11.3.0
% nix-build '<nixpkgs>' -A stdenv.cc --no-out-link
/nix/store/ag2bpk0lzjvj409znklrz5krkpc5imzs-gcc-wrapper-11.3.0

Running the build produces the following:

% nix-build pkgs.nix -A avx_test      
these 7 derivations will be built:
  /nix/store/iw7b67652zk3lrgzpndgcv34iwjhrsxa-gmp-with-cxx-x86_64-unknown-linux-gnu-6.2.1.drv
  /nix/store/zwa6dj66p5yw1ncn3rah9niq71asw5lm-attr-x86_64-unknown-linux-gnu-2.5.1.drv
  /nix/store/laqyjc7z08w7zvaf6b9kpz7l94qlz7h3-acl-x86_64-unknown-linux-gnu-2.3.1.drv
  /nix/store/gx90wflddl5kvmfh47kb49iwk52364ms-coreutils-x86_64-unknown-linux-gnu-9.0.drv
  /nix/store/zzvqcnsqbjqm4nz6b2jvah47ry1kzypv-zlib-x86_64-unknown-linux-gnu-1.2.12.drv
  /nix/store/h8z6c7p4dhj8d08q18b63dcllv26dp17-perl-x86_64-unknown-linux-gnu-5.34.1.drv
  /nix/store/qnyrpk77j6q0ifh1lx025577g7is5qya-build-flag-tester-x86_64-unknown-linux-gnu-0.0.1.drv
building '/nix/store/zzvqcnsqbjqm4nz6b2jvah47ry1kzypv-zlib-x86_64-unknown-linux-gnu-1.2.12.drv'...
building '/nix/store/zwa6dj66p5yw1ncn3rah9niq71asw5lm-attr-x86_64-unknown-linux-gnu-2.5.1.drv'...
building '/nix/store/iw7b67652zk3lrgzpndgcv34iwjhrsxa-gmp-with-cxx-x86_64-unknown-linux-gnu-6.2.1.drv'...

...

error: builder for '/nix/store/iw7b67652zk3lrgzpndgcv34iwjhrsxa-gmp-with-cxx-x86_64-unknown-linux-gnu-6.2.1.drv' failed with exit code 1;
       last 10 log lines:
       > checking whether x86_64-unknown-linux-gnu-gcc is gcc... yes
       > checking compiler x86_64-unknown-linux-gnu-gcc -O2 -pedantic -fomit-frame-pointer -m64 ... no
       > checking ABI=x32
       > checking whether x86_64-unknown-linux-gnu-gcc is gcc... yes
       > checking compiler x86_64-unknown-linux-gnu-gcc -O2 -pedantic -fomit-frame-pointer -mx32 ... no
       > checking ABI=32
       > checking whether x86_64-unknown-linux-gnu-gcc is gcc... yes
       > checking compiler x86_64-unknown-linux-gnu-gcc -m32 -O2 -pedantic -fomit-frame-pointer ... no
       > checking compiler x86_64-unknown-linux-gnu-gcc -O2 -pedantic -fomit-frame-pointer ... no
       > configure: error: could not find a working compiler, see config.log for details
       For full logs, run 'nix log /nix/store/iw7b67652zk3lrgzpndgcv34iwjhrsxa-gmp-with-cxx-x86_64-unknown-linux-gnu-6.2.1.drv'.
error: 1 dependencies of derivation '/nix/store/gx90wflddl5kvmfh47kb49iwk52364ms-coreutils-x86_64-unknown-linux-gnu-9.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/h8z6c7p4dhj8d08q18b63dcllv26dp17-perl-x86_64-unknown-linux-gnu-5.34.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qnyrpk77j6q0ifh1lx025577g7is5qya-build-flag-tester-x86_64-unknown-linux-gnu-0.0.1.drv' failed to build