pkgsCross.musl64 does not work with clangStdenvs and newer clangs

Help
1

Hi, i would like to crosscompile complex C++ applications to statically linked apps on top of musl.
One specialty of these apps is that they must be compiled with newer versions of clang (this is a requirement that i would like to get rid of, but unfortunately can’t).

Let’s have a look at the minimal example that triggers this behavior:

  • Create a mini derivation with a hello-world C++ app
  • create an overlay that results in the availability of pkgs.hello-app
  • obtain nixpkgs from current master branch (before anyone asks: 20.09 also doesnt work)
  • build the app
let
  mainCpp = builtins.toFile "main.cpp" ''
    #include <iostream>
    int main() { std::cout << "hello\n"; }
  '';

  deriv = { clang_11, overrideCC, clangStdenv }:
    # this already does not work. refering to as error type A
    #clangStdenv.mkDerivation {
    # this is what i finally want to to. refering to this error as type B
    (overrideCC clangStdenv clang_11).mkDerivation {
      name = "hello-app";
      src = mainCpp;
      unpackPhase = ":";
      buildPhase = "c++ $src -o hello";
      installPhase = "mkdir -p $out/bin && install -m755 hello $out/bin/hello";
    };
  overlay = self: super: {
    hello-app = self.callPackage deriv {};
  };
  nixpkgs = builtins.fetchTarball {
    url = "https://github.com/nixos/nixpkgs/archive/f217c0ea7c148ddc0103347051555c7c252dcafb.tar.gz";
    sha256 = "0cyksxg2lnzxd0pss09rmmk2c2axz0lf9wvgvfng59nwf8dpq2kf";
  };
  pkgs = import nixpkgs {
    overlays = [ overlay ];
  };
in
  # works:
  #pkgs.hello-app

  # the following both attributes do not work:
  # type B error
  # error: attribute 'llvmPackages_11' missing, at /nix/store/pl5nzbz7s187b7x2scj27b4wpkm1g93h-source/pkgs/top-level/all-packages.nix:10374:27
  #pkgs.pkgsCross.musl64.hello-app
  pkgs.pkgsCross.musl64.pkgsStatic.hello-app

The code contains comments with the error messages. For clarity they are repeated in the following:

for pkgs.pkgsCross.musl64.pkgsStatic.hello-app in combination with the clang11 clangStdenv i get error type B:

$ nix-build
error: attribute 'llvmPackages_11' missing, at /nix/store/pl5nzbz7s187b7x2scj27b4wpkm1g93h-source/pkgs/top-level/all-packages.nix:10374:27
(use '--show-trace' to show detailed location information)

for pkgs.pkgsCross.musl64.pkgsStatic.hello-app in combination with the cdefault clangStdenv i get error type A:

$ nix-build
these derivations will be built:
  /nix/store/2vi59br2ximhirad1ccg88lbvrplpgd0-hello-app-x86_64-unknown-linux-musl.drv
building '/nix/store/2vi59br2ximhirad1ccg88lbvrplpgd0-hello-app-x86_64-unknown-linux-musl.drv'...
unpacking sources
patching sources
updateAutotoolsGnuConfigScriptsPhase
configuring
no configure script, doing nothing
building
/nix/store/k29y87fvip8g0wvh6azg164m4xpghjjq-stdenv-linux/setup: line 1308: c++: command not found
builder for '/nix/store/2vi59br2ximhirad1ccg88lbvrplpgd0-hello-app-x86_64-unknown-linux-musl.drv' failed with exit code 127
error: build of '/nix/store/2vi59br2ximhirad1ccg88lbvrplpgd0-hello-app-x86_64-unknown-linux-musl.drv' failed

What is wrong here? I would expect that this just works, or did i do anything wrong?

3

I made some progress on this and came up with another minimal example:

# static.nix
let
  mainCpp = builtins.toFile "main.cpp" ''
    #include <iostream>
    int main() { std::cout << "hello\n"; }
  '';

  deriv = { stdenv }:
    stdenv.mkDerivation {
      name = "hello-app";
      src = mainCpp;
      unpackPhase = ":";
      patchPhase = "env";
      buildPhase = "$CXX $src -o hello -static";
      installPhase = "mkdir -p $out/bin && install -m755 hello $out/bin/hello";
    };
  overlay = self: super: {
    hello-app = self.callPackage deriv {};
  };
  nixpkgs = builtins.fetchTarball {
    url = "https://github.com/nixos/nixpkgs/archive/f217c0ea7c148ddc0103347051555c7c252dcafb.tar.gz";
    sha256 = "0cyksxg2lnzxd0pss09rmmk2c2axz0lf9wvgvfng59nwf8dpq2kf";
  };
  pkgs = import nixpkgs {
    overlays = [ overlay ];
  };
in
{
  hello-static-gcc = pkgs.pkgsCross.musl64.pkgsStatic.hello-app;
  hello-static-clang = pkgs.pkgsCross.musl64.pkgsStatic.hello-app.override {
    stdenv = pkgs.pkgsCross.musl64.llvmPackages.libcxxStdenv;
  };
}

the GCC version works:

$ nix-build static.nix -A hello-static-gcc
/nix/store/sk62aghbkw1rqmyaiypgd6iqg8j8fdj2-hello-app-x86_64-unknown-linux-musl
$ ldd result/bin/hello 
	not a dynamic executable

the clang version does not:

$ nix-build static.nix -A hello-static-clang
...
(.text+0x2f): undefined reference to `std::logic_error::~logic_error()'
/nix/store/2v3vkva29310005gipcqhdw9hxr57j6l-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/91ndwa08nid0k0mb0sdjbwjhl0f922vn-libc++-7.1.0-x86_64-unknown-linux-musl/lib/libc++.a(future.cpp.o):(.data.rel.ro._ZTINSt3__112future_errorE[_ZTINSt3__112future_errorE]+0x10): undefined reference to `typeinfo for std::logic_error'
/nix/store/2v3vkva29310005gipcqhdw9hxr57j6l-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ld: /nix/store/91ndwa08nid0k0mb0sdjbwjhl0f922vn-libc++-7.1.0-x86_64-unknown-linux-musl/lib/libc++.a(future.cpp.o):(.data.rel.ro._ZTVNSt3__112future_errorE[_ZTVNSt3__112future_errorE]+0x20): undefined reference to `std::logic_error::what() const'
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
builder for '/nix/store/ir55za0pv1nzavv0ks4h354fcx82d5gf-hello-app-x86_64-unknown-linux-musl.drv' failed with exit code 1
error: build of '/nix/store/ir55za0pv1nzavv0ks4h354fcx82d5gf-hello-app-x86_64-unknown-linux-musl.drv' failed

there’s lots of the c++ runtime stuff that seems to not be statically linkeable.
In the derivations of LLVM/clang/libc++(abi) there are already a few parameters that would allow for static linking, but i don’t really see how to activate them without patching nixpkgs.

Does anybody know if this is currently even supposed to work at all, or give me some pointers what would be the best way to enable that?
…

1 Like