Cross-compilation issues with CGo, glibc, and aarch64-linux

Help
1

Hello!

I am trying to cross-compile a Go program to aarch64-linux (GOARCH=arm64) from a x86_64-linux host (without emulation). Compiling for x86_64-linux works:

$ nix-build default.nix
/nix/store/zs9981qyq1xzfp514xakap5xcmq99icq-parca-agent-26f5194fd9477ec8ccb0e35f9677275c09211d7e

But for aarch64-linux, I get 'gnu/stubs-32.h' file not found:

$ nix-build --arg crossSystem '"aarch64-linux"'
...
Building subPackage ./cmd/parca-agent
# runtime/cgo
In file included from _cgo_export.c:3:
In file included from /nix/store/mmmjp0xcmr0dm307jv1l34hvi95z6d8s-glibc-2.35-224-dev/include/stdlib.h:26:
In file included from /nix/store/mmmjp0xcmr0dm307jv1l34hvi95z6d8s-glibc-2.35-224-dev/include/bits/libc-header-start.h:33:
In file included from /nix/store/mmmjp0xcmr0dm307jv1l34hvi95z6d8s-glibc-2.35-224-dev/include/features.h:514:
/nix/store/mmmjp0xcmr0dm307jv1l34hvi95z6d8s-glibc-2.35-224-dev/include/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
...

I have seen this error in a handful of GitHub issues and posts here, but none of the answers help (e.g. using glibc_multi throws error: i686 Linux package set can only be used with the x86 family.)

Here is what I use (simplified and self-contained):

{ pkgs ? import
    (builtins.fetchTarball {
      # nixos-unstable as of Tue, 6 Dec 2022 08:21:00 +0530 (latest revision with libbpf 1.0.1)
      name = "nixos-unstable-2022-12-06";
      url = "https://github.com/nixos/nixpkgs/archive/b10a520017ac319c1e57b07742efd2bcc918d160.tar.gz";
      sha256 = "0jdf78gjrhpj501i0qd79sy7nd2sfwvd5y67ixjggmbj93a6i7kx";
    })
    { crossSystem = crossSystem; }
, crossSystem ? null
, runTests ? false
}:

let
  inherit (pkgs) pkgsBuildBuild pkgsHostTarget;
  # bpf = pkgsBuildBuild.callPackage ./bpf { pkgs = pkgs; };
  # Dummy BPF program for simplification
  bpf = pkgsBuildBuild.writeTextDir "cpu.bpf.o" "";
in
(pkgsBuildBuild.buildGo119Module.override {
  stdenv = pkgsBuildBuild.llvmPackages_14.stdenv;
}) rec {
  pname = "parca-agent";
  version = "26f5194fd9477ec8ccb0e35f9677275c09211d7e";

  src = pkgsBuildBuild.fetchFromGitHub {
    owner = "parca-dev";
    repo = pname;
    rev = version;
    sha256 = "sha256-e+VO/BmXSygfyjIlnkFgPW8jsQrcItX0icYI1xOY7pc=";
  };
  vendorSha256 = "sha256-knGgdAY9YNBfV0xkGc0GZq7hyHeR1C6yyGEpZIBzKDo=";

  nativeBuildInputs = with pkgsBuildBuild; [ nukeReferences ];

  buildInputs = with pkgsHostTarget; [
    elfutils.dev
    glibc.static
    libbpf
    zlib.static
  ];

  CGO_CFLAGS = "--target=${pkgsHostTarget.hostPlatform.config}";
  CGO_LDFLAGS = "--target=${pkgsHostTarget.hostPlatform.config} -fuse-ld=ld -lbpf";
  GOOS = pkgsHostTarget.go.GOOS;
  GOARCH = pkgsHostTarget.go.GOARCH;

  ldflags = [ "-extldflags=-static" ];

  tags = [ "osusergo" "netgo" ];

  subPackages = "cmd/parca-agent";
  # Tests for these packages have additional requirements
  excludedPackages = "internal/pprof pkg/profiler e2e";

  preBuild = ''
    cp -f ${bpf}/cpu.bpf.o ./pkg/profiler/cpu/cpu-profiler.bpf.o
  '';

  # Nuke any references to other Nix store paths
  # https://github.com/NixOS/nixpkgs/blob/ccb52a00240f7bc4db651ac6521d42316361df56/pkgs/build-support/nuke-references/default.nix#L1-L4
  postBuild = ''
    nuke-refs "$GOPATH/bin/parca-agent"
  '';

  # Not necessary and/or desired (e.g. strip, patchelf...)
  # https://nixos.org/manual/nixpkgs/stable/#ssec-fixup-phase
  dontFixup = true;

  doCheck = runTests;
  # Test all packages
  preCheck = ''
    unset subPackages
  '';
}

I suspect I need to override the libc in stdenv to use the one for the host platform, but I cannot figure out how. (I also have a requirement to compile with clang since it is what we have been using so far)

This is my first attempt at cross-compilation with Nix, so any tips on the overall approach would also be welcome :slight_smile:

2

I tried to configure the stdenv compiler to use the libraries of the host platform with overrideCC:

  stdenv = pkgsBuildBuild.overrideCC pkgsBuildBuild.llvmPackages_14.stdenv (
    pkgsBuildBuild.clang.override {
      libc = pkgsHostTarget.glibc;
      bintools = pkgs.pkgsBuildHost.clang.bintools.override {
        libc = pkgsHostTarget.glibc;
      };
      gccForLibs = pkgs.pkgsBuildHost.gcc.cc;
    }
  );

But I think my compiler config is still wrong :confused::

$ nix-build default.nix --arg crossSystem '"aarch64-linux"'
...
Building subPackage ./cmd/parca-agent
# runtime/cgo
gcc_amd64.S:25:8: error: unknown token in expression
 pushq %rbx
       ^
gcc_amd64.S:25:8: error: invalid operand
 pushq %rbx
       ^
gcc_amd64.S:26:8: error: unknown token in expression
 pushq %rbp
       ^
gcc_amd64.S:26:8: error: invalid operand
 pushq %rbp
       ^
gcc_amd64.S:27:8: error: unknown token in expression
 pushq %r12
       ^
gcc_amd64.S:27:8: error: invalid operand
 pushq %r12
       ^
gcc_amd64.S:28:8: error: unknown token in expression
 pushq %r13
       ^
gcc_amd64.S:28:8: error: invalid operand
 pushq %r13
       ^
gcc_amd64.S:29:8: error: unknown token in expression
 pushq %r14
       ^
gcc_amd64.S:29:8: error: invalid operand
 pushq %r14
       ^
gcc_amd64.S:30:8: error: unknown token in expression
 pushq %r15
       ^
gcc_amd64.S:30:8: error: invalid operand
 pushq %r15
       ^
gcc_amd64.S:37:7: error: unknown token in expression
 movq %rdi, %rbx
      ^
gcc_amd64.S:37:7: error: invalid operand
 movq %rdi, %rbx
      ^
gcc_amd64.S:38:7: error: unknown token in expression
 movq %rdx, %rdi
      ^
gcc_amd64.S:38:7: error: invalid operand
 movq %rdx, %rdi
      ^
gcc_amd64.S:39:7: error: unknown token in expression
 call *%rsi
      ^
gcc_amd64.S:39:7: error: invalid operand
 call *%rsi
      ^
gcc_amd64.S:40:7: error: unknown token in expression
 call *%rbx
      ^
gcc_amd64.S:40:7: error: invalid operand
 call *%rbx
      ^
gcc_amd64.S:43:7: error: unknown token in expression
 popq %r15
      ^
gcc_amd64.S:43:7: error: invalid operand
 popq %r15
      ^
gcc_amd64.S:44:7: error: unknown token in expression
 popq %r14
      ^
gcc_amd64.S:44:7: error: invalid operand
 popq %r14
      ^
gcc_amd64.S:45:7: error: unknown token in expression
 popq %r13
      ^
gcc_amd64.S:45:7: error: invalid operand
 popq %r13
      ^
gcc_amd64.S:46:7: error: unknown token in expression
 popq %r12
      ^
gcc_amd64.S:46:7: error: invalid operand
 popq %r12
      ^
gcc_amd64.S:47:7: error: unknown token in expression
 popq %rbp
      ^
gcc_amd64.S:47:7: error: invalid operand
 popq %rbp
      ^
gcc_amd64.S:48:7: error: unknown token in expression
 popq %rbx
      ^
gcc_amd64.S:48:7: error: invalid operand
 popq %rbx
      ^
...

I must be missing something about cross-compilation. Something I do not even know I do not know. :persevere:

3

Figured it out! :tada: The answer:

pkgs.pkgsHostTarget.buildGo119Module.override {
  stdenv = pkgsHostTarget.llvmPackages_14.stdenv;
}

Here is the fixed derivation:

{ pkgs ? import
    (builtins.fetchTarball {
      # nixos-unstable as of Tue, 6 Dec 2022 08:21:00 +0530 (latest revision with libbpf 1.0.1)
      name = "nixos-unstable-2022-12-06";
      url = "https://github.com/nixos/nixpkgs/archive/b10a520017ac319c1e57b07742efd2bcc918d160.tar.gz";
      sha256 = "0jdf78gjrhpj501i0qd79sy7nd2sfwvd5y67ixjggmbj93a6i7kx";
    })
    { crossSystem = crossSystem; }
, crossSystem ? null
, runTests ? false
}:

let
  inherit (pkgs) pkgsBuildHost pkgsHostTarget;
  # bpf = pkgsBuildHost.callPackage ./bpf { pkgs = pkgs; };
  # Dummy BPF program for simplification
  bpf = pkgsHostTarget.writeTextDir "cpu.bpf.o" "";
in
(pkgs.pkgsHostTarget.buildGo119Module.override {
  stdenv = pkgsHostTarget.llvmPackages_14.stdenv;
}) rec {
  pname = "parca-agent";
  version = "26f5194fd9477ec8ccb0e35f9677275c09211d7e";

  src = pkgsBuildHost.fetchFromGitHub {
    owner = "parca-dev";
    repo = pname;
    rev = version;
    sha256 = "sha256-e+VO/BmXSygfyjIlnkFgPW8jsQrcItX0icYI1xOY7pc=";
  };
  vendorSha256 = "sha256-knGgdAY9YNBfV0xkGc0GZq7hyHeR1C6yyGEpZIBzKDo=";

  nativeBuildInputs = with pkgsBuildHost; [
    nukeReferences
  ];

  buildInputs = with pkgsHostTarget; [
    elfutils.dev
    glibc.static
    libbpf
    zlib.static
  ];

  CGO_LDFLAGS = "-lbpf";

  ldflags = [ "-extldflags=-static" ];

  tags = [ "osusergo" "netgo" ];

  subPackages = "cmd/parca-agent";
  # Tests for these packages have additional requirements
  excludedPackages = "internal/pprof pkg/profiler e2e";

  preBuild = ''
    cp -f ${bpf}/cpu.bpf.o ./pkg/profiler/cpu/cpu-profiler.bpf.o
  '';

  # Nuke any references to other Nix store paths
  # https://github.com/NixOS/nixpkgs/blob/ccb52a00240f7bc4db651ac6521d42316361df56/pkgs/build-support/nuke-references/default.nix#L1-L4
  postBuild = ''
    nuke-refs "$GOPATH/bin/parca-agent"
  '';

  # Not necessary and/or desired (e.g. strip, patchelf...)
  # https://nixos.org/manual/nixpkgs/stable/#ssec-fixup-phase
  dontFixup = true;

  doCheck = runTests;
  # Test all packages
  preCheck = ''
    unset subPackages
  '';
}

I am still a bit confused about the pkgs<host><target> attributes, but it works, it just need some more testing to make sure everything is good.

Documentation for reference: Nixpkgs 22.11 manual

1 Like
4

Cool, this might come in handy for later reference…

As a side note; I’ve seen cases where CGO is on by default even when it would not have been necessary; this then can make sensitivity to arch-related issues much worse. So I settled for always explicitly setting CGO=0 unless it is explicitly necessary (usually due to a dependency like e.g. sqlite).