Cannot build arm linux kernel on an actual arm device

netogallo · October 11, 2024, 9:10am

Greetings,

I was playing around building custom NixOS images. When building an image for an ARM device (the Pinetab2 which has a Rockchip), it failed with the message:

libbpf: failed to find '.BTF' ELF section in vmlinux
FAILED: load BTF from vmlinux: No data available
make[2]: *** [../scripts/Makefile.vmlinux:34: vmlinux] Error 255
make[2]: *** Deleting file 'vmlinux'
make[1]: *** [/build/linux-6.10.12/Makefile:1171: vmlinux] Error 2
make: *** [../Makefile:240: __sub-make] Error 2

I used the command below to validate that this failure happens with the official kernel from NixOS (not the patched kernel specific to the device).

nix-build -E "import <nixpkgs> {}" -A linuxKernel.kernels.linux_6_10 --no-substitute

Strangely, the image builds just fine (and much faster) on my x86_64 with QEMU using binfmt. The resulting image is also working fine on the PineTab2. I am just curious to know why it fails on the actual ARM device.

Ultimately, I guess few people would be affected by this as probably no ARM device (other than the MBP) has performance good enough for it to be worth the while to build kernels with it.

Atemu · October 11, 2024, 2:33pm

Are you running out of memory perhaps?

misuzu · October 11, 2024, 6:10pm

I can reproduce this error by running nix build github:NixOS/nixpkgs/nixos-unstable#legacyPackages.armv7l-linux.linuxKernel.kernels.linux_6_10 on the Community aarch64 NixOS box

  LD [M]  drivers/gpu/drm/amd/amdgpu/amdgpu.o
  AR      drivers/gpu/built-in.a
  AR      drivers/built-in.a
  AR      built-in.a
  AR      vmlinux.a
  LD      vmlinux.o
  OBJCOPY modules.builtin.modinfo
  GEN     modules.builtin
  MODPOST Module.symvers
  CC      .vmlinux.export.o
  UPD     include/generated/utsversion.h
  CC      init/version-timestamp.o
  KSYMS   .tmp_vmlinux0.kallsyms.S
  AS      .tmp_vmlinux0.kallsyms.o
  LD      .tmp_vmlinux1
  BTF     .tmp_vmlinux1.btf.o
memory exhausted
memory exhausted
  NM      .tmp_vmlinux1.syms
  KSYMS   .tmp_vmlinux1.kallsyms.S
  AS      .tmp_vmlinux1.kallsyms.o
  LD      .tmp_vmlinux2
  NM      .tmp_vmlinux2.syms
  KSYMS   .tmp_vmlinux2.kallsyms.S
  AS      .tmp_vmlinux2.kallsyms.o
  LD      vmlinux
  BTFIDS  vmlinux
libbpf: failed to find '.BTF' ELF section in vmlinux
FAILED: load BTF from vmlinux: No data available
make[2]: *** [../scripts/Makefile.vmlinux:34: vmlinux] Error 255
make[2]: *** Deleting file 'vmlinux'
make[1]: *** [/build/linux-6.10.13/Makefile:1171: vmlinux] Error 2
make: *** [../Makefile:240: __sub-make] Error 2

linux_6_11:

  LD [M]  drivers/gpu/drm/nouveau/nouveau.o
  AR      drivers/gpu/built-in.a
  AR      drivers/built-in.a
  AR      built-in.a
  AR      vmlinux.a
  LD      vmlinux.o
  OBJCOPY modules.builtin.modinfo
  GEN     modules.builtin
  MODPOST Module.symvers
  CC      .vmlinux.export.o
  UPD     include/generated/utsversion.h
  CC      init/version-timestamp.o
  KSYMS   .tmp_vmlinux0.kallsyms.S
  AS      .tmp_vmlinux0.kallsyms.o
  LD      .tmp_vmlinux1
  BTF     .tmp_vmlinux1.btf.o
memory exhausted
  NM      .tmp_vmlinux1.syms
  KSYMS   .tmp_vmlinux1.kallsyms.S
  AS      .tmp_vmlinux1.kallsyms.o
  LD      .tmp_vmlinux2
  NM      .tmp_vmlinux2.syms
  KSYMS   .tmp_vmlinux2.kallsyms.S
  AS      .tmp_vmlinux2.kallsyms.o
  LD      vmlinux
  BTFIDS  vmlinux
libbpf: failed to find '.BTF' ELF section in vmlinux
FAILED: load BTF from vmlinux: No data available
make[2]: *** [../scripts/Makefile.vmlinux:34: vmlinux] Error 255
make[2]: *** Deleting file 'vmlinux'
make[1]: *** [/build/linux-6.11.2/Makefile:1157: vmlinux] Error 2
make: *** [../Makefile:224: __sub-make] Error 2

linux_6_6:

  LD [M]  drivers/gpu/drm/nouveau/nouveau.o
  AR      drivers/gpu/built-in.a
  AR      drivers/built-in.a
  AR      built-in.a
  AR      vmlinux.a
  LD      vmlinux.o
  OBJCOPY modules.builtin.modinfo
  GEN     modules.builtin
  MODPOST Module.symvers
  CC      .vmlinux.export.o
  UPD     include/generated/utsversion.h
  CC      init/version-timestamp.o
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
memory exhausted
memory exhausted
  LD      .tmp_vmlinux.kallsyms1
  NM      .tmp_vmlinux.kallsyms1.syms
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.o
  LD      .tmp_vmlinux.kallsyms2
  NM      .tmp_vmlinux.kallsyms2.syms
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.o
  LD      vmlinux
  BTFIDS  vmlinux
libbpf: failed to find '.BTF' ELF section in vmlinux
FAILED: load BTF from vmlinux: No data available
make[2]: *** [../scripts/Makefile.vmlinux:37: vmlinux] Error 255
make[2]: *** Deleting file 'vmlinux'
make[1]: *** [/build/linux-6.6.54/Makefile:1164: vmlinux] Error 2
make: *** [../Makefile:234: __sub-make] Error 2

linux_6_1:

  LD [M]  drivers/gpu/drm/nouveau/nouveau.o
  AR      drivers/gpu/built-in.a
  AR      drivers/built-in.a
  AR      built-in.a
  AR      vmlinux.a
  LD      vmlinux.o
  OBJCOPY modules.builtin.modinfo
  GEN     modules.builtin
  MODPOST Module.symvers
  CC      .vmlinux.export.o
  UPD     include/generated/utsversion.h
  CC      init/version-timestamp.o
  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
[4976517] STRUCT shrink_control Error emitting BTF type
Encountered error while encoding BTF.
  LD      .tmp_vmlinux.kallsyms1
  NM      .tmp_vmlinux.kallsyms1.syms
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.o
  LD      .tmp_vmlinux.kallsyms2
  NM      .tmp_vmlinux.kallsyms2.syms
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.o
  LD      vmlinux
  BTFIDS  vmlinux
FAILED: load BTF from vmlinux: No such file or directory
make[1]: *** [../scripts/Makefile.vmlinux:34: vmlinux] Error 255
make[1]: *** Deleting file 'vmlinux'
make: *** [../Makefile:1250: vmlinux] Error 2

linux_5_15 is not affected

I think disabling CONFIG_DEBUG_INFO_BTF should allow you to build your kernel.

Atemu · October 11, 2024, 6:28pm

I have a hunch why that is.

The linux kernel build needed >30G worth of disk space for BTF files last time I checked. If your nix build directory is in memory, you likely won’t have enough memory.

The actual fix is to have enough space in your Nix build dir by i.e. moving it to /var/tmp.

misuzu · October 11, 2024, 6:40pm

The server has enough disk space:

$ df -h
Filesystem                  Size  Used Avail Use% Mounted on
devtmpfs                     13G     0   13G   0% /dev
tmpfs                       126G     0  126G   0% /dev/shm
tmpfs                        63G   18M   63G   1% /run
/dev/nvme0n1p2              1.8T  803G  976G  46% /
/dev/disk/by-label/persist   89M  126K   82M   1% /persist
tmpfs                       126G  1.9M  126G   1% /run/wrappers
tmpfs                        26G  4.0K   26G   1% /run/user/1044
tmpfs                        26G  4.0K   26G   1% /run/user/1030
tmpfs                        26G  4.0K   26G   1% /run/user/1070
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           250Gi       4.4Gi        77Gi        19Mi       168Gi       243Gi
Swap:             0B          0B          0B

Atemu · October 11, 2024, 7:08pm

Hm, is it actually running OOM then? You’d have to monitor the resources while it builds.

misuzu · October 11, 2024, 7:10pm

I doubt it. Most likely there’s a bug in pahole or something (32-bit apps can only address 2GB of RAM)

EDIT: Found this, probably related: [RFC] kbuild: bpf: Do not run pahole with -j on 32bit userspace

  -j, --jobs[=NR_JOBS]       run N jobs in parallel [default to number of
                             online processors + 10%]

I’ll try building the kernel with --cores 1, not sure if it’ll actually pass the 1 to the pahole

netogallo · October 11, 2024, 8:48pm

I don’t think it is memory or hhd. I have 8GB of ram and over 32GB of storage. I changed the location of the nix build directory so it is in the hhd rather than the ram (/tmp).

netogallo · October 11, 2024, 8:52pm

Here are my logs:

/nix/store/f3k0rdhcd2cx57phx755c2xixgifw5m5-binutils-2.42/bin/ld: warning: -z relro ignored
  AR      drivers/built-in.a
  CC [M]  drivers/peci/cpu.o
  LD [M]  drivers/counter/counter.o
/nix/store/f3k0rdhcd2cx57phx755c2xixgifw5m5-binutils-2.42/bin/ld: warning: -z relro ignored
  CC [M]  drivers/most/most_snd.o
  CC [M]  drivers/peci/controller/peci-npcm.o
  LD [M]  drivers/peci/peci.o
/nix/store/f3k0rdhcd2cx57phx755c2xixgifw5m5-binutils-2.42/bin/ld: warning: -z relro ignored
  LD [M]  drivers/most/most_core.o
/nix/store/f3k0rdhcd2cx57phx755c2xixgifw5m5-binutils-2.42/bin/ld: warning: -z relro ignored
  LD [M]  drivers/peci/peci-cpu.o
/nix/store/f3k0rdhcd2cx57phx755c2xixgifw5m5-binutils-2.42/bin/ld: warning: -z relro ignored
  AR      built-in.a
  AR      vmlinux.a
  LD      vmlinux.o
/nix/store/f3k0rdhcd2cx57phx755c2xixgifw5m5-binutils-2.42/bin/ld: warning: -z relro ignored
  OBJCOPY modules.builtin.modinfo
  GEN     modules.builtin
  MODPOST Module.symvers
  CC      .vmlinux.export.o
  UPD     include/generated/utsversion.h
  CC      init/version-timestamp.o
  KSYMS   .tmp_vmlinux0.kallsyms.S
  AS      .tmp_vmlinux0.kallsyms.o
  LD      .tmp_vmlinux1
/nix/store/f3k0rdhcd2cx57phx755c2xixgifw5m5-binutils-2.42/bin/ld: warning: -z relro ignored
  BTF     .tmp_vmlinux1.btf.o
../scripts/link-vmlinux.sh: line 109: 185475 Killed                  LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1}
  NM      .tmp_vmlinux1.syms
  KSYMS   .tmp_vmlinux1.kallsyms.S
  AS      .tmp_vmlinux1.kallsyms.o
  LD      .tmp_vmlinux2
/nix/store/f3k0rdhcd2cx57phx755c2xixgifw5m5-binutils-2.42/bin/ld: warning: -z relro ignored
  NM      .tmp_vmlinux2.syms
  KSYMS   .tmp_vmlinux2.kallsyms.S
  AS      .tmp_vmlinux2.kallsyms.o
  LD      vmlinux
/nix/store/f3k0rdhcd2cx57phx755c2xixgifw5m5-binutils-2.42/bin/ld: warning: -z relro ignored
  BTFIDS  vmlinux
libbpf: failed to find '.BTF' ELF section in vmlinux
FAILED: load BTF from vmlinux: No data available
make[2]: *** [../scripts/Makefile.vmlinux:34: vmlinux] Error 255
make[2]: *** Deleting file 'vmlinux'
make[1]: *** [/build/linux-6.10.12/Makefile:1171: vmlinux] Error 2
make: *** [../Makefile:240: __sub-make] Error 2

I think my issue is different from what @misuzu shared. I don’t get any message in my log indicating that I am out of memory or storage.

Atemu · October 11, 2024, 9:53pm

How much exactly? My data on this is rather outdated and it could very well be the case that a modern kernel requires >40GB of build space these days.

RossComputerGuy · October 12, 2024, 1:58am

I kinda want to poke at this, I have an Ampere system which is very similar to the community builder so it should be easy to reproduce this. Currently on a plane right now but I can take a look in a couple hours.

misuzu · October 12, 2024, 6:54am

Even with -j1 the pahole wants more than 3689M (I’m surprised it’s even capable to do so, but’ it’s probably because the kernel is 64-bit, no way around the hard 2^32 bytes limit though, using pahole from git also doesn’t help):

In your log the pahole is killed (check dmesg right away), pretty sure the core issue is the same (pahole needs too much ram).

misuzu · October 12, 2024, 6:55am

Make sure to include this to your configuration before building anything, linux32 uname -m should print armv7l.

netogallo · October 12, 2024, 7:21am

I see @misuzu. I just checked and I have exactly 58GB of free HDD space. I think it is probably the memory issue as you described. I will rebuild and check the logs.

logustus · November 13, 2024, 2:05am

Following the advice here is what helped me past this issue in the end. Thanks!

I had to do a little digging to figure out how to set these flags, and this is what I came up with, for posterity:

{
  boot.kernelPatches = [
    {
      name = "disable-bpf";
      patch = null;
      extraConfig = ''
        DEBUG_INFO_BTF n
        CONFIG_DEBUG_INFO_BTF n
      '';
    }
  ];
}

I got one of the flags from @misuzu’s post above but the other (DEBUG_INFO_BTF) I found here: linux-kernel: Enable BTF · NixOS/nixpkgs@34150f8 · GitHub
There seems to be no harm in having them both, if one of them is superfluous, but I haven’t done the 2.5 hour build tests to see if it’s just one flag that achieves the fix vs. both.