I am trying to build a specific commit from kernel 5.16.9 fetched from the stable kernel repo. Purpose being that I have to find a bug which first occurred on my machine starting with kernel 5.16.10. I use the following approach:
{ pkgs, ... }:
{
boot.kernelPackages =
let
linux_pkg = { fetchgit, buildLinux, ... } @ args:
buildLinux (args // rec {
version = "5.16.9";
modDirVersion = version;
src = fetchgit
{
url = "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/";
rev = "91fc147";
sha256 = "Dz+CpecLcVU/jA00gCnOenDhL8yYzMxM58uCXcTtzj0=";
};
kernelPatches = [ ];
extraMeta.branch = "5.16";
} // (args.argsOverride or { }));
linux_5-16-9 = pkgs.callPackage linux_pkg { };
in
pkgs.recurseIntoAttrs (pkgs.linuxPackagesFor linux_5-16-9);
}
However the build itself fails with the follow error that I cannot really wrap my head around:
error: builder for '/nix/store/bmmnyw3mimg3x01x9l8v396dn7vbqkj5-linux-5.16.9.drv' failed with exit code 2;
last 10 log lines:
> KSYMS .tmp_vmlinux.kallsyms2.S
> AS .tmp_vmlinux.kallsyms2.S
> LD vmlinux
> /nix/store/pnajkjw1fz03z1x27ski1qbx9n3hd4df-binutils-2.39/bin/ld: warning: .btf.vmlinux.bin.o: missing .note.GNU-stack section implies executable stack
> /nix/store/pnajkjw1fz03z1x27ski1qbx9n3hd4df-binutils-2.39/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
> /nix/store/pnajkjw1fz03z1x27ski1qbx9n3hd4df-binutils-2.39/bin/ld: warning: vmlinux has a LOAD segment with RWX permissions
> BTFIDS vmlinux
> FAILED: load BTF from vmlinux: Invalid argument
> make: *** [../Makefile:1161: vmlinux] Error 255
> make: *** Deleting file 'vmlinux'
For full logs, run 'nix log /nix/store/bmmnyw3mimg3x01x9l8v396dn7vbqkj5-linux-5.16.9.drv'.
How to fix this and get kernel 5.16.9 working? I’ve been googling a bit, but I am not really sure what to do. I found something about compiling the kernel with a different version of pahole or dwarves, but I am unsure how to implement that in nix.
I found a suggestion here:
https://lore.kernel.org/dwarves/Ywj6ZcqWsz8Au6qO@kernel.org/T/
The guy on the mailing list mentions this: “Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to v1.23 resolves the issue.”
So how would I go about downgrading dwarves to v1.23 for my kernel build?
I guess I could rephrase my question like so: How do I override the nativeBuildInputs for buildLinux as they are found here including pahole: https://github.com/NixOS/nixpkgs/blob/eb95aa0392d4a6f0eb3ac66eb0329397eafde4fb/pkgs/os-specific/linux/kernel/manual-config.nix
Because then I could just import an older nixpkgs for NixOS 22.05 instead of 22.11 which I am using right now and get pahole version 1.23 this way. Which supposedly would solve the build problem.
Where you successful at getting past this issue?
I’m trying to build custom kernels of 5.17 and 5.18 to trackdown an amdgpu bug, and I’m hitting this BTF error now 
That is interesting. I was also trying to track down an amdgpu bug 
To get around this stuff, I built myself an overlay that would provide me with the correct versions for pahole and libpbf. Which ended up looking more or less like this. (I have since removed it from my config again, so you’ll have to fiddle a bit with it):
overlay-custom-kernel = final: prev: {
pahole = prev.pahole.overrideAttrs (oldAttrs: {
version = "1.23";
src = prev.fetchgit {
url = "https://git.kernel.org/pub/scm/devel/pahole/pahole.git";
rev = "v1.23";
sha256 = "sha256-Dt3ZcUfjwdtTTv6qRFRgwK5GFWXdpN7fvb9KhpS1O94=";
};
buildInputs = [ prev.elfutils prev.zlib final.libbpf-0-7-0 ];
});
libbpf-0-7-0 = prev.libbpf.overrideAttrs (oldAttrs: {
version = "0.7.0";
src = prev.fetchFromGitHub {
owner = "libbpf";
repo = "libbpf";
rev = "v0.7.0";
sha256 = "sha256-NFVJ8JquWVzu+QoaaOMzhnu6/IqdP1FPhtJFidXA4L4=";
};
buildInputs = [ prev.libelf prev.zlib ];
});
};
And then I just decided to build the specific kernels one after the other like so. Obviously if you are going a git bisect you will have to iterate on the revision until you hopefully find your bug:
{ pkgs, ... }:
{
boot.kernelPackages =
let
linux_pkg = { fetchgit, buildLinux, ... } @ args:
buildLinux (args // rec {
version = "5.16.9";
modDirVersion = version;
src = fetchgit
{
url = "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/";
rev = "0398436e39b219cc26f44f9a301779e82f1684f7";
sha256 = "20BtNw6uuJDa4Vo8ti3JfMIHVvuQHha7zhLymKMmajo=";
};
kernelPatches = [ ];
extraMeta.branch = "5.16";
} // (args.argsOverride or { }));
linux_5-16-9 = pkgs.callPackage linux_pkg { };
in
pkgs.recurseIntoAttrs (pkgs.linuxPackagesFor linux_5-16-9);
}
I’m pretty new to nix flakes, I get the overlay is supposed to override the kernel’s buildInputs, but I don’t see the connection from how the kernelPackages definition is connecting to custom-kernel-overlay…
I tried to apply the overlay to my configuration.nix / flake.nix, but it’ doesn’t do anything (at least the way I tried it which is probably way wrong 
). What am I missing?
configuration.nix
{ pkgs, lib, linuxKernel, ... }: {
nixpkgs.overlays = [
(final: prev: {
overlay-custom-kernel = {
pahole = prev.pahole.overrideAttrs (oldAttrs: {
version = "1.23";
src = prev.fetchgit {
url = "https://git.kernel.org/pub/scm/devel/pahole/pahole.git";
rev = "v1.23";
sha256 = "sha256-Dt3ZcUfjwdtTTv6qRFRgwK5GFWXdpN7fvb9KhpS1O94=";
};
buildInputs = [ prev.elfutils prev.zlib final.libbpf-0-7-0 ];
});
libbpf_0 = prev.libbpf.overrideAttrs (oldAttrs: {
version = "0.7.0";
src = prev.fetchFromGitHub {
owner = "libbpf";
repo = "libbpf";
rev = "v0.7.0";
sha256 = "sha256-NFVJ8JquWVzu+QoaaOMzhnu6/IqdP1FPhtJFidXA4L4=";
};
buildInputs = [ prev.libelf prev.zlib ];
});
};
})
];
boot.supportedFilesystems = lib.mkForce [ "btrfs" "cifs" "f2fs" "jfs" "ntfs" "reiserfs" "vfat" "xfs" ]; # Disable support for ZFS
boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.linux_5_15.override {
argsOverride = rec {
version = "5.18";
# modDirVersion needs to be x.y.z, will automatically add .0 if needed
modDirVersion = lib.concatStringsSep "." (lib.take 3 (lib.splitVersion "${version}.0"));
# branchVersion needs to be x.y
extraMeta.branch = pkgs.versions.majorMinor version;
src = pkgs.fetchurl {
url = "mirror://kernel/linux/kernel/v5.x/linux-${version}.tar.xz";
sha256 = "1vjwhl4s8qxfg1aabn8xnpjza3qzrjcp5450h9qpjvl999lg3wsi";
};
# 5.18 related tweaks from https://github.com/alyssais/nixpkgs/blob/fa7ae8876f0b6bd05262a6be12ccd1fc551ed25d/pkgs/os-specific/linux/kernel/common-config.nix
extraConfig = ''
DEBUG_INFO y
DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT y
'';
};
});
nix.extraOptions = "experimental-features = nix-command flakes";
hardware.pulseaudio.enable = lib.mkForce false; # installation-cd-graphical-plasma5.nix says true; Jovian says false -> force false
services.xserver = {
desktopManager.plasma5.enable = lib.mkForce false; # KDE Plasma 5 must be disabled for Plasma 6
desktopManager.plasma6.enable = true;
# desktopManager.plasma6.enable = lib.mkForce true; # KDE Plasma 6
displayManager.sddm.enable = lib.mkForce false; # installation-cd-graphical-plasma5.nix sets displayManager.sddm.enable; Jovian says it can't be -> force false
# displayManager.defaultSession = "plasmawayland"; # Use Wayland as Graphics Manager
excludePackages = [ pkgs.xterm ]; # Exclude xterm, konsole is better
displayManager.sddm.theme = "breeze-dark"; # Dark Mode -- Doesn't work :\
};
environment.systemPackages = with pkgs; [
neofetch
git
byobu
tmux
bat
vscode
htop
wireguard-tools
brave
binutils
pciutils
lsof
jq
tio
unzip
curlie
rocmPackages.rocminfo
rocmPackages.rocm-smi
rocm-opencl-runtime
radeontop
libhdhomerun # hdhomerun_config command line
hdhomerun-config-gui # hdhomerun gui
];
time.timeZone = "America/Toronto";
};
flake.nix
{
inputs.nixpkgs.url = github:NixOS/nixpkgs/nixos-unstable;
# nix --extra-experimental-features flakes registry list
outputs = { self, nixpkgs, ... }: {
nixosConfigurations."isoimage" = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
./configuration.nix
/home/deftdawg/source/nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-graphical-plasma5.nix
];
};
};
}
Build command:
NIXPKGS_ALLOW_BROKEN=1 NIXPKGS_ALLOW_UNFREE=1 time nix --extra-experimental-features flakes build --impure .#nixosConfigurations.isoimage.config.system.build.isoImage
Got it!
Applied your overlay inline within the configuration.nix file my flake.nix calls and I was able to get past the kernel errors:
nixpkgs.overlays = [
(final: prev: {
pahole = prev.pahole.overrideAttrs (oldAttrs: {
version = "1.23";
src = prev.fetchgit {
url = "https://git.kernel.org/pub/scm/devel/pahole/pahole.git";
rev = "v1.23";
sha256 = "sha256-Dt3ZcUfjwdtTTv6qRFRgwK5GFWXdpN7fvb9KhpS1O94=";
};
buildInputs = [ prev.elfutils prev.zlib final.libbpf-0-7-0 ];
});
libbpf-0-7-0 = prev.libbpf.overrideAttrs (oldAttrs: {
version = "0.7.0";
src = prev.fetchFromGitHub {
owner = "libbpf";
repo = "libbpf";
rev = "v0.7.0";
sha256 = "sha256-NFVJ8JquWVzu+QoaaOMzhnu6/IqdP1FPhtJFidXA4L4=";
};
buildInputs = [ prev.libelf prev.zlib ];
});
})
];
There we go! I sadly didn’t have the time to follow up on your first reply.
What bug are you trying to find though?
1 Like
Basically it’s some kind of MST problem; any AMDGPU laptop connecting to multiple displays via USB-C dock/dongle could have it.
Connecting up with both screens turned on; X11/Wayland freeze until the dock is disconnected, 1 screen is powered off and the dock is reconnected; power on the second screen and it happens again.
In my particular case, the laptop is a Steam Deck and the Dock/Dongle is a Mokin 7-in-1 Dock with 1 DP and 1 HDMI.
Both displays work perfectly under Windows or Linux 5.15; but anything else best I can do without crashing is half bandwidth (half res or half refresh) on each screen.
I’m using NixOS ISOs to try to narrow down which kernel is causing the issue.
If I get it sorted, I’ll switch to NixOS + Jovian on the deck to fix it permanently.
Anyway, have a great weekend, thanks again for the pointers. 
1 Like