Kernel cross-compilation

[pkral78]

Hello there. I’m in trouble with cross compilation of Linux kernel to armv7l platform on my NixOS machine.This is how my shell.nix look-like:

let
pkgs = import <nixpkgs> {};
targetPkgs = import <nixpkgs> {
  crossSystem = (import <nixpkgs/lib>).systems.examples.armv7l-hf-multiplatform;
};
in
    targetPkgs.mkShell rec {
      nativeBuildInputs = with pkgs; [ 
        buildPackages.stdenv.cc
        bison 
        flex
        lzop
        pkgconfig
        ncurses
        openssl
       ];
      buildInputs = with targetPkgs; [ zlib ]; 
      shellHook = ''
       export CC=armv7l-unknown-linux-gnueabihf-
       export PKG_CONFIG_PATH="${pkgs.ncurses.dev}/lib/pkgconfig"
      '';
    }

Kernel compilation requires not just to compile target kernel but some tools running on host during target build.

First problem I’ve “solved” with shellHook was that pkg-config does not see ncurses.
Second one was that even when pkg-config sees ncurses *.pc files, they miss libdir and I have to patch kernel sources with something like:

-		echo libs=\"$(pkg-config --libs $PKG)\"
+		echo libs=\"$(pkg-config --libs $PKG) -L$(pkg-config --variable=libdir $PKG)\"

This may be problem with ncurses nix derivation itself or probably problem related to my first hack.

Third problem may be related to the first one and fact that shell derivation I use is somewhat crippled, because kernel build fails due missing openssl libraries during HOSTCC compilation.

So maybe I could generalize the question how to properly set $NIX_CFLAGS_COMPILE, $NIX_LDFLAGS variables for host compiler (buildPackages.stdenv.cc) and how for target compiler.

I’ve also tried to use callPackage splicing, because I’m still not sure when and what is evaluated in shell.nix but this did not change anything.

Thanks for anything that could enlightment the situation, I’m using nixos-unstable.

[Ericson2314]

This should at least get you closer:

@pkral78

let
pkgs = import <nixpkgs> {
  crossSystem = (import <nixpkgs/lib>).systems.examples.armv7l-hf-multiplatform;
};
in
    pkgs.mkShell rec {
      depsBuildBuild = [ buildPackages.stdenv.cc ];
      nativeBuildInputs = with pkgs.buildPackages; [
        bison 
        flex
        lzop
        pkgconfig
        ncurses
        openssl
       ];
      buildInputs = with pkgs; [ zlib ]; 
    }

[pkral78]

Thank you very much, @Ericson2314. I’ve end up with this solution, which works and I’m able to build kernel:

let
hostPkgs = import <nixpkgs> {};
pkgs = import <nixpkgs> {
  crossSystem = (import <nixpkgs/lib>).systems.examples.armv7l-hf-multiplatform;
};
in
    pkgs.mkShell rec {
      depsBuildBuild = [hostPkgs.buildPackages.stdenv.cc ];
      nativeBuildInputs = with hostPkgs.buildPackages; [
        bison 
        flex
        lzop
        pkgconfig
        ncurses
        openssl
       ];
      buildInputs = with pkgs; [ zlib ]; 
      shellHook = ''
       export CC=armv7l-unknown-linux-gnueabihf-
       export PKG_CONFIG_PATH="${hostPkgs.ncurses.dev}/lib/pkgconfig"
      '';
    }

This sets $CFLAGS and $LDFLAGS in shell environment which seems to be sufficient.

However the shellHook is still neccessary. The CC variable is related to build script, but what still bothers me is fact that without overriding $PKG_CONFIG_PATH pkg-config searchs in buildInputs packages (i.e. zlib) and not nativeBuildInputs. I suppose that this is expected behaviour, so to have $PKG_CONFIG_PATH pointing to nativeBuildsInput packages, I have to override something in pkg-config.

[Ericson2314]

1. Get rid of hostPkgs like I did, it will just result in things not working as expected. (It would be called buildPkgs not hostPlgs anyways because we use the autoconf parlance.)
2. buildPackages.pkgconfig will give you a prefixed package config that picks up host platform libs. Use depsBuildBuild = [ ... pkgs.pkgsBuildBuild.pkg-config ] to get a package config that picks up native libs.
3. CC should be set by the setup hook of that C compiler. If fit isn’t, that’s a sign other things are going wrong.

[pkral78]

@Ericson2314 you made my day, this explains lot of things, thank you!

at 1.) understood thanks
at 2.) that was the missing piece, pkg-config now sees all build dependencies, the only quirk that remains is missing path to libraries when pkg-config -libs ncurses is executed as it only prints -lncursesw and not -lncursesw -L/nix/store/..../lib but this is related to ncurses derivation and I’ll open issue for this
at 3.) CC was/is set correctly however build script that I’m using need it without compiler suffix

[Ericson2314]

Glad things are working out for you!

@pkral78

CC was/is set correctly however build script that I’m using need it without compiler suffix

OK, but you wrote

export CC=armv7l-unknown-linux-gnueabihf-

which seems to be the opposite? Just the prefix?

[pkral78]

Yes, I expressed myself a little vaguely:

I taken build scripts from https://www.digikey.com/eewiki/display/linuxonarm/ODYSSEY-STM32MP157C#ODYSSEY-STM32MP157C-ARMCrossCompiler:GCC which, in my opinion, incorrectly uses CC like CROSS_COMPILE (value containing only target-triplet prefix).

This is correct definition:

CC=$(CROSS_COMPILE)gcc

and mkShell populates it correctly as armv7l-unknown-linux-gnueabihf-gcc.
But to be able to run the script without modifications I have to trim gcc suffix.

I just want to clarify this, for readers of this thread.

[Ericson2314]

Ah ok. Perfect, our opinions match :).

Also do note that Nixpkgs should already be able to cross compile Linux kernels. If that wasn’t working for you, it could be a regression.