Build a yocto rootfs inside nix

This is the fix that stammw was talking about:

$ MACHINE=raspberrypi3-64 source setup-environment
$ export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE LOCALEARCHIVE LOCALE_ARCHIVE"
$ bitbake lmp-base-console-image

That works for me:

{ pkgs ? import (fetchTarball https://github.com/nixos/nixpkgs/archive/release-20.09.tar.gz) {} }:

(pkgs.buildFHSUserEnv {
  name = "yocto-env";

  # Packages Yocto is expecting on the host system by default
  targetPkgs = pkgs: (with pkgs; let
    sh = (pkgs.runCommand "sh" {} ''
      mkdir -p $out/bin
      cat > $out/bin/bash <<'EOF'
      #!${bashInteractive}/bin/bash
      export LOCALE_ARCHIVE=/usr/lib/locale/locale-archive
      export LOCALEARCHIVE=/usr/lib/locale/locale-archive
      exec -a /bin/bash ${bashInteractive}/bin/bash "$@"
      EOF
      chmod +x $out/bin/bash
      ln -s $out/bin/bash $out/bin/sh
    '');
    in [
    which gcc glibc glibcLocales shadow gnumake python27 gawk wget
    gitFull diffstat diffutils unzip texinfo bzip2 gzip perl patch chrpath file
    cpio utillinux nettools iproute procps openssh xterm SDL findutils
    socat gnutar ccache cmake vim binutils gitRepo
    (pkgs.runCommand "python3" {} ''
      mkdir -p $out/bin
      cat > $out/bin/python3 <<'EOF'
      #!${runtimeShell}
      export LOCALE_ARCHIVE=/usr/lib/locale/locale-archive
      export LOCALEARCHIVE=/usr/lib/locale/locale-archive
      exec -a /usr/bin/python3 ${python3}/bin/python "$@"
      EOF
      chmod +x $out/bin/python3
    '')
    (hiPrio sh)
  ]);

  # Headers are required to build
  extraOutputsToInstall = [ "dev" ];

  # Force install locale from "glibcLocales" since there are collisions
  extraBuildCommands = ''
    ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
  '';

  profile = ''
    export hardeningDisable=all
    export CC=gcc
    export LD=ld
    export EDITOR=vim
    export STRIP=strip
    export OBJCOPY=objcopy
    export RANLIB=ranlib
    export OBJDUMP=objdump
    export AS=as
    export AR=ar
    export NM=nm
    export CXX=g++
    export SIZE=size
    export LANG="en_US.UTF-8"
    export LC_ALL="en_US.UTF-8"
    export SHELL=/bin/bash
  '';

  multiPkgs = pkgs: (with pkgs; []);
  runScript = "bash";
}).env

Update this version still had some issues. I think the proper fix would be to include en_US.utf-8 in our glibc version by default (can be done via overrideAttrs) and rebuild python/bash/perl against that.

So, I had another opportunity at work to use Yocto, and decided to give this a shot again, thanks to all the comments here, I managed to go a lot further than ever before.

Adding LOCALE_ARCHIVE to $BB_ENV_EXTRAWHITE does fix the bitbake locale error, although I’m still seeing warnings all over the place (more on that later).

The m4 configure error does come from the usage of two different glibc at the same time, as @Mic92. Here’s what I think happened:

While compiling host tools, Yocto tries to make the compiler use Yocto’s own provided glibc (rightfully so, since a lot of different Linux distributions have different libc implementations / versions / etc.). One implication of this is the addition of this flag to GCC arguments:

-Wl,--dynamic-linker=$YOCTO_DIR/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2

This effectively sets the ELF interpreter to the one from Yocto’s glibc. However, it does not exactly tell the interpreter where to find libc.so, this still has to be done at runtime.

Where it goes wrong, is that NixOS’s GCC adds its own glibc lib path into the executable runpath, by appending these arguments to the linker:

-rpath /nix/store/$GLIBC_HASH-glibc-2.31-74/lib
-rpath /nix/store/$GCC_HASH-gcc-9.3.0-lib/lib

This tells the interpreter where to search for librairies, after trying the paths from $LD_LIBRARY_PATH.

In effect, this means that when running the program, it will run through Yocto’s interpreter, but will link against NixOS glibc, which may differ in their versions (at the time of writing, glibc-2.32 for nixpkgs-unstable, and glibc-2.33 for Yocto/Poky master). As far as I know glibc doesn’t have any ABI stability, so this is recipe for disaster (usually segfaults, or interpreter error).

After countless hours trying to reverse-engineer NixOS’ GCC wrapper, disabling the addition of the rpath flags is done by adding two environment variables:

export NIX_DONT_SET_RPATH=1
export NIX_CC_WRAPPER_TARGET_HOST_$HOST_ARCH=1
# For example NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu=1

The second variable was harder to find, I assume this is because it is meant to be an internal nixpkgs variable used for building things with Nix.

However, exporting and adding these to $BB_ENV_EXTRAWHITE does nothing… It seems that $BB_ENV_EXTRAWHITE is not meant to pass variables to the build environment, but the bitbake environment (this might also explain the locale warnings). I’m not sure if there is a better way to this in Yocto, but my quick fix was simply to wrap gcc and g++, and set these variables directly in there.

I stumbled upon a configure error while compiling cmake-native, telling me the compiler doesn’t support C++11 features, and I assume this is because it interprets the recurrent locale warning as a compiler warning, and fails the check. At this point I was a bit tired, and did an even dirtier fix, by going into the CMake source, under build/tmp/work/x86_64-linux/cmake-native/3.19.5-r0/cmake-3.19.5/Source/Checks/cm_cxx_features.cmake and replaced this line:

- if(check_output MATCHES "(^|[ :])[Ww][Aa][Rr][Nn][Ii][Nn][Gg]")
+ if(check_output MATCHES "(^|[ :])[Aa][Rr][Nn][Ii][Nn][Gg]")

Another issue I stumbled upon was that some C++ programs did not manage to find the libstdc++.so shared library. This was because both the rpath was not set (rightfully), the $LD_LIBRARY_PATH was emptied in the build environment, and if I remember correctly, this was a program that was using NixOS’ interpreter. Thankfully, this was fixed by using the latest unstable version of nixpkgs, thanks to this awesome PR: steam: fix proton versions with pressure-vessel by LuigiPiucco · Pull Request #114024 · NixOS/nixpkgs · GitHub

The reason for this is that now the Bubblewrap FHS user env generates an ld.so.conf with its ld.so.cache file, it is now able to find the library.

I would really like to know if someone knows a way to properly forward some environment variables to the build environment, so that we can upstream a fully-working version of this.

Anyway, here's my version, so far:
{ pkgs ? import <nixpkgs-unstable> {} }:

let

  bashrcFile = pkgs.writeText "bashrc" ''
    eval "$(${pkgs.starship}/bin/starship init bash)"
  '';

  gcc_multi_norpath = pkgs: (pkgs.gcc_multi.override (old: {
    extraBuildCommands = ''
      wrapProgram $out/bin/gcc \
        --set NIX_DONT_SET_RPATH 1 \
        --set NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu 1

      wrapProgram $out/bin/g++ \
        --set NIX_DONT_SET_RPATH 1 \
        --set NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu 1
    '';
  })).overrideAttrs (old: {
      nativeBuildInputs = [ pkgs.makeWrapper ];
  });

in pkgs.buildFHSUserEnvBubblewrap {
  name = "yocto-on-nixos";

  targetPkgs = pkgs: with pkgs; [
    bc file gnumake python3 unzip which patch perl util-linux rsync wget diffstat

    (gcc_multi_norpath pkgs)
    glibcLocales gdb valgrind

    libglvnd mesa ncurses pkgconfig qt5.qtbase
    bzip2 lzma
  ];

  multiPkgs = pkgs: (with pkgs; []);

  extraOutputsToInstall = [ "dev" ];

  profile = ''
    export hardeningDisable=all

    export CC=gcc
    export LD=ld
    export STRIP=strip
    export OBJCOPY=objcopy
    export RANLIB=ranlib
    export OBJDUMP=objdump
    export AS=as
    export AR=ar
    export NM=nm
    export CXX=g++
    export SIZE=size

    export LANG="en_US.UTF-8"
    export LC_ALL="en_US.UTF-8"

    export LOCALEARCHIVE=/usr/lib/locale/locale-archive
    export BB_ENV_EXTRAWHITE="LOCALEARCHIVE LOCALE_ARCHIVE NIX_DONT_SET_RPATH NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu"

    export NIX_DONT_SET_RPATH=1
  '';

  extraBuildCommands = ''
    ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
  '';

  runScript = "bash --rcfile ${bashrcFile}";
}

Edit: Just managed to build core-image-base with this, I’m so happy.

2 Likes

You might be able to use an unwrapped GCC but than you would need additional variables.
Devshell wants to go in this direction and provide unwrapped compilers. An alternative to a wrapped CC might be to define SYSROOT as posted here: [c/c++] Investigate $SYSROOT · Issue #103 · numtide/devshell · GitHub

Feel free to add this to nix-environments

Also if you have envfs you might not need fhsuserenv, in case it becomes annoying.

Based upon this environment you have made, I added a few dependencies, overwrote the BB_ENV_EXTRAWHITE after sourcing setup-environment, and set C.UTF-8 for LC-ALL + LANG instead of en_US.UTF-8. Additionally, I didn’t have a channel named nixpkgs-unstable added to my system. I think the correct modern approach to this is to use a Flake devShell, since this would allow us to define inputs, which I’m going to be trying to port this to.

The result is the following diff:

diff --git a/forum-post.nix b/yocto.nix
index e425e55..64e756c 100644
--- a/forum-post.nix
+++ b/yocto.nix
@@ -1,4 +1,4 @@
-{ pkgs ? import <nixpkgs-unstable> {} }:
+{ pkgs ? import <nixpkgs> {} }:
 
 let
 
@@ -26,6 +26,11 @@ in pkgs.buildFHSUserEnvBubblewrap {
   targetPkgs = pkgs: with pkgs; [
     bc file gnumake python3 unzip which patch perl util-linux rsync wget diffstat
 
+    gcc glibc glibcLocales shadow  python27 gawk gitFull gitRepo diffutils
+    texinfo bzip2 gzip chrpath bash cpio utillinux nettools iproute procps
+    openssh xterm SDL findutils socat gnutar ccache cmake vim binutils bash
+    bashInteractive rpcsvc-proto
+
     (gcc_multi_norpath pkgs)
     glibcLocales gdb valgrind
 
@@ -52,8 +57,8 @@ in pkgs.buildFHSUserEnvBubblewrap {
     export CXX=g++
     export SIZE=size
 
-    export LANG="en_US.UTF-8"
-    export LC_ALL="en_US.UTF-8"
+    export LANG="C.UTF-8"
+    export LC_ALL="C.UTF-8"
 
     export LOCALEARCHIVE=/usr/lib/locale/locale-archive
     export BB_ENV_EXTRAWHITE="LOCALEARCHIVE LOCALE_ARCHIVE NIX_DONT_SET_RPATH NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu"

I’m now able to build my company’s Yocto based distribution after I put this into a file named yocto.nix:

nix-build yocto.nix
./result/bin/yocto-on-nixos
mkdir lmp && cd lmp
repo init -u https://github.com/foundriesio/lmp-manifest -b refs/tags/79
repo sync
MACHINE=raspberrypi3-64 source setup-environment
export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE LOCALEARCHIVE LOCALE_ARCHIVE"
bitbake lmp-base-console-image

This is incredibly impressive. I am over the moon. :cowboy_hat_face:

2 Likes

This is my variant for building a poky core-image-minimal as a Nix derivation:

Code
let
  pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/7aa78642c5f2be070c92b6f2d8762ef1390234cb.tar.gz") {};

  env = pkgs.buildFHSUserEnvBubblewrap {
    name = "yocto-buildenv";
    targetPkgs = pkgs: with pkgs; [
      python3 gcc nettools util-linux binutils rpcsvc-proto chrpath cpio diffstat file getopt git perl wget which
    ];
    multiPkgs = pkgs: (with pkgs; []);

    extraOutputsToInstall = [ "dev" ];
    extraBuildCommands = ''
      ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
    '';

    runScript = "bash";
  };

  fixLocale = ''
    export LOCALE_ARCHIVE=/usr/lib/locale/locale-archive
    export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE LOCALE_ARCHIVE"
  '';

  deps = pkgs.stdenv.mkDerivation {
    name = "yocto-deps";

    src = fetchGit {
      url = "https://git.yoctoproject.org/git/poky";
      rev = "e35dc77f77f9ab3f3a96819201bc84fb14e205a8";
      ref = "refs/heads/hardknott";
    };

    buildPhase = ''
      ${env}/bin/yocto-buildenv -c '
      ${fixLocale}
      source oe-init-build-env
      bitbake core-image-minimal --runall=fetch
      '
    '';

    installPhase = ''
      cp -r --reflink=auto build/downloads $out
    '';

    dontFixup = true;

    outputHash = "0r273xj2ac3rid0qwgmy52zlazjbdhyp35l8d74haa712cz876yb";
    outputHashMode = "recursive";
    outputHashAlgo = "sha256";
  };

in
  pkgs.stdenv.mkDerivation {
    name = "yocto";

    src = fetchGit {
      url = "https://git.yoctoproject.org/git/poky";
      rev = "e35dc77f77f9ab3f3a96819201bc84fb14e205a8";
      ref = "refs/heads/hardknott";
    };

    postPatch = ''
      cat << EOF >> meta/recipes-extended/shadow/shadow_*.bb
      do_configure_prepend () {
        sed "s/4755/0755/g" -i \''${S}/src/Makefile.am
      }
      EOF
      cat << EOF >> meta/recipes-core/util-linux/util-linux_*.bb
      do_configure_prepend () {
        sed "s/4755/0755/g" -i \''${S}/sys-utils/Makemodule.am
      }
      EOF
      cat << EOF >> meta/classes/base.bbclass
      export LOCALE_ARCHIVE = "/usr/lib/locale/locale-archive"
      export hardeningDisable = "all"
      export NIX_DONT_SET_RPATH = "1"
      export NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu = "1"
      EOF
    '';

    configurePhase = ''
      mkdir build
      cp -r --reflink=auto ${deps} build/downloads
      chmod u+w -R build/downloads
    '';

    buildPhase = ''
      ${env}/bin/yocto-buildenv -c '
      ${fixLocale}
      export BB_NO_NETWORK=1
      source oe-init-build-env
      bitbake core-image-minimal
      '
    '';

    installPhase = ''
      cp -r build/tmp/deploy $out
    '';

    passthru = {
      inherit deps env;
    };

    dontFixup = true;
  }
1 Like

The shell file in my last post had some issues still so i removed it, however

I looked in to this a bit, its possible to forward variables to the bitbake build environment by exporting variables from a config file. e.g in local.conf:

NIX_DONT_SET_RPATH_class-native = "1"
NIX_CC_WRAPPER_TARGET_HOST_unknown_linux_gnu_class-native  = "1"

export NIX_DONT_SET_RPATH
export NIX_CC_WRAPPER_TARGET_HOST_unknown_linux_gnu

This combined with setting the variable BBPOSTCONF to make bitbake read a extra config led me to the following file. There is still some collisions warnings although and now the variables are passed to all packeages, if possible that should only be -native packages since those are the ones that use the host compiler.

{ pkgs ? import <nixpkgs> { } }:

let

  fhs = pkgs.buildFHSUserEnvBubblewrap {
    name = "yocto-fhs";
    targetPkgs = pkgs: (with pkgs;

      [

        bc
        binutils
        bzip2
        chrpath
        cpio
        diffstat
        expect
        file
        gcc
        gdb
        git
        gnumake
        hostname
        kconfig-frontends
        lzma
        ncurses
        patch
        perl
        python3
        rpcsvc-proto
        unzip
        util-linux
        wget
        which

      ]);

    # Forces 64bit only environment
    multiPkgs = null;

    extraOutputsToInstall = [ "dev" ];

    profile =
      let

        wrapperEnvar = "NIX_CC_WRAPPER_TARGET_HOST_${pkgs.stdenv.cc.suffixSalt}";

        # TODO limit export to native pkgs?
        nixconf = pkgs.writeText "nixvars.conf" ''
          # This exports the variables
          # to actual build environments

          # From BB_ENV_EXTRAWHITE
          export LOCALE_ARCHIVE 
          export ${wrapperEnvar}

          export NIX_DONT_SET_RPATH = "1"

          # Exclude these when hashing
          # the packages in yocto
          BB_HASHBASE_WHITELIST_append = " LOCALE_ARCHIVE \
                                           NIX_DONT_SET_RPATH \ 
                                           ${wrapperEnvar} "
        '';

      in
      ''
        # These are set by buildFHSUserEnvBubblewrap
        export BB_ENV_EXTRAWHITE="LOCALE_ARCHIVE \
                                  ${wrapperEnvar} \
                                  $BB_ENV_EXTRAWHITE"

        # source the config for bibake
        # equal to --postread
        export BBPOSTCONF="${nixconf}"
      '';

  };

in
fhs.env

I also noted that this issue prevents any sudo use which is need for qemu. It would be nice to use a mkShell but there is a lot of /bin/bash assumptions in yocto it seems.

It turns out bintools also is wrapped with a reference to glibc from nixpkgs,
setting:

export NIX_BINTOOLS_WRAPPER_TARGET_HOST_${pkgs.stdenv.cc.suffixSalt} = "1"

and adding it to the hash whitelist, seems to fix this.

yocto was now also added to nix-environments: add environemnt for Yocto by Kaisrlik · Pull Request #12 · nix-community/nix-environments · GitHub

4 Likes