So, I had another opportunity at work to use Yocto, and decided to give this a shot again, thanks to all the comments here, I managed to go a lot further than ever before.
Adding LOCALE_ARCHIVE
to $BB_ENV_EXTRAWHITE
does fix the bitbake locale error, although I’m still seeing warnings all over the place (more on that later).
The m4
configure error does come from the usage of two different glibc at the same time, as @Mic92. Here’s what I think happened:
While compiling host tools, Yocto tries to make the compiler use Yocto’s own provided glibc (rightfully so, since a lot of different Linux distributions have different libc implementations / versions / etc.). One implication of this is the addition of this flag to GCC arguments:
-Wl,--dynamic-linker=$YOCTO_DIR/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2
This effectively sets the ELF interpreter to the one from Yocto’s glibc. However, it does not exactly tell the interpreter where to find libc.so
, this still has to be done at runtime.
Where it goes wrong, is that NixOS’s GCC adds its own glibc lib path into the executable runpath, by appending these arguments to the linker:
-rpath /nix/store/$GLIBC_HASH-glibc-2.31-74/lib
-rpath /nix/store/$GCC_HASH-gcc-9.3.0-lib/lib
This tells the interpreter where to search for librairies, after trying the paths from $LD_LIBRARY_PATH
.
In effect, this means that when running the program, it will run through Yocto’s interpreter, but will link against NixOS glibc, which may differ in their versions (at the time of writing, glibc-2.32 for nixpkgs-unstable, and glibc-2.33 for Yocto/Poky master). As far as I know glibc doesn’t have any ABI stability, so this is recipe for disaster (usually segfaults, or interpreter error).
After countless hours trying to reverse-engineer NixOS’ GCC wrapper, disabling the addition of the rpath flags is done by adding two environment variables:
export NIX_DONT_SET_RPATH=1
export NIX_CC_WRAPPER_TARGET_HOST_$HOST_ARCH=1
# For example NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu=1
The second variable was harder to find, I assume this is because it is meant to be an internal nixpkgs variable used for building things with Nix.
However, exporting and adding these to $BB_ENV_EXTRAWHITE
does nothing… It seems that $BB_ENV_EXTRAWHITE
is not meant to pass variables to the build environment, but the bitbake environment (this might also explain the locale warnings). I’m not sure if there is a better way to this in Yocto, but my quick fix was simply to wrap gcc and g++, and set these variables directly in there.
I stumbled upon a configure error while compiling cmake-native, telling me the compiler doesn’t support C++11 features, and I assume this is because it interprets the recurrent locale warning as a compiler warning, and fails the check. At this point I was a bit tired, and did an even dirtier fix, by going into the CMake source, under build/tmp/work/x86_64-linux/cmake-native/3.19.5-r0/cmake-3.19.5/Source/Checks/cm_cxx_features.cmake
and replaced this line:
- if(check_output MATCHES "(^|[ :])[Ww][Aa][Rr][Nn][Ii][Nn][Gg]")
+ if(check_output MATCHES "(^|[ :])[Aa][Rr][Nn][Ii][Nn][Gg]")
Another issue I stumbled upon was that some C++ programs did not manage to find the libstdc++.so
shared library. This was because both the rpath was not set (rightfully), the $LD_LIBRARY_PATH
was emptied in the build environment, and if I remember correctly, this was a program that was using NixOS’ interpreter. Thankfully, this was fixed by using the latest unstable version of nixpkgs, thanks to this awesome PR: steam: fix proton versions with pressure-vessel by LuigiPiucco · Pull Request #114024 · NixOS/nixpkgs · GitHub
The reason for this is that now the Bubblewrap FHS user env generates an ld.so.conf
with its ld.so.cache
file, it is now able to find the library.
I would really like to know if someone knows a way to properly forward some environment variables to the build environment, so that we can upstream a fully-working version of this.
Anyway, here's my version, so far:
{ pkgs ? import <nixpkgs-unstable> {} }:
let
bashrcFile = pkgs.writeText "bashrc" ''
eval "$(${pkgs.starship}/bin/starship init bash)"
'';
gcc_multi_norpath = pkgs: (pkgs.gcc_multi.override (old: {
extraBuildCommands = ''
wrapProgram $out/bin/gcc \
--set NIX_DONT_SET_RPATH 1 \
--set NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu 1
wrapProgram $out/bin/g++ \
--set NIX_DONT_SET_RPATH 1 \
--set NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu 1
'';
})).overrideAttrs (old: {
nativeBuildInputs = [ pkgs.makeWrapper ];
});
in pkgs.buildFHSUserEnvBubblewrap {
name = "yocto-on-nixos";
targetPkgs = pkgs: with pkgs; [
bc file gnumake python3 unzip which patch perl util-linux rsync wget diffstat
(gcc_multi_norpath pkgs)
glibcLocales gdb valgrind
libglvnd mesa ncurses pkgconfig qt5.qtbase
bzip2 lzma
];
multiPkgs = pkgs: (with pkgs; []);
extraOutputsToInstall = [ "dev" ];
profile = ''
export hardeningDisable=all
export CC=gcc
export LD=ld
export STRIP=strip
export OBJCOPY=objcopy
export RANLIB=ranlib
export OBJDUMP=objdump
export AS=as
export AR=ar
export NM=nm
export CXX=g++
export SIZE=size
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export LOCALEARCHIVE=/usr/lib/locale/locale-archive
export BB_ENV_EXTRAWHITE="LOCALEARCHIVE LOCALE_ARCHIVE NIX_DONT_SET_RPATH NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu"
export NIX_DONT_SET_RPATH=1
'';
extraBuildCommands = ''
ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
'';
runScript = "bash --rcfile ${bashrcFile}";
}
Edit: Just managed to build core-image-base
with this, I’m so happy.