This is the fix that stammw was talking about:
$ MACHINE=raspberrypi3-64 source setup-environment
$ export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE LOCALEARCHIVE LOCALE_ARCHIVE"
$ bitbake lmp-base-console-image
That works for me:
{ pkgs ? import (fetchTarball https://github.com/nixos/nixpkgs/archive/release-20.09.tar.gz) {} }:
(pkgs.buildFHSUserEnv {
name = "yocto-env";
# Packages Yocto is expecting on the host system by default
targetPkgs = pkgs: (with pkgs; let
sh = (pkgs.runCommand "sh" {} ''
mkdir -p $out/bin
cat > $out/bin/bash <<'EOF'
#!${bashInteractive}/bin/bash
export LOCALE_ARCHIVE=/usr/lib/locale/locale-archive
export LOCALEARCHIVE=/usr/lib/locale/locale-archive
exec -a /bin/bash ${bashInteractive}/bin/bash "$@"
EOF
chmod +x $out/bin/bash
ln -s $out/bin/bash $out/bin/sh
'');
in [
which gcc glibc glibcLocales shadow gnumake python27 gawk wget
gitFull diffstat diffutils unzip texinfo bzip2 gzip perl patch chrpath file
cpio utillinux nettools iproute procps openssh xterm SDL findutils
socat gnutar ccache cmake vim binutils gitRepo
(pkgs.runCommand "python3" {} ''
mkdir -p $out/bin
cat > $out/bin/python3 <<'EOF'
#!${runtimeShell}
export LOCALE_ARCHIVE=/usr/lib/locale/locale-archive
export LOCALEARCHIVE=/usr/lib/locale/locale-archive
exec -a /usr/bin/python3 ${python3}/bin/python "$@"
EOF
chmod +x $out/bin/python3
'')
(hiPrio sh)
]);
# Headers are required to build
extraOutputsToInstall = [ "dev" ];
# Force install locale from "glibcLocales" since there are collisions
extraBuildCommands = ''
ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
'';
profile = ''
export hardeningDisable=all
export CC=gcc
export LD=ld
export EDITOR=vim
export STRIP=strip
export OBJCOPY=objcopy
export RANLIB=ranlib
export OBJDUMP=objdump
export AS=as
export AR=ar
export NM=nm
export CXX=g++
export SIZE=size
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export SHELL=/bin/bash
'';
multiPkgs = pkgs: (with pkgs; []);
runScript = "bash";
}).env
Update this version still had some issues. I think the proper fix would be to include en_US.utf-8 in our glibc version by default (can be done via overrideAttrs) and rebuild python/bash/perl against that.
So, I had another opportunity at work to use Yocto, and decided to give this a shot again, thanks to all the comments here, I managed to go a lot further than ever before.
Adding LOCALE_ARCHIVE to $BB_ENV_EXTRAWHITE does fix the bitbake locale error, although I’m still seeing warnings all over the place (more on that later).
The m4 configure error does come from the usage of two different glibc at the same time, as @Mic92. Here’s what I think happened:
While compiling host tools, Yocto tries to make the compiler use Yocto’s own provided glibc (rightfully so, since a lot of different Linux distributions have different libc implementations / versions / etc.). One implication of this is the addition of this flag to GCC arguments:
-Wl,--dynamic-linker=$YOCTO_DIR/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2
This effectively sets the ELF interpreter to the one from Yocto’s glibc. However, it does not exactly tell the interpreter where to find libc.so, this still has to be done at runtime.
Where it goes wrong, is that NixOS’s GCC adds its own glibc lib path into the executable runpath, by appending these arguments to the linker:
-rpath /nix/store/$GLIBC_HASH-glibc-2.31-74/lib
-rpath /nix/store/$GCC_HASH-gcc-9.3.0-lib/lib
This tells the interpreter where to search for librairies, after trying the paths from $LD_LIBRARY_PATH.
In effect, this means that when running the program, it will run through Yocto’s interpreter, but will link against NixOS glibc, which may differ in their versions (at the time of writing, glibc-2.32 for nixpkgs-unstable, and glibc-2.33 for Yocto/Poky master). As far as I know glibc doesn’t have any ABI stability, so this is recipe for disaster (usually segfaults, or interpreter error).
After countless hours trying to reverse-engineer NixOS’ GCC wrapper, disabling the addition of the rpath flags is done by adding two environment variables:
export NIX_DONT_SET_RPATH=1
export NIX_CC_WRAPPER_TARGET_HOST_$HOST_ARCH=1
# For example NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu=1
The second variable was harder to find, I assume this is because it is meant to be an internal nixpkgs variable used for building things with Nix.
However, exporting and adding these to $BB_ENV_EXTRAWHITE does nothing… It seems that $BB_ENV_EXTRAWHITE is not meant to pass variables to the build environment, but the bitbake environment (this might also explain the locale warnings). I’m not sure if there is a better way to this in Yocto, but my quick fix was simply to wrap gcc and g++, and set these variables directly in there.
I stumbled upon a configure error while compiling cmake-native, telling me the compiler doesn’t support C++11 features, and I assume this is because it interprets the recurrent locale warning as a compiler warning, and fails the check. At this point I was a bit tired, and did an even dirtier fix, by going into the CMake source, under build/tmp/work/x86_64-linux/cmake-native/3.19.5-r0/cmake-3.19.5/Source/Checks/cm_cxx_features.cmake and replaced this line:
- if(check_output MATCHES "(^|[ :])[Ww][Aa][Rr][Nn][Ii][Nn][Gg]")
+ if(check_output MATCHES "(^|[ :])[Aa][Rr][Nn][Ii][Nn][Gg]")
Another issue I stumbled upon was that some C++ programs did not manage to find the libstdc++.so shared library. This was because both the rpath was not set (rightfully), the $LD_LIBRARY_PATH was emptied in the build environment, and if I remember correctly, this was a program that was using NixOS’ interpreter. Thankfully, this was fixed by using the latest unstable version of nixpkgs, thanks to this awesome PR: steam: fix proton versions with pressure-vessel by LuigiPiucco · Pull Request #114024 · NixOS/nixpkgs · GitHub
The reason for this is that now the Bubblewrap FHS user env generates an ld.so.conf with its ld.so.cache file, it is now able to find the library.
I would really like to know if someone knows a way to properly forward some environment variables to the build environment, so that we can upstream a fully-working version of this.
Anyway, here's my version, so far:
{ pkgs ? import <nixpkgs-unstable> {} }:
let
bashrcFile = pkgs.writeText "bashrc" ''
eval "$(${pkgs.starship}/bin/starship init bash)"
'';
gcc_multi_norpath = pkgs: (pkgs.gcc_multi.override (old: {
extraBuildCommands = ''
wrapProgram $out/bin/gcc \
--set NIX_DONT_SET_RPATH 1 \
--set NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu 1
wrapProgram $out/bin/g++ \
--set NIX_DONT_SET_RPATH 1 \
--set NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu 1
'';
})).overrideAttrs (old: {
nativeBuildInputs = [ pkgs.makeWrapper ];
});
in pkgs.buildFHSUserEnvBubblewrap {
name = "yocto-on-nixos";
targetPkgs = pkgs: with pkgs; [
bc file gnumake python3 unzip which patch perl util-linux rsync wget diffstat
(gcc_multi_norpath pkgs)
glibcLocales gdb valgrind
libglvnd mesa ncurses pkgconfig qt5.qtbase
bzip2 lzma
];
multiPkgs = pkgs: (with pkgs; []);
extraOutputsToInstall = [ "dev" ];
profile = ''
export hardeningDisable=all
export CC=gcc
export LD=ld
export STRIP=strip
export OBJCOPY=objcopy
export RANLIB=ranlib
export OBJDUMP=objdump
export AS=as
export AR=ar
export NM=nm
export CXX=g++
export SIZE=size
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export LOCALEARCHIVE=/usr/lib/locale/locale-archive
export BB_ENV_EXTRAWHITE="LOCALEARCHIVE LOCALE_ARCHIVE NIX_DONT_SET_RPATH NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu"
export NIX_DONT_SET_RPATH=1
'';
extraBuildCommands = ''
ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
'';
runScript = "bash --rcfile ${bashrcFile}";
}
Edit: Just managed to build core-image-base with this, I’m so happy.
2 Likes
You might be able to use an unwrapped GCC but than you would need additional variables.
Devshell wants to go in this direction and provide unwrapped compilers. An alternative to a wrapped CC might be to define SYSROOT as posted here: [c/c++] Investigate $SYSROOT · Issue #103 · numtide/devshell · GitHub
Feel free to add this to nix-environments
Also if you have envfs you might not need fhsuserenv, in case it becomes annoying.
Based upon this environment you have made, I added a few dependencies, overwrote the BB_ENV_EXTRAWHITE after sourcing setup-environment, and set C.UTF-8 for LC-ALL + LANG instead of en_US.UTF-8. Additionally, I didn’t have a channel named nixpkgs-unstable added to my system. I think the correct modern approach to this is to use a Flake devShell, since this would allow us to define inputs, which I’m going to be trying to port this to.
The result is the following diff:
diff --git a/forum-post.nix b/yocto.nix
index e425e55..64e756c 100644
--- a/forum-post.nix
+++ b/yocto.nix
@@ -1,4 +1,4 @@
-{ pkgs ? import <nixpkgs-unstable> {} }:
+{ pkgs ? import <nixpkgs> {} }:
let
@@ -26,6 +26,11 @@ in pkgs.buildFHSUserEnvBubblewrap {
targetPkgs = pkgs: with pkgs; [
bc file gnumake python3 unzip which patch perl util-linux rsync wget diffstat
+ gcc glibc glibcLocales shadow python27 gawk gitFull gitRepo diffutils
+ texinfo bzip2 gzip chrpath bash cpio utillinux nettools iproute procps
+ openssh xterm SDL findutils socat gnutar ccache cmake vim binutils bash
+ bashInteractive rpcsvc-proto
+
(gcc_multi_norpath pkgs)
glibcLocales gdb valgrind
@@ -52,8 +57,8 @@ in pkgs.buildFHSUserEnvBubblewrap {
export CXX=g++
export SIZE=size
- export LANG="en_US.UTF-8"
- export LC_ALL="en_US.UTF-8"
+ export LANG="C.UTF-8"
+ export LC_ALL="C.UTF-8"
export LOCALEARCHIVE=/usr/lib/locale/locale-archive
export BB_ENV_EXTRAWHITE="LOCALEARCHIVE LOCALE_ARCHIVE NIX_DONT_SET_RPATH NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu"
I’m now able to build my company’s Yocto based distribution after I put this into a file named yocto.nix:
nix-build yocto.nix
./result/bin/yocto-on-nixos
mkdir lmp && cd lmp
repo init -u https://github.com/foundriesio/lmp-manifest -b refs/tags/79
repo sync
MACHINE=raspberrypi3-64 source setup-environment
export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE LOCALEARCHIVE LOCALE_ARCHIVE"
bitbake lmp-base-console-image
This is incredibly impressive. I am over the moon. 
2 Likes
This is my variant for building a poky core-image-minimal as a Nix derivation:
Code
let
pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/7aa78642c5f2be070c92b6f2d8762ef1390234cb.tar.gz") {};
env = pkgs.buildFHSUserEnvBubblewrap {
name = "yocto-buildenv";
targetPkgs = pkgs: with pkgs; [
python3 gcc nettools util-linux binutils rpcsvc-proto chrpath cpio diffstat file getopt git perl wget which
];
multiPkgs = pkgs: (with pkgs; []);
extraOutputsToInstall = [ "dev" ];
extraBuildCommands = ''
ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
'';
runScript = "bash";
};
fixLocale = ''
export LOCALE_ARCHIVE=/usr/lib/locale/locale-archive
export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE LOCALE_ARCHIVE"
'';
deps = pkgs.stdenv.mkDerivation {
name = "yocto-deps";
src = fetchGit {
url = "https://git.yoctoproject.org/git/poky";
rev = "e35dc77f77f9ab3f3a96819201bc84fb14e205a8";
ref = "refs/heads/hardknott";
};
buildPhase = ''
${env}/bin/yocto-buildenv -c '
${fixLocale}
source oe-init-build-env
bitbake core-image-minimal --runall=fetch
'
'';
installPhase = ''
cp -r --reflink=auto build/downloads $out
'';
dontFixup = true;
outputHash = "0r273xj2ac3rid0qwgmy52zlazjbdhyp35l8d74haa712cz876yb";
outputHashMode = "recursive";
outputHashAlgo = "sha256";
};
in
pkgs.stdenv.mkDerivation {
name = "yocto";
src = fetchGit {
url = "https://git.yoctoproject.org/git/poky";
rev = "e35dc77f77f9ab3f3a96819201bc84fb14e205a8";
ref = "refs/heads/hardknott";
};
postPatch = ''
cat << EOF >> meta/recipes-extended/shadow/shadow_*.bb
do_configure_prepend () {
sed "s/4755/0755/g" -i \''${S}/src/Makefile.am
}
EOF
cat << EOF >> meta/recipes-core/util-linux/util-linux_*.bb
do_configure_prepend () {
sed "s/4755/0755/g" -i \''${S}/sys-utils/Makemodule.am
}
EOF
cat << EOF >> meta/classes/base.bbclass
export LOCALE_ARCHIVE = "/usr/lib/locale/locale-archive"
export hardeningDisable = "all"
export NIX_DONT_SET_RPATH = "1"
export NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu = "1"
EOF
'';
configurePhase = ''
mkdir build
cp -r --reflink=auto ${deps} build/downloads
chmod u+w -R build/downloads
'';
buildPhase = ''
${env}/bin/yocto-buildenv -c '
${fixLocale}
export BB_NO_NETWORK=1
source oe-init-build-env
bitbake core-image-minimal
'
'';
installPhase = ''
cp -r build/tmp/deploy $out
'';
passthru = {
inherit deps env;
};
dontFixup = true;
}
1 Like
The shell file in my last post had some issues still so i removed it, however
I looked in to this a bit, its possible to forward variables to the bitbake build environment by exporting variables from a config file. e.g in local.conf:
NIX_DONT_SET_RPATH_class-native = "1"
NIX_CC_WRAPPER_TARGET_HOST_unknown_linux_gnu_class-native = "1"
export NIX_DONT_SET_RPATH
export NIX_CC_WRAPPER_TARGET_HOST_unknown_linux_gnu
This combined with setting the variable BBPOSTCONF to make bitbake read a extra config led me to the following file. There is still some collisions warnings although and now the variables are passed to all packeages, if possible that should only be -native packages since those are the ones that use the host compiler.
{ pkgs ? import <nixpkgs> { } }:
let
fhs = pkgs.buildFHSUserEnvBubblewrap {
name = "yocto-fhs";
targetPkgs = pkgs: (with pkgs;
[
bc
binutils
bzip2
chrpath
cpio
diffstat
expect
file
gcc
gdb
git
gnumake
hostname
kconfig-frontends
lzma
ncurses
patch
perl
python3
rpcsvc-proto
unzip
util-linux
wget
which
]);
# Forces 64bit only environment
multiPkgs = null;
extraOutputsToInstall = [ "dev" ];
profile =
let
wrapperEnvar = "NIX_CC_WRAPPER_TARGET_HOST_${pkgs.stdenv.cc.suffixSalt}";
# TODO limit export to native pkgs?
nixconf = pkgs.writeText "nixvars.conf" ''
# This exports the variables
# to actual build environments
# From BB_ENV_EXTRAWHITE
export LOCALE_ARCHIVE
export ${wrapperEnvar}
export NIX_DONT_SET_RPATH = "1"
# Exclude these when hashing
# the packages in yocto
BB_HASHBASE_WHITELIST_append = " LOCALE_ARCHIVE \
NIX_DONT_SET_RPATH \
${wrapperEnvar} "
'';
in
''
# These are set by buildFHSUserEnvBubblewrap
export BB_ENV_EXTRAWHITE="LOCALE_ARCHIVE \
${wrapperEnvar} \
$BB_ENV_EXTRAWHITE"
# source the config for bibake
# equal to --postread
export BBPOSTCONF="${nixconf}"
'';
};
in
fhs.env
I also noted that this issue prevents any sudo use which is need for qemu. It would be nice to use a mkShell but there is a lot of /bin/bash assumptions in yocto it seems.
It turns out bintools also is wrapped with a reference to glibc from nixpkgs,
setting:
export NIX_BINTOOLS_WRAPPER_TARGET_HOST_${pkgs.stdenv.cc.suffixSalt} = "1"
and adding it to the hash whitelist, seems to fix this.
yocto was now also added to nix-environments: add environemnt for Yocto by Kaisrlik · Pull Request #12 · nix-community/nix-environments · GitHub
4 Likes