Could be also added to GitHub - nix-community/nix-environments: Repository to maintain out-of-tree shell.nix files (maintainer=@mic92)
I’m trying to do the same here but I have a weird error.
ERROR: Can't get compiler version from gcc --version output
my shell.nix:
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
(pkgs.buildFHSUserEnv {
name = "yocto-env";
# Packages Yocto is expecting on the host system by default
targetPkgs = pkgs: (with pkgs; [which gcc7 glibc glibcLocales shadow gnumake python3 python27 gawk wget gitFull diffstat diffutils unzip texinfo bzip2 gzip perl patch chrpath file bash cpio utillinux nettools iproute procps openssh xterm SDL findutils socat gnutar ccache cmake vim binutils bash bashInteractive]);
# Headers are required to build
extraOutputsToInstall = [ "dev" ];
# Force install locale from "glibcLocales" since there are collisions
extraBuildCommands = ''
ln -sf ${glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
'';
profile = ''
export hardeningDisable=all
export CC=gcc
export LD=ld
export EDITOR=vim
export STRIP=strip
export OBJCOPY=objcopy
export RANLIB=ranlib
export OBJDUMP=objdump
export AS=as
export AR=ar
export NM=nm
export CXX=g++
export SIZE=size
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
# Yocto is using the $LOCALEARCHIVE variable
# instead of NixOS's $LOCALE_ARCHIVE
export LOCALEARCHIVE=/usr/lib/locale/locale-archive
'';
multiPkgs = pkgs: (with pkgs; []);
#runScript = "bash";
}).env
I’m stuck at the same error. It used to partially work, seems like a yocto update broke that.
The problem seems to be the following:
Because of $reasons, when the GCC is executed by the bitbake scripts, it gives the following output:
/bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/nix/store/ypag3bh7y7i15xf24zihr343wi6x5i6g-bash-4.4-p23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
gcc (GCC) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
But the bitbake scripts only parse the first line, which doesn’t contain the version number in this case.
For me this was fixed by adding LOCALE_ARCHIVE to BB_EXTRAWHITE.
Following the hints here, I got closer to running shell.nix file:
{ pkgs ? import (fetchTarball https://github.com/nixos/nixpkgs/archive/release-16.09.tar.gz) {} }:
(pkgs.buildFHSUserEnv {
name = "yocto-env";
# Packages Yocto is expecting on the host system by default
targetPkgs = pkgs: (with pkgs; [
which gcc glibc glibcLocales shadow gnumake python3 python27 gawk wget
gitFull diffstat diffutils unzip texinfo bzip2 gzip perl patch chrpath file
bash cpio utillinux nettools iproute procps openssh xterm SDL findutils
socat gnutar ccache cmake vim binutils bash bashInteractive
]);
# Headers are required to build
extraOutputsToInstall = [ "dev" ];
# Force install locale from "glibcLocales" since there are collisions
extraBuildCommands = ''
ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
'';
profile = ''
export hardeningDisable=all
export CC=gcc
export LD=ld
export EDITOR=vim
export STRIP=strip
export OBJCOPY=objcopy
export RANLIB=ranlib
export OBJDUMP=objdump
export AS=as
export AR=ar
export NM=nm
export CXX=g++
export SIZE=size
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
# Yocto is using the $LOCALEARCHIVE variable
# instead of NixOS's $LOCALE_ARCHIVE
export LOCALEARCHIVE=/usr/lib/locale/locale-archive
export BB_ENV_EXTRAWHITE=LOCALEARCHIVE
'';
multiPkgs = pkgs: (with pkgs; []);
runScript = "bash";
}).env
and now, fighting with the m4-native build error. Looking at the config.log file, it shows:
configure:4900: ./conftest
Inconsistency detected by ld.so: dl-call-libc-early-init.c: 37: _dl_call_libc_early_init: Assertion `sym != NULL' failed!
configure:4904: $? = 127
configure:4911: error: in `.../build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/build':
configure:4913: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
Does someone have any idea about the cause of the error?
Could it be that you mix different libc’s here? Which libc is conftest linked against and which libc is loaded at runtime for executing? LD_DEBUG=libs might provide an answer to the latter one.
Yes, this seems to be one of the issues. I wonder now if we could fix it to allow it to link with the uninative glibc?
since I assume that yocto tries to link its own glibc (?), you could try to use this: C - NixOS Wiki
You can debug with execsnoop from bcc what commands are executed during the build
1 Like
I’m also trying to make OpenEmbedded build on NixOs.
Exporting LOCALEARCHIVE=${LOCALE_ARCHIVE} and adding those to BB_ENV_EXTRAWHITE seems to make OE’s sanity check pass. But the cannot set locale error are still poping on every shell call. And this make some configuration check scripts fail. Some of them do not like the message on stderr and will fail because they don’t get the expected output because of them.
Setting LOCALEARCHIVE in build/conf/local.conf seems to make those errors disapear:
LOCALE_ARCHIVE := "$LOCALE_ARCHIVE"
LOCALEARCHIVE := "$LOCALE_ARCHIVE"
export LOCALE_ARCHIVE
export LOCALEARCHIVE
Here is the shell expression I use: yocto.nix · GitHub
Sorry for taking long to reply. I believe this is what I need but I couldn’t figure now to fit it in my existing shell.nix file. Is there an example I can base on?
Is your existing shell.nix the one above?
I think @petabyteboy was also talking about building yocto on NixOS. We can add this to GitHub - nix-community/nix-environments: Repository to maintain out-of-tree shell.nix files (maintainer=@mic92) afterwards.
Sure; we can provide it there so more people can use it.
This is the fix that stammw was talking about:
$ MACHINE=raspberrypi3-64 source setup-environment
$ export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE LOCALEARCHIVE LOCALE_ARCHIVE"
$ bitbake lmp-base-console-image
That works for me:
{ pkgs ? import (fetchTarball https://github.com/nixos/nixpkgs/archive/release-20.09.tar.gz) {} }:
(pkgs.buildFHSUserEnv {
name = "yocto-env";
# Packages Yocto is expecting on the host system by default
targetPkgs = pkgs: (with pkgs; let
sh = (pkgs.runCommand "sh" {} ''
mkdir -p $out/bin
cat > $out/bin/bash <<'EOF'
#!${bashInteractive}/bin/bash
export LOCALE_ARCHIVE=/usr/lib/locale/locale-archive
export LOCALEARCHIVE=/usr/lib/locale/locale-archive
exec -a /bin/bash ${bashInteractive}/bin/bash "$@"
EOF
chmod +x $out/bin/bash
ln -s $out/bin/bash $out/bin/sh
'');
in [
which gcc glibc glibcLocales shadow gnumake python27 gawk wget
gitFull diffstat diffutils unzip texinfo bzip2 gzip perl patch chrpath file
cpio utillinux nettools iproute procps openssh xterm SDL findutils
socat gnutar ccache cmake vim binutils gitRepo
(pkgs.runCommand "python3" {} ''
mkdir -p $out/bin
cat > $out/bin/python3 <<'EOF'
#!${runtimeShell}
export LOCALE_ARCHIVE=/usr/lib/locale/locale-archive
export LOCALEARCHIVE=/usr/lib/locale/locale-archive
exec -a /usr/bin/python3 ${python3}/bin/python "$@"
EOF
chmod +x $out/bin/python3
'')
(hiPrio sh)
]);
# Headers are required to build
extraOutputsToInstall = [ "dev" ];
# Force install locale from "glibcLocales" since there are collisions
extraBuildCommands = ''
ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
'';
profile = ''
export hardeningDisable=all
export CC=gcc
export LD=ld
export EDITOR=vim
export STRIP=strip
export OBJCOPY=objcopy
export RANLIB=ranlib
export OBJDUMP=objdump
export AS=as
export AR=ar
export NM=nm
export CXX=g++
export SIZE=size
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export SHELL=/bin/bash
'';
multiPkgs = pkgs: (with pkgs; []);
runScript = "bash";
}).env
Update this version still had some issues. I think the proper fix would be to include en_US.utf-8 in our glibc version by default (can be done via overrideAttrs) and rebuild python/bash/perl against that.
So, I had another opportunity at work to use Yocto, and decided to give this a shot again, thanks to all the comments here, I managed to go a lot further than ever before.
Adding LOCALE_ARCHIVE to $BB_ENV_EXTRAWHITE does fix the bitbake locale error, although I’m still seeing warnings all over the place (more on that later).
The m4 configure error does come from the usage of two different glibc at the same time, as @Mic92. Here’s what I think happened:
While compiling host tools, Yocto tries to make the compiler use Yocto’s own provided glibc (rightfully so, since a lot of different Linux distributions have different libc implementations / versions / etc.). One implication of this is the addition of this flag to GCC arguments:
-Wl,--dynamic-linker=$YOCTO_DIR/build/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2
This effectively sets the ELF interpreter to the one from Yocto’s glibc. However, it does not exactly tell the interpreter where to find libc.so, this still has to be done at runtime.
Where it goes wrong, is that NixOS’s GCC adds its own glibc lib path into the executable runpath, by appending these arguments to the linker:
-rpath /nix/store/$GLIBC_HASH-glibc-2.31-74/lib
-rpath /nix/store/$GCC_HASH-gcc-9.3.0-lib/lib
This tells the interpreter where to search for librairies, after trying the paths from $LD_LIBRARY_PATH.
In effect, this means that when running the program, it will run through Yocto’s interpreter, but will link against NixOS glibc, which may differ in their versions (at the time of writing, glibc-2.32 for nixpkgs-unstable, and glibc-2.33 for Yocto/Poky master). As far as I know glibc doesn’t have any ABI stability, so this is recipe for disaster (usually segfaults, or interpreter error).
After countless hours trying to reverse-engineer NixOS’ GCC wrapper, disabling the addition of the rpath flags is done by adding two environment variables:
export NIX_DONT_SET_RPATH=1
export NIX_CC_WRAPPER_TARGET_HOST_$HOST_ARCH=1
# For example NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu=1
The second variable was harder to find, I assume this is because it is meant to be an internal nixpkgs variable used for building things with Nix.
However, exporting and adding these to $BB_ENV_EXTRAWHITE does nothing… It seems that $BB_ENV_EXTRAWHITE is not meant to pass variables to the build environment, but the bitbake environment (this might also explain the locale warnings). I’m not sure if there is a better way to this in Yocto, but my quick fix was simply to wrap gcc and g++, and set these variables directly in there.
I stumbled upon a configure error while compiling cmake-native, telling me the compiler doesn’t support C++11 features, and I assume this is because it interprets the recurrent locale warning as a compiler warning, and fails the check. At this point I was a bit tired, and did an even dirtier fix, by going into the CMake source, under build/tmp/work/x86_64-linux/cmake-native/3.19.5-r0/cmake-3.19.5/Source/Checks/cm_cxx_features.cmake and replaced this line:
- if(check_output MATCHES "(^|[ :])[Ww][Aa][Rr][Nn][Ii][Nn][Gg]")
+ if(check_output MATCHES "(^|[ :])[Aa][Rr][Nn][Ii][Nn][Gg]")
Another issue I stumbled upon was that some C++ programs did not manage to find the libstdc++.so shared library. This was because both the rpath was not set (rightfully), the $LD_LIBRARY_PATH was emptied in the build environment, and if I remember correctly, this was a program that was using NixOS’ interpreter. Thankfully, this was fixed by using the latest unstable version of nixpkgs, thanks to this awesome PR: steam: fix proton versions with pressure-vessel by LuigiPiucco · Pull Request #114024 · NixOS/nixpkgs · GitHub
The reason for this is that now the Bubblewrap FHS user env generates an ld.so.conf with its ld.so.cache file, it is now able to find the library.
I would really like to know if someone knows a way to properly forward some environment variables to the build environment, so that we can upstream a fully-working version of this.
Anyway, here's my version, so far:
{ pkgs ? import <nixpkgs-unstable> {} }:
let
bashrcFile = pkgs.writeText "bashrc" ''
eval "$(${pkgs.starship}/bin/starship init bash)"
'';
gcc_multi_norpath = pkgs: (pkgs.gcc_multi.override (old: {
extraBuildCommands = ''
wrapProgram $out/bin/gcc \
--set NIX_DONT_SET_RPATH 1 \
--set NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu 1
wrapProgram $out/bin/g++ \
--set NIX_DONT_SET_RPATH 1 \
--set NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu 1
'';
})).overrideAttrs (old: {
nativeBuildInputs = [ pkgs.makeWrapper ];
});
in pkgs.buildFHSUserEnvBubblewrap {
name = "yocto-on-nixos";
targetPkgs = pkgs: with pkgs; [
bc file gnumake python3 unzip which patch perl util-linux rsync wget diffstat
(gcc_multi_norpath pkgs)
glibcLocales gdb valgrind
libglvnd mesa ncurses pkgconfig qt5.qtbase
bzip2 lzma
];
multiPkgs = pkgs: (with pkgs; []);
extraOutputsToInstall = [ "dev" ];
profile = ''
export hardeningDisable=all
export CC=gcc
export LD=ld
export STRIP=strip
export OBJCOPY=objcopy
export RANLIB=ranlib
export OBJDUMP=objdump
export AS=as
export AR=ar
export NM=nm
export CXX=g++
export SIZE=size
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export LOCALEARCHIVE=/usr/lib/locale/locale-archive
export BB_ENV_EXTRAWHITE="LOCALEARCHIVE LOCALE_ARCHIVE NIX_DONT_SET_RPATH NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu"
export NIX_DONT_SET_RPATH=1
'';
extraBuildCommands = ''
ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
'';
runScript = "bash --rcfile ${bashrcFile}";
}
Edit: Just managed to build core-image-base with this, I’m so happy.
2 Likes
You might be able to use an unwrapped GCC but than you would need additional variables.
Devshell wants to go in this direction and provide unwrapped compilers. An alternative to a wrapped CC might be to define SYSROOT as posted here: [c/c++] Investigate $SYSROOT · Issue #103 · numtide/devshell · GitHub