Build a yocto rootfs inside nix

Learn
#1

I’m trying to see if I can build images with yocto inside of the nix build environment.

For those not familair, yocto is a “rootfs build system”. It’s similair to nixpkgs in that it’s a meta-build system that contains the necessary information about packages in order to build them and put them together into a rootfs.

The problem I just ran into was with networking inside the nix build sandbox. For obvious reasons, networking is disabled. However, the way yocto works is that it each package contains the remote URL to download it’s source from, thus it needs network access.

I’d still like to use the sandbox build but would like to enable networking for the time being, is that possible?

As an analogy for nix users, imagine you are trying to run nix-build inside a nix container…you would run into the same issue since most derivations need network access to download their sources.

#2

I’m interested in this as well. My method was different in that I didn’t try to build it directly, but use a FHS in conjunction with nix-shell and then build it inside the shell (which allows network access). So far, I got to this:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;
let
  fhsEnv = buildFHSUserEnv {
    name = "yocto-env";

    # Packages Yocto is expecting on the host system by default
    targetPkgs = pkgs: (with pkgs; [
      gcc_multi
      which gnumake (python3.withPackages (p: [ p.pip p.pexpect ])) python2
      gawk wget gitFull diffstat diffutils unzip texinfo bzip2 gzip perl patch
      chrpath file bash cpio utillinux nettools iproute procps openssh socat xz
      debianutils iputils SDL.dev xterm binutils
    ]);

    # Headers are required to build
    extraOutputsToInstall = [ "dev" ];

    # Force install locale from "glibcLocales" since there are collisions
    extraBuildCommands = ''
      ln -sf ${glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
    '';

    # For some reason, nix-shell stalls when adding things to profile
    #profile = ''
    #  export LANG="en_US.UTF-8"
    #  export LC_ALL="en_US.UTF-8"
    #  # Yocto is using the $LOCALEARCHIVE variable
    #  # instead of NixOS's $LOCALE_ARCHIVE
    #  export LOCALEARCHIVE=/usr/lib/locale/locale-archive
    #'';

    multiPkgs = pkgs: (with pkgs; []);
  };

in

(fhsEnv.overrideAttrs (oldAttrs: { hardeningDisable = [ "all" ]; })).env

Unfortunately, there are still some problems with this:

  • A lot of warnings says that “can’t set locale” despite the fact that the locale can be set in the FHS
  • Some programs don’t compile (maybe because of said warnings), in my case it was m4
  • I’m not sure my method of disabling hardening is the right one :-/

For now, I’m using Yocto inside a Debian LXC container, which feels quite sad

I’m sorry I can’t help with the networking issue within the sandbox, but you might run into these issues as well, so I thought it was worth mentioning. I also would be very interested in your current config!

#3

Just building a vanilla image, here’s what I put together so far:

{ stdenv,
  fetchGit ? builtins.fetchGit,
  coreutils,
  gnumake,
  bash,
  bashInteractive,
  which,
  utillinux,
  strace,
  gnugrep,
  gnused,
  gnutar,
  xz,
  patch,
  diffstat,
  bzip2,
  gzip,
  gawk,
  chrpath,
  cpio,
  texinfo,
  file,
  wget,
  hostname,
  perl,
  python,
  python3,
  git,
  gitRepo,
  glibc,
  glibcLocales,
  gcc,
  # yocto, at least on morty branch seems to be using the host useradd/groupadd tools
  shadow,
  # yocto, at least on morty branch seems to require diff even though it doesn't provide
  # an error when it is missing (I added binutils for the 'ar' program)
  diffutils,
  findutils,
  binutils,
}:

stdenv.mkDerivation rec {
  pname = "myrootfs";
  version = "1.0.0";
  srcs = [
    (fetchGit {
      url = git://git.yoctoproject.org/poky;
      ref = "thud";
      rev = "50f33d3bfebcbfb1538d932fb487cfd789872026";
      name = "oe-poky";
    })
    (fetchGit {
      url = git://git.openembedded.org/meta-openembedded;
      ref = "thud";
      rev = "4cd3a39f22a2712bfa8fc657d09fe2c7765a4005";
      name = "oe-embedded";
    })
  ];
  sourceRoot = ".";

  buildInputs = [
    coreutils
    gnumake
    bash
    bashInteractive
    which
    utillinux
    strace
    gnugrep
    gnused
    gnutar
    xz
    patch
    diffstat
    bzip2
    gzip
    gawk
    chrpath
    cpio
    texinfo
    file
    wget
    hostname
    perl
    python
    python3
    git
    gitRepo
    glibc
    glibcLocales
    gcc
    # yocto, at least on morty branch seems to be using the host useradd/groupadd tools
    shadow
    # yocto, at least on morty branch seems to require diff even though it doesn't provide
    # an error when it is missing (I added binutils for the 'ar' program)
    diffutils
    findutils
    binutils
  ];

  buildPhase = ''
    patchShebangs .
    source oe-poky/oe-init-build-env
    echo '# CUSTOM YOCTO SETTINGS'
    echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf
    cat conf/local.conf
    echo running bitbake...
    bitbake core-image-sato
  '';
}
3 Likes
#4

@marler8997 Thanks for sharing the snippet! It would be fantastic if we could have this in the wiki similar to the OpenWRT wiki entry.
Cheers!

#5

Could be also added to https://github.com/nix-community/nix-environments/

#6

I was able to fix the locales issue of minijackson by adding LOCALE_ARCHIVE to BB_ENV_EXTRAWHITE.

#7

I’m trying to do the same here but I have a weird error.

ERROR: Can't get compiler version from gcc  --version output

my shell.nix:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

(pkgs.buildFHSUserEnv {
  name = "yocto-env";
  
  # Packages Yocto is expecting on the host system by default
  targetPkgs = pkgs: (with pkgs; [which gcc7 glibc glibcLocales shadow gnumake python3 python27 gawk wget gitFull diffstat diffutils unzip texinfo bzip2 gzip perl patch chrpath file bash cpio utillinux nettools iproute procps openssh xterm SDL findutils socat gnutar ccache cmake vim binutils bash bashInteractive]);
  
  # Headers are required to build  
  extraOutputsToInstall = [ "dev" ];

  # Force install locale from "glibcLocales" since there are collisions
  extraBuildCommands = ''
    ln -sf ${glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
  '';

  profile = ''
    export hardeningDisable=all
    export CC=gcc
    export LD=ld
    export EDITOR=vim
    export STRIP=strip
    export OBJCOPY=objcopy
    export RANLIB=ranlib
    export OBJDUMP=objdump
    export AS=as
    export AR=ar
    export NM=nm
    export CXX=g++
    export SIZE=size
    export LANG="en_US.UTF-8"
    export LC_ALL="en_US.UTF-8"
    # Yocto is using the $LOCALEARCHIVE variable
    # instead of NixOS's $LOCALE_ARCHIVE
    export LOCALEARCHIVE=/usr/lib/locale/locale-archive
  '';
  
  multiPkgs = pkgs: (with pkgs; []);
  #runScript = "bash";
}).env
#8

I’m stuck at the same error. It used to partially work, seems like a yocto update broke that.

#9

The problem seems to be the following:
Because of $reasons, when the GCC is executed by the bitbake scripts, it gives the following output:

/bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/nix/store/ypag3bh7y7i15xf24zihr343wi6x5i6g-bash-4.4-p23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
gcc (GCC) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

But the bitbake scripts only parse the first line, which doesn’t contain the version number in this case.
For me this was fixed by adding LOCALE_ARCHIVE to BB_EXTRAWHITE.

#10

Following the hints here, I got closer to running shell.nix file:

{ pkgs ? import (fetchTarball https://github.com/nixos/nixpkgs/archive/release-16.09.tar.gz) {} }:

(pkgs.buildFHSUserEnv {
  name = "yocto-env";

  # Packages Yocto is expecting on the host system by default
  targetPkgs = pkgs: (with pkgs; [
    which gcc glibc glibcLocales shadow gnumake python3 python27 gawk wget
    gitFull diffstat diffutils unzip texinfo bzip2 gzip perl patch chrpath file
    bash cpio utillinux nettools iproute procps openssh xterm SDL findutils
    socat gnutar ccache cmake vim binutils bash bashInteractive
  ]);

  # Headers are required to build
  extraOutputsToInstall = [ "dev" ];

  # Force install locale from "glibcLocales" since there are collisions
  extraBuildCommands = ''
    ln -sf ${pkgs.glibcLocales}/lib/locale/locale-archive $out/usr/lib/locale
  '';

  profile = ''
    export hardeningDisable=all
    export CC=gcc
    export LD=ld
    export EDITOR=vim
    export STRIP=strip
    export OBJCOPY=objcopy
    export RANLIB=ranlib
    export OBJDUMP=objdump
    export AS=as
    export AR=ar
    export NM=nm
    export CXX=g++
    export SIZE=size
    export LANG="en_US.UTF-8"
    export LC_ALL="en_US.UTF-8"
    # Yocto is using the $LOCALEARCHIVE variable
    # instead of NixOS's $LOCALE_ARCHIVE
    export LOCALEARCHIVE=/usr/lib/locale/locale-archive
    export BB_ENV_EXTRAWHITE=LOCALEARCHIVE
  '';

  multiPkgs = pkgs: (with pkgs; []);
  runScript = "bash";
}).env

and now, fighting with the m4-native build error. Looking at the config.log file, it shows:

configure:4900: ./conftest
Inconsistency detected by ld.so: dl-call-libc-early-init.c: 37: _dl_call_libc_early_init: Assertion `sym != NULL' failed!
configure:4904: $? = 127
configure:4911: error: in `.../build/tmp/work/x86_64-linux/m4-native/1.4.18-r0/build':
configure:4913: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details

Does someone have any idea about the cause of the error?

#11

Could it be that you mix different libc’s here? Which libc is conftest linked against and which libc is loaded at runtime for executing? LD_DEBUG=libs might provide an answer to the latter one.

#12

Yes, this seems to be one of the issues. I wonder now if we could fix it to allow it to link with the uninative glibc?

#13

since I assume that yocto tries to link its own glibc (?), you could try to use this: https://nixos.wiki/wiki/C#Get_a_compiler_without_default_libc

#14

You can debug with execsnoop from bcc what commands are executed during the build

#15

I’m also trying to make OpenEmbedded build on NixOs.

Exporting LOCALEARCHIVE=${LOCALE_ARCHIVE} and adding those to BB_ENV_EXTRAWHITE seems to make OE’s sanity check pass. But the cannot set locale error are still poping on every shell call. And this make some configuration check scripts fail. Some of them do not like the message on stderr and will fail because they don’t get the expected output because of them.

Setting LOCALEARCHIVE in build/conf/local.conf seems to make those errors disapear:

LOCALE_ARCHIVE := "$LOCALE_ARCHIVE"
LOCALEARCHIVE := "$LOCALE_ARCHIVE"
export LOCALE_ARCHIVE
export LOCALEARCHIVE

Here is the shell expression I use: https://gist.github.com/stammw/d55958fc7ac6185e3446ccf637e53683

1 Like
#16

Sorry for taking long to reply. I believe this is what I need but I couldn’t figure now to fit it in my existing shell.nix file. Is there an example I can base on?

#17

Is your existing shell.nix the one above?

#18

I think @petabyteboy was also talking about building yocto on NixOS. We can add this to https://github.com/nix-community/nix-environments afterwards.

#19

Yes; the link with it is Build a yocto rootfs inside nix

#20

Sure; we can provide it there so more people can use it.