"chroot" into ARM container with systemd-nspawn

Help
1

Hi,

I configured NixOS to emulate aarch64 with:

boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

However, when I try to create an ARM container (“chroot” into an ARM root filesystem) with systemd-nspawn, I get the error that no shell was found, even though these files exist:

# systemd-nspawn -D ./arm-root
execv(/bin/bash, /bin/bash, /bin/sh) failed: No such file or directory

Do you know if the NixOS option mentioned above uses QEMU static user mode emulation? With Arch Linux, one can install qemu-user-static-binfmt and running systemd-nspawn works. Any Ideas, how I could achieve this with NixOS?

Here are the steps I followed (taken from The Simplest Way To Create an ARM Chroot in Arch Linux - nerdstuff.org):

  1. Download an image from Arch Linux ARM
  2. Create a folder that you want to use as root for the “chroot” environment.
    $ mkdir ./arm-root
  3. Extract the image files from the archive into that folder:
    $ bsdtar -xpf -C ./arm-root
  4. Make the new root folder a mount point (not sure if this is required):
    $ mount --bind ./arm-root ./arm-root
  5. “chroot”
    $ sudo systemd-nspawn -D ./arm-root
2

Not sure how this might work on NixOS. On any other distro, you need to register the qemu-user-static emulator on the host (using binfmt_misc: Kernel Support for miscellaneous Binary Formats (binfmt_misc) — The Linux Kernel documentation), but also ensure that a copy of it exists in (or is bind-mounted into) the chroot, at the same location. Obviously if qemu-user-static is installed natively on a NixOS host, it’s going to need to be at an “odd” location in the chroot.

I would be surprised if this wasn’t already solved somewhere, it’s just a question of finding it:)

3

Ah, looking at https://github.com/NixOS/nixpkgs/blob/b41bb11ed86547cf80f8f144639ceaae8807cb19/nixos/modules/system/boot/binfmt.nix, I think if you set wrapInterpreterInShell to false, the interpreter would end up symlinked into /run/binfmt/ - you might be able to bind mount that into the container? (I think if you bind mount the individual file, not the dir, it will follow symlinks, but not 100% sure.)

4

I got this problem, too. Do you have any solution?

5

If I remember correctly, I did not find a solution to this exact problem. If you want to build custom images for NixOS on ARM, this might help you: NixOS on a Raspberry Pi: creating a custom SD image with OpenSSH out of the box | Roberto Frenna

6

I’m tring to step into a risc-v archlinux for develop. Thanks anyway.

7

got it.

riscv64 for example:

  boot.binfmt = {
    emulatedSystems = [ "aarch64-linux" "riscv64-linux" ];

    # https://github.com/felixonmars/archriscv-packages/blob/7c270ecef6a84edd6031b357b7bd1f6be2d6d838/devtools-riscv64/z-archriscv-qemu-riscv64.conf#L1
    registrations."riscv64-linux" = {
      preserveArgvZero = true;
      matchCredentials = true;
      fixBinary = true;
    };
  };

Also take a look at Cannot chroot to a different CPU architecture using `boot.binfmt.emulatedSystems` · Issue #160300 · NixOS/nixpkgs · GitHub

2 Likes
8

All you should need to do is bind-mount the host system Nix store into the container.

This is necessary because it contains the binaries registered for emulation.

9

Update: the only thing you need now is just boot.binfmt.preferStaticEmulators = true;