Docker ignoring platform when run in NixOS

1

I’m runnning NixOS in a VM on an M1 mac. My application runs via docker compose.

One of the docker services is a mhart/alpine-node:14. When I run docker compose natively on macOS for my application, everything runs fine.

If I do the same in the nixOS VM, the docker mhart/alpine-node:14 image doesn’t run on the right platform and I get error:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
exec /usr/bin/npm: exec format error

I can’t figure out why docker behaves differently wrt to platforms when run in Nix.

2

What is the nature of the VM? How are you running it?

3

Vmware fusion tech preview. Used GitHub - mitchellh/nixos-config: My NixOS configurations. as a starting point.

4

Is uname -m returning aarch64 in the nixos VM?

5

Yes it is. I assume docker compose is behaving different in nixos than on native mac. Platform is not being set somehow in nixos, my best guess.

6

I actually ran docker build on native mac and passed --platform linux/aarch64 and it worked fine but it failed on nixos when trying the same thing.

7

Docker desktop for mac includes x86 emulation. Are you sure the image isn’t actually an x86 one in both cases?

8

Firstly just wanna say thanks for replying everyone.

The image is x86 but I’ve run the same stack on my Ubuntu desktop and I didn’t have this problem. Docker-compose I assume passes platform target automatically, but somehow that isn’t happening in nix ? Not sure.

9

Ubuntu desktop’s dockerd might have a default platform set for the daemon.

10

Right now I am trying to figure out qemu emulation in my nixos vm. I’ve come to the point where running
docker buildx inspect --bootstrap
returns

 docker buildx inspect --bootstrap                                                                                  107ms  Sat 27 Aug 2022 07:39:57 AM PDT
[+] Building 0.5s (1/1) FINISHED                                                                                                                               
 => [internal] booting buildkit                                                                                                                           0.5s
 => => starting container buildx_buildkit_bldr0                                                                                                           0.5s
Name:   bldr
Driver: docker-container

Nodes:
Name:      bldr0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/arm64

So my nixos isn’t capable of building for other platforms. I’m not sure how exactly to track down what’s missing but I am following this post to see if I can get a clue for now…

11

I have no idea about docker’s buildx but be aware than NixOS has boot.binfmt.emulatedSystems.

12

I’m assuming buildx is getting it’s platform emulation via binfmt. But even after adding
boot.binfmt.emulatedSystems= ["i386-linux" "x86_64-linux" ];
to my config, I still don’t get those platforms available to buildx builder.

From Docker docs multi arch:

Docker Desktop* provides binfmt_misc multi-architecture support, which means you can run
containers for different Linux architectures such as arm , mips , ppc64le , and even s390x .
This does not require any special configuration in the container itself as it uses qemu-static from the Docker for Mac VM . Because of this, you can run an ARM
container, like the arm32v7 or ppc64le variants of the busybox image.

13

Have you verified that you are able to execute x86 programs outside Docker?

14

So the entire problem was that my system didn’t have the emulators for other platforms somehow.

In all the docker buildx tutorials you see out there, people had cross platform emulation out of the box but my system only showed linux/arm64 when I ran docker buildx inspect .
Until I found this Repo. I ran docker run --privileged --rm tonistiigi/binfmt --install all just to make sure and finally my platform option in docker-compose or docker build worked.

So if anyone is stuck with the same problem, check what you get under the platforms column when you run docker buildx ls. If there’s only your own platform, you need to add the qemu emulators manually or via this docker command from the above linked repo. Thanks for your help @Atemu

4 Likes
15

just want to say thanks for documenting this. It solved my problem :slight_smile:
I was curious if there is some other way to do it instead of relying on a image

16

Hello,
I have recently started tinkering with NixOS and I faced the same issue. For me the problem was that the qemu-{platform} given by NixOS are dynamically linked which Docker does not like.

As a workaround (still ugly but at least declarative), I stole the qemu-user-static binaries from the debian repos:

This looks like this:

{ pkgs, ... }:

let
  qemu-static-derivation = {
    stdenv,
    fetchurl,
    dpkg,
    ...
  }:
    stdenv.mkDerivation {
      name = "qemu-static";
      pname = "qemu-static";

      dontPatchELF = true;
      dontConfigure = true;
      dontPatch = true;

      src = fetchurl {
        url = "http://ftp.fr.debian.org/debian/pool/main/q/qemu/qemu-user-static_9.0.0~rc2+ds-1_amd64.deb";
        sha256 = "sha256-JrYf2oKUCIptqllolzPhOopVuSEfATPo+fi2bjDlQl4=";
      };

      unpackPhase = ''
        mkdir -p $out

        ${dpkg}/bin/dpkg-deb -x $src $out
      '';
    };

  qemu-static = pkgs.callPackage qemu-static-derivation {};
in 
{
  environment.systemPackages = [
    qemu-static
  ];

  boot.binfmt.registrations."arm64-linux" = {
    interpreter = "${qemu-static}/usr/bin/qemu-aarch64-static";
    magicOrExtension = ''\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00'';
    mask = ''\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\x00\xff\xfe\xff\xff\xff'';
    fixBinary = true;
    matchCredentials = true;
    wrapInterpreterInShell = false;
  };
}

In my case I am doing linux/arm64 emulation from an amd64 laptop, but it would be possible to do the other way around by grabbing the qemu-user-static binaries in the debian arm repos instead and setting up the binfmt registration for amd64 by changing the interpreter path, magic and mask.

Additionally, as you can see in the above config, I had to add the fixBinary flag and disable the wrapInterpreterInShell. By the way I am curious to know why NixOS default behavior is to wrap the interpreter in a shell script, what does it accomplish?

Hope this helps. And if someone thinks of a way to improve on that solution, please share it. Maybe some version of qemu static binaries should be present in nixpkgs to simplify this use case, what do you think?

17

The option to use statically-linked QEMU is being worked on. That seems to have stalled a bit due to merge conflicts but #300070 rebased the changes and is currently active.

I think the wrapper handles the binfmt_misc P flag (see here for more details, ctrl-f for preserve-argv), and prevents LD_* environment variables intended to be passed to the emulated binary from affecting QEMU itself. Newer versions of QEMU handle the P flag natively (see this commit) and LD_* is irrelevant to statically-linked executables anyway, so the PR I linked to above also disables the wrapper when you are using qemu-user-static.

1 Like
18

As of nixos/binfmt: Add option to use static emulators when available by jcaesar · Pull Request #334859 · NixOS/nixpkgs · GitHub
Setting:

  boot.binfmt = {
    emulatedSystems = [ "aarch64-linux" ];
    preferStaticEmulators = true; # Make it work with Docker
  };

Just Works™

Unfortunately that is not on stable yet, but the following works on stable (using the fixed unstable packages as pkgsUnstable):

  boot.binfmt =
    let
      system = "aarch64-linux";
    in
    {
      emulatedSystems = [ system ];
      registrations.${system} = {
        interpreter = (pkgsUnstable.lib.systems.elaborate { inherit system; }).emulator pkgsUnstable.pkgsStatic;
        fixBinary = true;
      };
    };

Either way (stable or unstable), setting registrations.aarch64-linux.matchCredentials = true will typically be useful as well if using suid binaries.

3 Likes
19

Thank you Ten, that works flawlessly!! (crediting your comment in my latest blog post!)

1 Like
20

Many thanks.

  boot.binfmt = {
    emulatedSystems = [ "aarch64-linux" ];
    preferStaticEmulators = true; # Make it work with Docker
  };

Seem to be enough now.
I was banging my head against why i was getting

docker run --platform linux/arm64 --rm my-app:latest  
exec /nix/store/8s9a7drrh6hwzmivwmbdpjr146awk7l5-my-app-aarch64-unknown-linux-gnu/bin/my-app: no such file or directory