Building a docker image on macOS with remote builder

buildImage doesn’t use Docker itself and just adds to an image whatever has been built on host system, which means that if I’m building it on macOS - despite my building suceeding I won’t be able to run the image because executable format will be wrong:

$ docker run $SOME_IMAGE
standard_init_linux.go:211: exec user process caused "exec format error"

As as workaround I tried to use remote builder on top of LnL7/nix-docker image. Remote builder works as expected for nix-build <nixpkgs>:

$ nix-build '<nixpkgs>' -A hello --argstr system x86_64-linux
/nix/store/hs6rg4zbsclx660s6i5938605zmv6lgh-hello-2.10
$ ./result/bin/hello # error as expected
zsh: exec format error: ./result/bin/hello

But when I try to use it for my own nix expression, it still builds the macOS executable:

$ cat docker.nix
{ pkgs ? import <nixpkgs> { } }:
with pkgs;
dockerTools.buildImage {
  name = "nix-docker-test";
  tag = "0.0.1";
  contents = [ hello ];
  config = {
    Cmd = [ hello ];
    Entrypoint = [ stdenv.shell ];
  };
}
$ nix-build docker.nix --argstr system x86_64-linux
...
$ docker run $ANOTHER_IMAGE
standard_init_linux.go:211: exec user process caused "exec format error"

How do tell nix-build to use remote builder in this case? I tried to overwrite { system = "x86_64-linux" } in <nixpkgs>, but got:

error: a 'x86_64-linux' with features {} is required to build '/nix/store/i3bv2aghqy4ib5xczlm3wv6r39vpdlj4-nix-docker-test-config.json.drv', but I am a 'x86_64-darwin' with features {benchmark, big-parallel, nixos-test}

Try updating your source file like so to specify the target system type appropriately:

{ system ? "x86_64-linux", pkgs ? import <nixpkgs> { inherit system; } }:
with pkgs;
# ...
1 Like

Nope, didn’t work. Still the same

error: a 'x86_64-linux' with features {} is required to build '/nix/store/i3bv2aghqy4ib5xczlm3wv6r39vpdlj4-nix-docker-test-config.json.drv', but I am a 'x86_64-darwin' with features {benchmark, big-parallel, nixos-test}

Last time I did just:

{ pkgs ? import <nixpkgs> { system = "x86_64-linux"; } }

I think, result is the same?

Ok, I think it’s something weird going on with my SSH connection. I tried to force it to use remote builders only by:

$ nix-build docker.nix --max-jobs 0 --builders nix-docker -vvvvvvvvv 2>&1

I was getting different results depending on some other minor parameters (lost track of them), but it does try to build on remote host and eventually I get

hook reply is 'decline'

If I ssh into the container and try to install something, I get:

$ ssh nix-docker
lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
-bash-4.4# nix-env -i hello
warning: there are multiple derivations named 'hello-2.10'; using the first one
installing 'hello-2.10'
warning: unable to download 'https://cache.nixos.org/kmnqp3hvhz7ym9inm4m4bnkmmwxc6mzz.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 305 ms
warning: unable to download 'https://cache.nixos.org/kmnqp3hvhz7ym9inm4m4bnkmmwxc6mzz.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 699 ms
warning: unable to download 'https://cache.nixos.org/kmnqp3hvhz7ym9inm4m4bnkmmwxc6mzz.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 1019 ms
warning: unable to download 'https://cache.nixos.org/kmnqp3hvhz7ym9inm4m4bnkmmwxc6mzz.narinfo': SSL peer certificate or SSH remote key was not OK (60); retrying in 2376 ms
error: unable to download 'https://cache.nixos.org/kmnqp3hvhz7ym9inm4m4bnkmmwxc6mzz.narinfo': SSL peer certificate or SSH remote key was not OK (60)

But what is weirder, if I just login to the container by docker exec -it $CONT /bin/sh - everything works just fine. So I think some SSL config just gets propagated from my host machine.