Why do darwin build derivations end up on a NixOS system?

I’m trying to deploy a NixOS machine using nixops on MacOS. I configured a x86_64-linux build machine so that my nix on MacOS can build derivations for Linux:

cat /etc/nix/machines
ssh://nixBuild@X.X.X.X x86_64-linux /var/root/.ssh/id_nixBuild 48 2 benchmark,big-parallel,nixos-test

The NixOS configuration that I’m trying to deploy also contains the following to make sure all derivations are build for Linux:

nixpkgs.localSystem = { system = "x86_64-linux"; };

Let’s try to deploy it (note that the configuration also enables hydra):

$ nixops deploy ...
building '/nix/store/k1i3mnzl3rx66inm990xkm4hpdn5ndc4-hydra-0.1.2544.63a294.drv' on 'ssh://nixBuild@X.X.X.X'...
checking whether /nix/store/dhz34xy8k97i7551dhmmv9kbipd1gvji-hydra-perl-deps/bin/nix-store is recent enough... ./configure: line 16254: /nix/store/dhz34xy8k97i7551dhmmv9kbipd1gvji-hydra-perl-deps/bin/nix-store: cannot execute binary file: Exec format error
builder for '/nix/store/k1i3mnzl3rx66inm990xkm4hpdn5ndc4-hydra-0.1.2544.63a294.drv' failed with exit code 1

So Nix is correctly performing the build of the hydra derivation on my Linux build machine. However the build fails during the configure phase when executing the nix-store executable. The reason is that this executable is build for MacOS and not for Linux. As in I can run it correctly from my Mac:

$ /nix/store/dhz34xy8k97i7551dhmmv9kbipd1gvji-hydra-perl-deps/bin/nix-store --version
nix-store (Nix) 2.1pre6377_954d1f4d

But if I execute the same on the NixOS build server:

$ ssh root@X.X.X.X /nix/store/dhz34xy8k97i7551dhmmv9kbipd1gvji-hydra-perl-deps/bin/nix-store --version
bash: /nix/store/dhz34xy8k97i7551dhmmv9kbipd1gvji-hydra-perl-deps/bin/nix-store: cannot execute binary file: Exec format error

Any idea why these Mac executables end up in the hydra derivation that is configured to be build for x86_64-linux?

One suspicion I have is that this is caused by buildEnv preferring local builds over using a remote builder. Note that the hydra derivation is using buildEnv for building the problematic hydra-perl-deps derivation.

I’m currently doing a build where I commented out preferLocalBuild = true in buildenv/default.nix to see if that fixes things…

Unfortunately disabling preferLocalBuild in buildEnv doesn’t help. It also seems that all the symlinks in /nix/store/dhz34xy8k97i7551dhmmv9kbipd1gvji-hydra-perl-deps/bin/ point to correctly build executables for Linux except for the Nix tools.

Any idea why the Nix derivation gets build for MacOS while the rest is correctly build for Linux?

I found the bug and submitted a PR for hydra to fix it: