How do I get my "aarch64-linux" machine to build "x86_64-linux"? `extra-platforms` doesn't seem to work

runeksvendsen · January 11, 2024, 7:38am

My /etc/nix/nix.conf looks like this:

build-users-group = nixbld
extra-platforms = [ "x86_64-linux" ]

binary-caches = https://cache.nixos.org https://nixcache.reflex-frp.org
binary-cache-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= ryantrinkle.com-1:JJiAKaRv9mWgpVAz8dwewnZe0AzzEAzPkagE9SP5NWI=
binary-caches-parallel-connections = 40

notice the extra-platforms.

But when I try to build this repo/commit, using an x86_64-linux nix binary I get the error:

$ NIX_PATH="nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-23.11.tar.gz" $(nix-build -A nix '<nixpkgs>' --argstr system "x86_64-linux")/bin/nix-build --show-trace
error:
       … while calling anonymous lambda

         at /home/parallels/code/crypto-orderbook-db-app/default.nix:1:1:

            1| { sources ? import ./nix/sources.nix { system = "x86_64-linux"; }
             | ^
            2| , haskellNix ? import sources.haskellNix {}

       … from call site

         at /home/parallels/code/crypto-orderbook-db-app/default.nix:10:12:

            9|   # Docs: https://github.com/ilyakooo0/haskell-nix-extra-hackage/blob/master/README.md#how-to-use-it
           10|   hsPkgs = pkgs.haskell-nix.cabalProject ({
             |            ^
           11|     src = ./.;

       … while calling 'cabalProject'

         at /nix/store/kvj9d8fa47jfvpq11zawv6x4l5hcdbsw-haskellNix-src/overlays/haskell.nix:918:24:

          917|
          918|         cabalProject = args: let p = cabalProject' args;
             |                        ^
          919|             in p.hsPkgs // p;

       … from call site

         at /nix/store/kvj9d8fa47jfvpq11zawv6x4l5hcdbsw-haskellNix-src/overlays/haskell.nix:655:27:

          654|               callProjectResults = callCabalProjectToNix args;
          655|               plan-pkgs = importAndFilterProject {
             |                           ^
          656|                 inherit (callProjectResults) projectNix sourceRepos src;

       … while calling anonymous lambda

         at /nix/store/kvj9d8fa47jfvpq11zawv6x4l5hcdbsw-haskellNix-src/lib/import-and-filter-project.nix:5:1:

            4| { pkgs, haskellLib }:
            5| { projectNix, sourceRepos, src }:
             | ^
            6| let

       error: a 'x86_64-linux' with features {} is required to build '/nix/store/hnls6fras52s2wlw1hdfcxvshmx73phf-cabal.project.drv', but I am a 'aarch64-linux' with features {benchmark, big-parallel, nixos-test, uid-range}

I’m building using an x86-linux nix binary because if I use an aarch64 binary then the system is assumed to be aarch64-linux, which I don’t want.

EDIT: FYI I’m on aarch64 Ubuntu 22.04.3:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.3 LTS
Release:	22.04
Codename:	jammy

$ arch
aarch64

NobbZ · January 11, 2024, 7:50am

Make sure you also have binfmt setup correctly.

Then just use --system x86_64-linux when running nix build.

github.com

NobbZ/nixos-config/blob/f64fa125fd565b3e98abbb1011aa1ce6cb6e2e67/nixos/configurations/mimas.nix#L46-L47


      
          boot.binfmt.emulatedSystems = ["i686-linux" "aarch64-linux"];
          nix.settings.extra-platforms = config.boot.binfmt.emulatedSystems;

This is what I have to build aarch64-linux.

IIRC you have to reboot to make binfmt work

runeksvendsen · January 11, 2024, 8:16am

@NobbZ I already have execution of x86 binaries working (using Rosetta 2 with an aarch64 Linux VM), hence why execution of this x86 Nix binary works:

$ NIX_PATH="nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-23.11.tar.gz" file $(readlink -f $(nix-build -A nix '<nixpkgs>' --argstr system "x86_64-linux")/bin/nix-build)
/nix/store/frs3bna4fwzacqfi7mi3zbcryx9pgwi4-nix-2.17.1/bin/nix: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/ld-linux-x86-64.so.2, BuildID[sha1]=64bbf66f1a4427d2a36cf48a97635f189d39e582, for GNU/Linux 3.10.0, not stripped

$ NIX_PATH="nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-23.11.tar.gz" $(nix-build -A nix '<nixpkgs>' --argstr system "x86_64-linux")/bin/nix-build --version
nix-build (Nix) 2.17.1

The problem is getting Nix to understand that my system can also run x86 executables (and therefore can build x86_64-linux derivations). I thought setting extra-platforms would be sufficient, but it’s not.

NobbZ · January 11, 2024, 8:20am

I do not understand, are you on MacOS (as you mention Rosetta) or NixOS or something else (as you mention Ubuntu in your OP)?

ElvishJerricco · January 11, 2024, 10:58am

@NobbZ macOS provides a linux version of rosetta. When you start a linux VM on macOS, it can provide a virtiofs containing a statically linked rosetta binary that can be used to run x86_64 linux software on aarch64 linux via binfmt. It is tremendously faster than qemu emulation.

NobbZ · January 11, 2024, 11:21am

Then my only guess is, that for some reason the IFD switches into some wrong execution context that isn’t aware of the foreign running mechanism.

Clean up the IFD and it will probably work.

Sadly this also means to reimplement the subject in a language other than Haskell.

It might be easier to create an x86_64-linux VM

ElvishJerricco · January 11, 2024, 11:26am

I don’t think this should be a problem. I just tried IFD’ing an aarch64-linux derivation on my x86_64-linux machine and it worked fine.

Not if the goal is to have rosetta-levels of performance.

NobbZ · January 11, 2024, 12:03pm

So far I never had the problem either, though:

It was using aarch64-linux binfmt on native x86_64-linux
It was using nix3 tooling not nix2

Sadly I lack the aarch64 physical machine to even try something close to what the OP is doing here.

Though from what we see here the IFD seems to be related. Nixpkgs seems to work the way OP is using it, only the build of the local callCabal thing, which involves IFD, is failing when building the input parameters.

This is either because the original system parameters didn’t get threaded correctly, or because something in the IFD “escapes” the system.

I sadly have no clue how builtins.currentSystem is used in callCabal or how it would behave when overriding the system argument like this, rather than using the nix3 --system flag!

runeksvendsen · January 15, 2024, 7:32am

@NobbZ @ElvishJerricco thank you both for your input! In particular I wasn’t aware that this appears to be a haskell.nix issue. Consequently, I have logged a haskell.nix bug here: Failure when trying to build for "x86_64-linux" on "aarch64-linux" · Issue #2143 · input-output-hk/haskell.nix · GitHub. Let’s see what they say.

ElvishJerricco · January 15, 2024, 7:53am

This shouldn’t be relevant. We already know the right system is making it through evaluation, because Nix is complaining about lacking an x86_64-linux system to build on. i.e. This most likely isn’t a haskell.nix bug, because we know evaluation is resulting in what we expect: an x86_64-linux derivation.

@runeksvendsen Are you able to build any packages at all using x86_64-linux? Like, can you nix-build something like this?

with import <nixpkgs> { system = "x86_64-linux"; };
# currentTime to make sure we always do a new build for testing's sake
runCommand "foo" { inherit (builtins) currentTime; } "touch $out"

One thing that stands out to me is that you don’t have anything in extra-sandbox-paths in nix.conf. For qemu-based emulation, I have this:

extra-sandbox-paths = /run/binfmt /nix/store/c9w9ixlgqxq0msss9nqwjq9dkd5xvczc-qemu-aarch64-binfmt-P-x86_64-unknown-linux-musl

But, it would surprise me if this was your issue too. Even without these lines, extra-platforms should get you to the point that it attempts to build a derivation, and then it should fail with an exec format error or something like that in the build log. But for you, it’s bailing out before that.

runeksvendsen · January 16, 2024, 4:19pm

Good catch. No, I can’t apparently:

$ cat test.nix 
with import <nixpkgs> { system = "x86_64-linux"; };
# currentTime to make sure we always do a new build for testing's sake
runCommand "foo" { inherit (builtins) currentTime; } "touch $out"

$ NIX_PATH="nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-23.11.tar.gz" nix-build test.nix 
this derivation will be built:
  /nix/store/k16vv2qhxhfhwwrlafbkpk0w8mq1nhym-foo.drv
error: a 'x86_64-linux' with features {} is required to build '/nix/store/k16vv2qhxhfhwwrlafbkpk0w8mq1nhym-foo.drv', but I am a 'aarch64-linux' with features {benchmark, big-parallel, nixos-test, uid-range}

I thought I was able to because the below command works, but I guess it just fetched from a cache

$ NIX_PATH="nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-23.11.tar.gz" $(nix-build -A nix '<nixpkgs>' --argstr system "x86_64-linux")/bin/nix-build --version
nix-build (Nix) 2.17.1
$ NIX_PATH="nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-23.11.tar.gz" file $(nix-build -A nix '<nixpkgs>' --argstr system "x86_64-linux")/bin/nix
/nix/store/frs3bna4fwzacqfi7mi3zbcryx9pgwi4-nix-2.17.1/bin/nix: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/ld-linux-x86-64.so.2, BuildID[sha1]=64bbf66f1a4427d2a36cf48a97635f189d39e582, for GNU/Linux 3.10.0, not stripped

Thanks for the correction. This is how I thought Nix worked, but I was confused for a moment. I will close the haskell.nix GitHub ticket.

runeksvendsen · January 16, 2024, 4:33pm

Doh! I found the mistake.

My nix.conf was badly formatted: it was using Nix syntax instead of whatever format is used by nix.conf.

I replaced

extra-platforms = [ "x86_64-linux" ]

with

extra-platforms = x86_64-linux

and now the error is gone.