Problem with remote building on different architecture

Help
1

I’m trying to set up a Raspberry Pi v4b (running Ubuntu 20.04) as a remote builder for my local x86 machine (also Ubuntu 20.04) and I’m getting a rather confusing error message when trying to build a package that targets aarch64-linux:

error: build of '/nix/store/xf0nqqhaq68jqnfh132f7w3g53pmrlcm-pigpio-78.drv' on 'ssh://build-rpi' failed: a 'x86_64-linux' with features {} is required to build '/nix/store/xf0nqqhaq68jqnfh132f7w3g53pmrlcm-pigpio-78.drv', but I am a 'aarch64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

The package in question: https://github.com/drewrisinger/nur-packages/blob/10649b8c8be82f990d21051410f10226df750f08/pkgs/raspberrypi/pigpio/default.nix

Output of nix-shell -p nix-info --run "nix-info -m" (host):

 - system: `"x86_64-linux"`
 - host os: `Linux 5.8.0-41-generic, Ubuntu, 20.04.2 LTS (Focal Fossa)`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.7`
 - channels(brad): `""`
 - channels(root): `"<other channels>, nixpkgs-20.03.3324.929768261a3"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

(not sure why it says ‘no’ for ‘multi-user’ - it actually is a multi-user installation)
rpi:

 - system: `"aarch64-linux"`
 - host os: `Linux 5.4.0-1028-raspi, Ubuntu, 20.04.2 LTS (Focal Fossa)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.10`
 - channels(root): `"nixpkgs-20.03.3324.929768261a3"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

nix.conf (host):

substituters = https://cache.nixos.org <other substituters>
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= <other keys>
cores = 0
build-users-group = nixbld
builders = ssh://build-rpi
#builders-use-substitutes = true  # tried this both ways, didn't seem to make a difference as far as the error

rpi:

system = aarch64-linux
extra-platforms = armv7l-linux  # I don't even know if this is true, basically just added it to see what would happen
build-users-group = nixbld
max-jobs = 1
cores = 0
trusted-users = root build-rpi

And I also have allowUnsupportedSystem = true; in my local user’s config.nix (this was actually a key thing in getting as far as I have).

It took me quite a few attempts to get nix-build to even try to use the rpi, so I probably have some unnecessary stuff in my configuration - please ignore that. The extra channels and substituters can be ignored too.

As far as why I want to do distributed builds in this way - this is just a local test before I attempt to use the rpi as a builder for a hydra server I’m running.

Any help would be appreciated, TIA.

2

This doc page is quite good, and covers this in detail: Remote Builds

The problem is that you didn’t tell your x86_64-linux machine that ssh://build-rpi is an aarch64-linux.

So, what’s happening is that you’re evaluating an x86_64-linux build, then it thinks it can build on build-rpi because it assumes it’s x86_64-linux, and then when the build starts on build-rpi, it complains because it was asked to build an x86_64-linux derivation, but it cannot, since it is a aarch64-linux.

builders = ssh://build-rpi aarch64-linux

Also,

I’d probably go ahead and undo that, that shouldn’t be necessary here, from what I can tell. In fact, that didn’t actually help. Instead, it forced nix to try to build pigpio on x86_64-linux when the package declares itself as not buildable there. So it sort of just made it even less likely to succeed…

If you had not done this, instead your error message would be telling you that pigpio refuses to evaluate for x86_64-linux, and then whenever you fixed that, it would’ve correctly told you that it didn’t know about any available aarch64-linux builders.

Which brings us to… when you build this, you’re going to need to specify that the architecture you want to build for is aarch64-linux so that the nix scheduler will try to schedule it on the build-rpi (after you fix the config).

nix-build --argstr system aarch64-linux ...

1 Like
3

Yeah, I’ve been following that doc page, as well as this one: Distributed build - NixOS Wiki (and I’ve taken a look at the things it links to at the bottom as well).

I’m glad to hear that - I was hesitant to do it in the first place (since it didn’t make sense to me that I should need to), but found myself out of ideas, so I gave it a shot. And after that was the first time that nix-build seemed to even try to use the rpi, so I took that as a good sign.

Yep, you are spot on with that assessment. Unfortunately I’ve made your suggested changes and I’m still getting that error. Here’s what I’ve tried:

builders = ssh://build-rpi aarch64-linux in nix.conf along with:

  • nix-build --argstr system aarch64-linux ...
  • nix-build --argstr system aarch64-linux --builders 'ssh://build-rpi aarch64-linux' ... (after adding my user to trusted-users)
  • nix-build --argstr system aarch64-linux --builders 'ssh://build-rpi aarch64-linux' --max-jobs 0 ...

as well as builders = @/etc/nix/machines with all of the above nix-build commands, where my machines file contains ssh://build-rpi aarch64-linux <root ssh key location> 1 1.

I should also (maybe) add that I’m able to ping the rpi’s store from both my user as well as root.

4

Hm, okay, I seem to have found somewhat of a solution. For testing purposes I’ve modified the top-level default.nix from the NUR that contains these packages such that it only includes the raspberry pi packages:

{ pkgs ? import <nixpkgs> {  } }:

rec {
  # Raspberry Pi Packages
  raspberryPi = pkgs.recurseIntoAttrs {
    argonone-rpi4 = pkgs.callPackage ./pkgs/raspberrypi/argonone-rpi4 { inherit (python3Packages) rpi-gpio smbus2; };
    pigpio-c = pkgs.callPackage ./pkgs/raspberrypi/pigpio { };
    steamlink = pkgs.callPackage ./pkgs/raspberrypi/steamlink {};
    vc-log = pkgs.callPackage ./pkgs/raspberrypi/vc-log { };
  };

  python3Packages = pkgs.recurseIntoAttrs rec {
    # Raspberry Pi Packages
    colorzero = pkgs.python3Packages.callPackage ./pkgs/raspberrypi/colorzero { };
    gpiozero = pkgs.python3Packages.callPackage ./pkgs/raspberrypi/gpiozero {
      inherit colorzero pigpio-py rpi-gpio;
    };
    pigpio-py = pkgs.python3.pkgs.callPackage ./pkgs/raspberrypi/pigpio/python.nix { inherit (raspberryPi) pigpio-c; };
    rpi-gpio = pkgs.python3Packages.callPackage ./pkgs/raspberrypi/rpi-gpio { };
    rpi-gpio2 = pkgs.python3Packages.callPackage ./pkgs/raspberrypi/rpi-gpio2 { };
    smbus2 = pkgs.python3.pkgs.callPackage ./pkgs/python-modules/smbus2 { };
  };

}

With the file as-is, I get the “refusing to evaluate” error, but if I change the nixpkgs import to { pkgs ? import <nixpkgs> { system = "aarch64-linux"; } }:, then it works. But shouldn’t that have been taken care of by the --argstr ... part of the build command?

1 Like
5

I think argstr just passes it as a function arg to the thing being built with nix-build. So, if the thing you’re building wasn’t nixpkgs (which itself takes system as an arg) then you’re just at the mercy of the nix api of whatever you’re building.

Can you please give the full commands and errors, it would help be able to understand how you’re trying to build the package. The problem is still the same - you’re still calling this package with an x86_64-linux packages.

6

I find implicit things like NIX_PATH and <nixpkgs> to cause a lot of headaches for me and others. So much so that I’m going to give an example using an experimental feature called Flakes. Flakes is also a nice fit here since you’re importing nix from an external source, and flakes encourages and helps you to import them in a reliable, repeatable way via a generated flake.lock that records import revisions.

Note how it is very explicit about where the sources are coming from, how nixpkgs is being imported (for aarch64-linux.

flake.nix:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    drew.url = "github:drewrisinger/nur-packages";
    drew.flake = false;
  };

  outputs = inputs:
    let
      nixpkgs_ = import inputs.nixpkgs { system = "aarch64-linux"; };
      drew_ = import inputs.drew { pkgs = nixpkgs_; };
    in
      {
        pigpio = drew_.raspberryPi.pigpio-c;
      };
}

And then build it. When we build it, we tell Nix about our extra rpi4 builder(which is aarch64-linux), and we enable the experimental nix-command cli and the flakes feature.

cole@slynux ~/code/test
❯ nix build --experimental-features 'nix-command flakes' \
  ".#pigpio" --builders 'ssh://cole@rpifour1.ts.r10e.tech aarch64-linux'

cole@slynux ~/code/test
❯ file ./result/bin/pigpiod
./result/bin/pigpiod: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/vkfsgh5yp6x7c9id6w2vz2hz5dhfmm8n-glibc-2.32-35/lib/ld-linux-aarch64.so.1, for GNU/Linux 2.6.32, not stripped

cole@slynux ~/code/test
❯ tree result
result
├── bin
│   ├── pig2vcd
│   ├── pigpiod
│   └── pigs
├── include
│   ├── pigpiod_if2.h
│   ├── pigpiod_if.h
│   └── pigpio.h
├── lib
│   ├── cmake
│   │   └── pigpio
│   │       ├── pigpioConfig.cmake
│   │       ├── pigpioConfigVersion.cmake
│   │       ├── pigpioTargets.cmake
│   │       └── pigpioTargets-release.cmake
│   ├── libpigpiod_if2.so
│   ├── libpigpiod_if.so
│   └── libpigpio.so
└── share
    └── man
        ├── man1
        │   ├── pig2vcd.1.gz
        │   ├── pigpiod.1.gz
        │   └── pigs.1.gz
        └── man3
            ├── pigpio.3.gz
            ├── pigpiod_if2.3.gz
            └── pigpiod_if.3.gz

9 directories, 19 files