Looking for the guide to setup distributed builds

Help
1

Hello everyone,

I’ve set up a build machine in a macos aarch64-darwin, and it seems to work correctly when using root:

› sudo su                                                                                                                    
sh-3.2# nix build --impure --expr '(with import <nixpkgs> { system = "x86_64-linux"; }; runCommand "foo5" {} "uname > $out")'
sh-3.2# cat result 
Linux
sh-3.2# nix store info --store ssh://yzx9-ws 
Store URL: ssh://yzx9-ws

However, I’m unable to build with an unprivileged user:

› nix store info --store ssh://yzx9-ws                                                                                 
Store URL: ssh://yzx9-ws
› nix build --impure --expr '(with import <nixpkgs> { system = "x86_64-linux"; }; runCommand "foo6" {} "uname > $out")'
cannot build on 'ssh-ng://yzx9-ws': error: failed to start SSH connection to 'yzx9-ws'
Failed to find a machine for remote build!
derivation: yya9pb0hjd09cw9k5v0g2c1gpmx79f92-fooA.drv
required (system, features): (x86_64-linux, [])
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([x86_64-linux], 16, [benchmark, big-parallel, kvm, nixos-test], [])
error: a 'x86_64-linux' with features {} is required to build '/nix/store/yya9pb0hjd09cw9k5v0g2c1gpmx79f92-fooA.drv', but I am a 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}

Any idea? Any helps would be appreciated!

Additional information

I have restart the nix-daemon:

sudo launchctl stop org.nixos.nix-daemon
sudo launchctl start org.nixos.nix-daemon

Here is my configuration, I am using latest nixpkgs-unstable with nix-darwin:

{
  nix.distributedBuilds = true;

  age.secrets = {
    id-auth = {
      file = ../../secrets/id-git_root.age;
      path = "/var/root/.ssh/id_auth";
      owner = "root";
      group = "wheel";
      mode = "400";
    };

    ssh-config = {
      file = ../../secrets/ssh-config.age;
      path = "/var/root/.ssh/config";
      owner = "root";
      group = "wheel";
      mode = "400";
    };
  };

  nix.buildMachines = [
    {
      hostName = "yzx9-ws";
      systems = [ "x86_64-linux" ];
      protocol = "ssh-ng";
      maxJobs = 16;
      speedFactor = 2;
      supportedFeatures = [
        "nixos-test"
        "benchmark"
        "big-parallel"
        "kvm"
      ];
      mandatoryFeatures = [ ];
    }
}
2

Still looking for your help!

3

I would check the last two steps from here, that is if you have the keys added to authorized keys and the user to trusted users.

1 Like
4

I finally found the solution. The key idea is to make sure you can connect via SSH and excepted key! Add -vvvvv and reading the authorization process can be helpful.

The store setup can be more complex than expected. I realized I made several mistakes, including:

  • File permission → ~/.ssh/ should be 700, keys should be 600
  • Using a GPG agent → fixed by running unset SSH_AUTH_SOCK
  • Wrong secret path in nix config → fix
  • Using a relative IdentityFile in ssh_config, which caused the private key to not be found → changed to an absolute path
  • Having more than 6 keys (the typical server MaxTry value), which caused the server to reject further key authorization requests → added IdentitiesOnly yes to the config

Especially thanks to @jaen , the link was really helpful