SSH works, but remote building says it can't connect

Help
1

Everything seems to line up to be able to build remotely:

  • config.nix.buildMachines:
    [
  {
    hostName = "hostname";
    mandatoryFeatures = [ ];
    maxJobs = 1;
    protocol = "ssh-ng";
    publicHostKey = null;
    speedFactor = 1;
    sshKey = "/home/username/.ssh/id_ed25519";
    sshUser = "username";
    supportedFeatures = [
      "benchmark"
      "big-parallel"
      "flakes"
      "kvm"
      "nix-command"
      "nixos-test"
    ];
    system = "x86_64-linux";
    systems = [ ];
  }
]
  • ssh username@hostname succeeds
  • nix store info --store ssh-ng://username@hostname succeeds
  • /etc/nix/nix.conf on the builder contains allowed-users = * and trusted-users = root @wheel
  • username is part of the wheel group

But when running deploy, it prints

cannot build on ‘ssh-ng://username@hostname’: error: failed to start SSH connection to ‘username@hostname’

What else do I need to do?

2 Likes
2

I don’t remember the exact details anymore, but after fixing it I made a quick write-up of the findings: Remote Nix build footguns.

3

Adding users to trusted-users is a bad idea, as they can modify things like substituters.

https://nix.dev/manual/nix/2.32/command-ref/conf-file.html#conf-trusted-users

Warning

Adding a user to trusted-users is essentially equivalent to giving that user root access to the system. For example, the user can access or replace store path contents that are critical for system security.

Use allowed-users instead.

4 Likes
4

Thank you for the tip! In this case I already have root on both machines, so trust isn’t an issue, but I’ll amend the article.

5

In my testing, adding users to trusted-users is required (even with ssh-ng) for any build where some inputs are built locally (e.g., because preferLocalBuild is set on the derivation).

Unfortunately, this means remote builders are only useful when either (a) you have a separate remote builder VM/store per client machine or (b) you’re fine with root access to any one machine implying root access to all machines that use the builder.