My understanding is that the nix-daemon does the actual work of SSHing to the remote builder.
How do I manage this when using a gpg-agent-protected SSH key?
$ nix build -f default.nix --builders 'ssh://colemickens@aarch64.nixos.community aarch64-linux'
warning: dumping very large path (> 256 MiB); this may run out of memory
cannot build on 'ssh://colemickens@aarch64.nixos.community': cannot connect to 'colemickens@aarch64.nixos.community': colemickens@aarch64.nixos.community: Permission denied (publickey).