Build failed due to lack of signature by trusted key in distributed build

I have an aarch64-linux system with nixos-23.11 that I’ve setup to run builds remotely on a more capable machine. I’ve found however, that it sometimes fails with an error like this:

❯ nixos-rebuild switch --flake git+https://git.icanttype.org/cdombroski/nix-configs.git -v --option tarball-ttl 0
$ nix --extra-experimental-features nix-command flakes build --out-link /tmp/nixos-rebuild.ekyvQW/nixos-rebuild git+https://git.icanttype.org/cdombroski/nix-configs.git#nixosConfigurations."orangepihole".config.system.build.nixos-rebuild -v --option tarball-ttl 0
$ exec /nix/store/yzi4xwkw1zg4mmib5m3cvygwdjjvbsk8-nixos-rebuild/bin/nixos-rebuild switch --flake git+https://git.icanttype.org/cdombroski/nix-configs.git -v --option tarball-ttl 0
building the system configuration...
Building in flake mode.
$ nix --extra-experimental-features nix-command flakes build git+https://git.icanttype.org/cdombroski/nix-configs.git#nixosConfigurations."orangepihole".config.system.build.toplevel -v --option tarball-ttl 0 --out-link /tmp/nixos-rebuild.FpnKVS/result
these 10 derivations will be built:
  /nix/store/28nkxk5gyj7x088f0wcq1d95nhl4prf9-unit-script-adblock-post-stop.drv
  /nix/store/4scgx33rkq9ww3j47ls2xf8g2n04625w-etc-nix-registry.json.drv
  /nix/store/9ixv6j6kbd4z2gwhg2zlf0lzf27h7x3j-boot.json.drv
  /nix/store/ajsf8z9j3rm9lfl104prfdgvqy9n7sqr-unit-console-getty.service-disabled.drv
  /nix/store/q22b82pb1mi4382rr33vs8vrmwj2hmcs-unit-adblock.service.drv
  /nix/store/dqdsd6661vpvdb3gphg4p45kdqrx16la-system-units.drv
  /nix/store/g9bdq5fb7hrd9bhqm09fijvd73r3vgvs-ensure-all-wrappers-paths-exist.drv
  /nix/store/hq1snppk5mm4y1979mzkjrm1cf5whprr-stage-2-init.sh.drv
  /nix/store/iwwzkgy84rdp4gi9bvfbc0njjqzjvjsb-etc.drv
  /nix/store/vk1kr3qkxphy3amk4pjzqly7nmq9ypz6-nixos-system-orangepihole-23.11.20240504.651b470.drv
building '/nix/store/9ixv6j6kbd4z2gwhg2zlf0lzf27h7x3j-boot.json.drv' on 'ssh-ng://nixremote@zeus'...
copying 32 paths...
copying path '/nix/store/19dgdkwhm7aff175y4brmlfblwmsl3rh-linux-6.6.30-modules-shrunk' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/50bi65cl0gxlwwccbp17y1a09133h8d5-xz-5.4.4' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/5qfd12srfil08lq8lmnz869h439vnp5w-xz-5.4.4-bin' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/7rrsh4pinzs5g74l3gafqmpfd345csc4-gcc-12.3.0-lib' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/765ldzh0228ajs327gbifi2fx2r4ld2m-gmp-with-cxx-6.3.0' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/xb53x2qrspfwkjq1j6lnpyi9f7xayvrx-bzip2-1.0.8' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/869sfkyc4mmhrjsaqji4g801y7si1imd-bzip2-1.0.8-bin' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/a967sandalpxwd978c02j86gkcwlnwwv-zstd-1.5.5' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/aiir1sygbh2d85gq6vmwynrdsp8pww4s-gnugrep-3.11' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/c97dqcrswap5fi9h58dx8z3psdj6f7jb-bash-5.2-p15' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/95r5pam8gsl5z3mxwxcfbskygnggwigl-zstd-1.5.5-bin' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/9dx4xrylvfvflh02v2rzpli36y20lv7r-gawk-5.2.2' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/z5msfwc267gkq4bmh6ji41ynrmrdr3im-attr-2.5.1' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/bpylcld47kwzpca0pcav9zj17rbv4c71-acl-2.3.1' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/skz7l7ypv1qa4ki1gypyci80cc97nlac-openssl-3.0.13' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/yssq83h9rdhfkisrwyimq4kssgmsw3ha-libxml2-2.11.7' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/w38kl66fkplfg91agmzjm15chshrd2p9-libarchive-3.7.2-lib' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/9hrfxh7cjfqv9kh540x2b6v43xvky5dw-libarchive-3.7.2' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/bjkm8zxgp0phw7p9yy2cp0jg7p5r6v02-ed-1.19' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/cyf8jnmaw8fmsfii36fcvjqqi1j2i3av-coreutils-9.3' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/dpiw6z4851467dcs625x7jqilmcb85vd-diffutils-3.10' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/k6mijqf2gs8q57m2q0dmkk4b4wrw5bh1-initrd-linux-6.6.30' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/l7y22r3lxi5kcxlmdcp4vyzl6q2wwfyj-patchelf-0.15.0' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/lzrmw56k0l23p052qadflh61ljpm3zx9-patch-2.7.6' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/n7p4gw0dkkr0cizhinpzccilpvx2v4q5-gzip-1.13' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/qfnbgf49igyw0rfygjv36ngr3984lqx0-file-5.45' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/vs8v7xv5zfhnpffzyk9gygqgp9j6yvd4-findutils-4.9.0' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/wqq7x3v5ppp4zi9z9xqyvmxk8llhzqm7-gnumake-4.4.1' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/yxa12hxr9vzyp0ssxq270hhrqjwjqxip-gnutar-1.35' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/mibd6zimf0rg63k6y8jf7lz2z4k21z4j-stdenv-linux' to 'ssh-ng://nixremote@zeus'...
copying path '/nix/store/w0rignkgyk4dr9nmadj21y82j8ll2w47-append-initrd-secrets' to 'ssh-ng://nixremote@zeus'...
error: cannot add path '/nix/store/19dgdkwhm7aff175y4brmlfblwmsl3rh-linux-6.6.30-modules-shrunk' because it lacks a signature by a trusted key
error: builder for '/nix/store/9ixv6j6kbd4z2gwhg2zlf0lzf27h7x3j-boot.json.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/vk1kr3qkxphy3amk4pjzqly7nmq9ypz6-nixos-system-orangepihole-23.11.20240504.651b470.drv' failed to build

This is the nix.conf file on the local system:

❯ cat /etc/nix/nix.conf 
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
allowed-users = *
auto-optimise-store = false
cores = 0
experimental-features = nix-command flakes
extra-sandbox-paths = 
flake-registry = 
max-jobs = 0
nix-path = deploy-rs=flake:deploy-rs flake-utils=flake:flake-utils home-manager=flake:home-manager impermanence=flake:impermanence nixpkgs=flake:nixpkgs self=flake:self sops-nix=flake:sops-nix
require-sigs = true
sandbox = true
sandbox-fallback = false
substituters = https://cache.nixos.org/
system-features = nixos-test benchmark big-parallel kvm gccarch-armv8-a
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
trusted-substituters = 
trusted-users = root

I’ve not set the remote up as a substituter since it didn’t seem like it was required and I only have one remote builder with the same internet connection as the local system. nix log says there’s no log available for the failed job

I am getting this sort of problem with many different systems not just aarch64 at them moment using colmena

on a related note this page indicates that nix store info is a command, was this recently changed?

nix store info --store ssh://mac       

error: 'info' is not a recognised command
Try 'nix --help' for more information.

so I am uncertain how to check other than to just try ssh’ing to/from the machines (which works fine).