Use ssh substituters in system wide flake

Help
1

I want to be able to nixos-rebuild on a machine, then have other machines rebuild, pointing them to the first machine to pull any builds already done (which should be around 99% for all the machines in my house).
I’m specifying all of my machines to use identical refs in the input:

inputs.nixpkgs = {
    type  = "github";
    owner = "NixOS";
    repo  = "nixpkgs";
    # nixos-23.05 branch
    ref   = "5d017a8822e0907fb96f7700a319f9fe2434de02"; # Sep 17, 2023 12:56am PDT
  };

I specified

            nix.settings.extraSubstituters = "ssh://senorsmile@machine1";

But everything just seems to pull from the public servers and eventually gave me an error that this extraSubstituters is not valid.
I did already verify that the public ssh rsa key for my client node is on the server and can ssh from the root account on the client to the “server”.

How would would you set this up?