Substituters on a remote builder for a flake causes build to fail with "no such file"

Hi all!
I’m trying to build a flake w/ a remote builder, while having the remote builder pull from a binary cache.
When i run my nix build command, i get the following error.

Any thoughts ? Someone mentioned that Yep, the issue only happens if you attempt to copy a fixed output derivation from a substituter into a chroot store. It's kind of rare but very unpleasant for people who try to do this thing that should work during installation. here https://github.com/NixOS/nixpkgs/issues/126141 - is that what’s happening here?

To be specific:
I run nix run nixpkgs#darwin.linux-builder and leave it open.
I configure nix.conf on the host system (an M1 Mac, darwin-aarch64) to use the remote builder i just started, and to consider my user a trusted substituter, etc.

Then, I attempt to build a linux-aarch64 derivation like this:
nix build 'github:arilotter/dotfiles#nixosConfigurations.kronos.config.system.build.sdImage'
This derivation is supposed to build a linux-aarch64 SD card image that I can boot on a linux-aarch64 system. It uses a remote substituter so I don’t have to build the entire kernel.
I get no problems or warnings about substituters at all after the initial allowance.

This is the error I hit:

error: build of '/nix/store/an92k3fp3g3spzmq6k26fwfywpp09sjd-X-Restart-Triggers.drv' on 'ssh://builder@linux-builder' failed: error: getting status of '/nix/store/7yb9ll5s5byybq4mrgwss07yslvbqpgj-openssh-9.4p1': No such file or directory
error: builder for '/nix/store/an92k3fp3g3spzmq6k26fwfywpp09sjd-X-Restart-Triggers.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/3n6mn6m0kli3df57qqqwdggq72h6qz6j-unit-sshd.service.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qd7nlrdyhp4w9gz8895220m4gpy1a2m8-system-units.drv' failed to build
error: 1 dependencies of derivation '/nix/store/s4mw8xsy6p6nm5sji2scha7b8hm7fm83-etc.drv' failed to build
error: 1 dependencies of derivation '/nix/store/9lfp5xcfdy57yd6da0zh5m05y13z8a9y-nixos-system-kronos-23.11.20230819.d680ded.drv' failed to build
error: 1 dependencies of derivation '/nix/store/85rgnj02i5153pjcfsyfasaimc0cwds7-ext4-fs.img.zst.drv' failed to build
error: 1 dependencies of derivation '/nix/store/0dpxhwl3y1gi4vcbq48khcbfpyfnd333-nixos-sd-image-23.11.20230819.d680ded-aarch64-linux.img.drv' failed to build

Any thoughts?