To use --system you run into a bit of a chicken or egg problem with the disk partitioning. For a configuration to be usable with the --system option it needs to exactly fit that machine and that means it needs to be built with the info normally stored in /etc/nixos/hardware-configuration.nix which is generated by nixos-generate-config in a normal installation. There are ways to generalize the partition info by using labels instead of UUID for disks and if you are going to install many machines on identical hardware it might be worth your while. But there is also a different approach.
Have you considered using your build server as a remote builder instead of just a cache? With a setup like that the machine you want to install on just needs SSH access to the build server and some configuration and then it will offload either all building or just some of the building to the build server.