In addition, you can totally use multiple channels and just use a non-small channel for whatever packages you depend on that take longer to build and you don’t want to build downstream for some reason. There can still be CVEs that affect lower level libraries where this doesn’t help, but someone ultimately has to run those builds, and upstream can only prioritize so much.
Even without non-small channels, for your typical headless server, the number of packages that need to be built downstream in addition to what is in the *-small channels should be quite doable even on a small build host.
If nothing else, deliberately keeping that number small is probably a good exercise in figuring out what your system closure contents actually are - I realized today that I depend on two different postgres versions, for example.