I would be really interested in how big that cache would be (without sources and with sources).
The sources are also just derivations and as such they are also cached. One possible starting point could be to write a nix functions that walks over all derivations/attributes in the package set and and extracts src.outPath from each of them. You can then create a list and feed that to your nix copy solution above. However, this might miss some fetch* statements that are not directly placed as src attributes.