Yes, you can fetch directly from SWH, but then it isn’t clear how to do so as a substitutor without more changes to nix. Conversely, you can use the IPFS gateway without having IPFS installed — that has improved since we did the Nix IFPS work so we could add support for that.
SWH is supposed to be an archive of last resort to potentially tons of people flooding their API with similar requests just waste’s the non-profit’s bandwidth. IPFS however can act as a CDN.