My PC lag somewhat even with --options cores 8 while rebuilding. Eventually I found that almost all of the time building is for flakes without caches specified. Also, setting flake-name.inputs.nixpkgs.follows = "nixpkgs"; makes flakes use the same nixpkgs version (source).
Anything else I can do minimize rebuild or compile times?
Many projects are also directly packaged in nixpkgs, it’s worth checking if you can just use those packages instead of third flakes, so you can make use of the hydra cache (and, well, the actual distro itself).
Depending on flakes instead of packages is only really worthwhile if you need to use in-dev versions of things for one reason or another. You can use the community caches if you really want to run in-dev software permanently, but you’ll end up with a lot of trusted substituters which is a bit of a risk, and you really ought not to run that much bleeding edge code.
Some flakes are of course used to publish nix code dependencies (i.e. nixosModules or lib), but in those cases you don’t need to build anything, so I suspect you’re just depending on a bunch of flakes because you didn’t realize you don’t have to.