Some packages will pass specific cpu extensions to package builds. To do this generically, I demonstrated this in Tensorflow slower as NixOS native than inside a Docker container - #4 by jonringer. Essentially this will set the following flags on targetPlatform and hostPlatform.
sse3Support = featureSupport "sse3";
ssse3Support = featureSupport "ssse3";
sse4_1Support = featureSupport "sse4_1";
sse4_2Support = featureSupport "sse4_2";
sse4_aSupport = featureSupport "sse4a";
avxSupport = featureSupport "avx";
avx2Support = featureSupport "avx2";
avx512Support = featureSupport "avx512";
aesSupport = featureSupport "aes";
fmaSupport = featureSupport "fma";
fma4Support = featureSupport "fma4";
However, this will likely cause a specific machine to do a LOT of rebuilds. And not all packages will be “aware” of these extensions, and it’s not communicated to the build to enable the extensions for a given build.
Back to the original question. Some of these extensions may help a lot with video decoding, which may explain some of the increased cpu usage.
Another idea is to make videoDriver builds a little “impure”, since installation is already impure. Having the drivers compiled with as many optimizations as possible would probably help with compute usage.