The libGL library is essentially an interface to the hardware. At least on nvidia, the version of libGL is tied to the version of the nvidia driver kernel module, so application A and application B cannot use different versions.
If you linked the nvidia libGL to every binary directly, you would achieve purity, but you would need to rebuild the world on every nvidia driver update. Packages built this way wouldn’t work on any machine that isn’t using the exact version of the nvidia kernel module it was built with, so it wouldn’t really be portable even across nvidia machines, not to mention that it obviously wouldn’t work on any other brand of GPU.
To me, a very important benefit of Nix is that if a package works on one machine, it works on (basically) all machines. You would sacrifice this guarantee in practice by introducing a seperate package for every possible GPU driver version for all graphical apps.