The 6.10 kernel will not work. You must use an earlier version like boot.kernelPackages = pkgs.linuxPackages_6_9;
For an explanation, see Unable to build nix due to nvidia drivers due or kernel 6.10(?) - #5 by TLATER
It’s worth noting that this isn’t fully open source and will probably not solve the issue according to the Nvidia forum. The fully open source driver is NVK and using it will probably solve this, but you won’t be able to use CUDA anymore.
Note: Under 555.58.02, it hasn’t been happening to me as much, lately, but it still occurs nonetheless.