Ah, the above commands for creating symlinks are wrong. It should be
sudo mkdir -p /run/opengl-driver/lib
sudo ln -s /usr/lib/x86_64-linux-gnu/libcuda* /run/opengl-driver/lib/
sudo ln -s /usr/lib/x86_64-linux-gnu/libnvidia-* /run/opengl-driver/lib/
Also, on some distros, e.g., Ubuntu, /run/ is mounted to tmpfs so it might be a good idea to recreate the symlinks on system startup (also because the symlinks might become broken after a gpu driver update). One could do something along the lines of:
# A script that creates symlinks
sudo tee /usr/local/bin/nix-symlink-gpu <<EOF
#! /bin/sh
mkdir -p /run/opengl-driver/lib/
rm /run/opengl-driver/lib/*
ln -sf /usr/lib/x86_64-linux-gnu/libcuda* /run/opengl-driver/lib/
ln -sf /usr/lib/x86_64-linux-gnu/libnvidia-* /run/opengl-driver/lib/
EOF
sudo chmod a+x /usr/local/bin/nix-symlink-gpu
# A systemd unit that runs the script at startup
sudo tee /etc/systemd/system/nix-symlink-gpu.service <<EOF
[Unit]
Description=Symlink GPU drivers for use by software packaged with Nix
[Service]
Type=oneshot
ExecStart=/usr/local/bin/nix-symlink-gpu
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl enable --now nix-symlink-gpu.service