Why does nvidia-smi report the wrong CUDA version?

Help
1

Ok, so I think I’ve got the right NVIDIA drivers now and nvidia-smi correctly identifies my GPU:

❯ ns -p cudatoolkit

[nix-shell:~]$ nvidia-smi
Sat Sep  4 01:41:24 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.38       Driver Version: 455.38       CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   45C    P0    26W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

This is great. Except that it says CUDA version 11.1… This is odd to me since the current nixpkgs cudatoolkit package points to version 10. I confirmed this by checking my PATH:

[nix-shell:~]$ echo $PATH | tr ":" "\n" | grep cuda
/nix/store/v73iypl6icngq31ihl3jvhiw7p5z3g3f-cudatoolkit-10.2.89/bin

So is this right? Has nix loaded up CUDA version 10.2 or 11.1? Why the discrepancy?

2

I think that is the maximum supported cuda toolkit. The driver ships separately from the toolkit so nvidia-smi can’t possibly know what toolkit you’re actually using

1 Like
3

Ah, ok. Thanks for explaining! That seems like confusing UI on NVIDIA’s part, but happy to hear it’s not a bug.

4

Oh yeah, and I just remembered that the 11.x series is supposed to be forward-compatible as well. So I think you can actually use cuda toolkit 11.2 as well without needing to upgrade your driver.

2 Likes