Version mismatch between NVIDIA driver and CUDA packages?

Help
1

I want to preface this topic by saying I’m very new to NixOS; despite daily driving it for 3 months now, after I powered through the initial setup, I haven’t had much time to really explore, deep dive and learn.

Now, what happened is that after updating my NixOS to version 25.11, I noticed it also updated the NVIDIA driver to version 580.119.02 (I don’t recall which version it was before, but I assume it was one that supported CUDA v12.8). Then, when I tried running koboldcpp using CUDA, I got this error: OSError: /nix/store/skbrsi28jc0rgpq5jlgsikd0la4rli10-koboldcpp-1.103/bin/koboldcpp_cublas.so: undefined symbol: cuMemCreate.

Since I had just updated my flake’s nixpkgs input from 25.05 to 25.11, I immediately thought it was a problem with a package. Eventually I checked the versions of the packages in cudaPackages, and saw they’re still 12.8, even in the unstable branch, while this new NVIDIA driver’s version has CUDA 13.

I confirmed this discrepancy by running nvidia-smi (which told me CUDA 13) and nvcc -V (which told me CUDA 12.8). I then asked ChatGPT (I know, not the most reliable source of information, but I couldn’t find anything about this anywhere), and it also confirmed this was the issue. The only things that changed by updating to 25.11 were the NVIDIA driver and the koboldcpp version, so one of them has to be the culprit.

My question is: is that really the problem? If so, has the CUDA team talked about updating the packages to version 13 anywhere (I was unable to find anything related to that as well)? If not, any help would be much appreciated!

And, just to clarify: I know this is easily solvable by rolling back, and I’m not trying to pressure the CUDA team or anything like that. I play around with LLMs as a hobby, so this isn’t even that important or world-ending, I just want to learn and understand why it started failing!

2

Hi. Did you reboot? There is a known userspace driver-kernel module mismatch issue that we’ve been too lazy to special-case: libcuda.so: driver mismatch on nixos-rebuild switch · Issue #255070 · NixOS/nixpkgs · GitHub

3

I did, yes. Multiple times, in fact, as this started happening a couple days ago. I might’ve stumbled upon that GitHub issue while looking for a solution before making this topic, but I wasn’t able to fully understand the situation; I shamefully don’t know much about drivers and such.

4

Oh wait you’re running past the version string check, that’s something else. But cuMemCreate does sound like it’s a driver and not a runtime function :thinking:. Could you open an issue under NixOS/nixpkgs please, and tag cuda?

P.S. Also do attach logs with LD_DEBUG=libs under a spoiler or in a gist, ty

5

I mean, I could open an issue, but honestly I don’t even know what the problem is :sweat_smile:. What I said in this topic was all speculation, and I don’t know for sure if that’s correct or not. I don’t think I know enough and have enough information to open an useful issue…

6

It’s fine, you probably have more information already than most of the issues in nixpkgs, and you’ll get a different audience on the github tracker.

1 Like
7

Alright, then. I’ll do it during my lunch break in a few minutes. Thanks.

(Edit: opened, linking here for reference https://github.com/NixOS/nixpkgs/issues/472447)

1 Like
8

A related thread:

Can we solve the nvidia situation?

https://discourse.nixos.org/t/can-we-solve-the-nvidia-situation/73198/2

1 Like