Ollama ggml-cuda error

Help
1

when running various ollama run MODEL commands the following error prevents models like QwQ from running - this occurs on ollama and ollama-rocm, the error identical with various models on each package:

writing manifest 
success 
Error: 500 Internal Server Error: llama runner process has terminated: ROCm error: invalid device function
  current device: 0, in function ggml_cuda_compute_forward at /build/source/ml/backend/ggml/ggml/src/ggml-cuda/ggml-cuda.cu:2672
  err
/build/source/ml/backend/ggml/ggml/src/ggml-cuda/ggml-cuda.cu:88: ROCm error

It doesn’t appear to be related to the model size as I’ve attempted to install other models and gotten a different error claiming the ram available was insufficient.
Is this common behavior due to nixpkgs misconfiguration or a problem with my amd cpu/amd gpu NixOS system?

2

Have any of you gotten QwQ running in a NixOS setup - this might indicate the level of the error’s source.

3

I can’t diagnose the specific issue, but it is notable that you’re seeing a cuda issue with a rocm card.

If you’ve installed ollama with NixOS, what’s the value of your services.ollama.acceleration? Are you using the package or the module? Are there any models that do run successfully currently?

2 Likes
4

I just ran nix-shell -p ollama-rocm.