Ollama isn't using GPU ( AMD RX 7900 XTX )

I’m trying to use ollama with GPU acceleration on my system, following the documentation here using the ollama-rocm package. This is not working. The stock ollama package does not attempt to access the hardware through the ROCm libraries.

Here are some lines of interest from the strace, which make me think there’s some sort of linking bug in the package:

strace.txt:144:openat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/glibc-hwcaps/x86-64-v3/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace.txt:145:newfstatat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/glibc-hwcaps/x86-64-v3/", 0x7ffe3f18cc70, 0) = -1 ENOENT (No such file or directory)
strace.txt:146:openat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/glibc-hwcaps/x86-64-v2/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace.txt:147:newfstatat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/glibc-hwcaps/x86-64-v2/", 0x7ffe3f18cc70, 0) = -1 ENOENT (No such file or directory)
strace.txt:148:openat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/libresolv.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace.txt:149:newfstatat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/", {st_mode=S_IFDIR|0555, st_size=1094, ...}, 0) = 0
strace.txt:165:openat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace.txt:176:openat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace.txt:193:openat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/librt.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace.txt:204:openat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace.txt:215:openat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace.txt:227:openat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace.txt:239:openat(AT_FDCWD, "/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1 Like

are u using stable? working just fine for me atm

I am on stable, 24.11.

Try running with OLLAMA_DEBUG=1 to get more information.

No changes in observable behavior.

Ollama should produce more output when using OLLAMA_DEBUG, are you setting it on the server (ollama serve/start)?

[anadon@puttimon:~]$ ROCR_VISIBLE_DEVICES="1,2" OLLAMA_DEBUG=1 ollama run deepseek 
>>> This is a test.  Please only respond with 'acknowledged.'
<think>
Okay, so I just received this message that says, "Please only respond with 'acknowledged.'" Hmm, let me think about how to approach this.

First off, the user sent a test message and specified they want only an acknowledgment in response. So, the main point here is to follow their instructions precisely. They probably want 
to make sure that I can understand and execute simple commands, or maybe it's just for testing purposes.

I need to make sure that my reply strictly adheres to what they asked. That means no extra information, just "acknowledged." But wait, should I say anything else? Maybe I should also 
consider the context. Since it's a test, perhaps there's an expectation of accuracy and quick response.

Also, I wonder if there's any underlying purpose to this test. Are they checking my ability to follow directions, or is it part of a larger interaction? Either way, sticking to their 
exact request seems safe.

I should also think about the structure of the response. They want only "acknowledged," so no additional text or explanations. It needs to be concise and exactly as specified. Maybe I 
can just type that without any formatting, keeping it simple.

Is there a chance they might expect something else after this? Probably not, since the instruction is clear. But just in case, I'll make sure my reply doesn't include anything beyond 
"acknowledged."

Another thing to consider: sometimes in tests, they look for consistency and adherence to rules. So by strictly following their command, I'm demonstrating that I can be reliable.

I should also think about possible variations. What if they had said something slightly different? But no, the instruction is straightforward here. Just respond with "acknowledged."

Alright, so putting it all together, my response should simply be "acknowledged" without any extra words or explanations.
</think>

acknowledged

>>> Send a message (/? for help)

Set OLLAMA_DEBUG=1 on the server as that is actually what’s running the model. I’m sorry if that wasn’t clear

I’m running it directly on the terminal. I’m keeping it as simple as is possible.

ollama run does not run the model itself, it connects to a service that runs ollama serve, try setting services.ollama.environmentVariables.OLLAMA_DEBUG=1 and looking at the logs for ollama.service

[anadon@puttimon:~]$ ROCR_VISIBLE_DEVICES="0" OLLAMA_DEBUG=1 ollama serve
2025/01/31 16:10:52 routes.go:1153: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/anadon/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:0 http_proxy: https_proxy: no_proxy:]"
time=2025-01-31T16:10:52.725-05:00 level=INFO source=images.go:753 msg="total blobs: 0"
time=2025-01-31T16:10:52.725-05:00 level=INFO source=images.go:760 msg="total unused blobs removed: 0"
time=2025-01-31T16:10:52.725-05:00 level=INFO source=routes.go:1200 msg="Listening on 127.0.0.1:11434 (version 0.3.12)"
time=2025-01-31T16:10:52.725-05:00 level=INFO source=common.go:135 msg="extracting embedded files" dir=/tmp/ollama2247270498/runners
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/libggml.so.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/libllama.so.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu payload=linux/amd64/cpu/ollama_llama_server.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/libggml.so.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/libllama.so.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx payload=linux/amd64/cpu_avx/ollama_llama_server.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/libggml.so.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/libllama.so.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=cpu_avx2 payload=linux/amd64/cpu_avx2/ollama_llama_server.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm payload=linux/amd64/rocm/libggml.so.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm payload=linux/amd64/rocm/libllama.so.gz
time=2025-01-31T16:10:52.726-05:00 level=DEBUG source=common.go:168 msg=extracting runner=rocm payload=linux/amd64/rocm/ollama_llama_server.gz
time=2025-01-31T16:10:54.718-05:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama2247270498/runners/cpu/ollama_llama_server
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama2247270498/runners/cpu_avx/ollama_llama_server
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama2247270498/runners/cpu_avx2/ollama_llama_server
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=/tmp/ollama2247270498/runners/rocm/ollama_llama_server
time=2025-01-31T16:10:54.719-05:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu_avx2 rocm cpu cpu_avx]"
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=common.go:50 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2025-01-31T16:10:54.719-05:00 level=INFO source=gpu.go:199 msg="looking for compatible GPUs"
time=2025-01-31T16:10:54.719-05:00 level=WARN source=gpu.go:669 msg="unable to locate gpu dependency libraries"
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=gpu.go:86 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-01-31T16:10:54.719-05:00 level=WARN source=gpu.go:669 msg="unable to locate gpu dependency libraries"
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcuda.so*
time=2025-01-31T16:10:54.719-05:00 level=WARN source=gpu.go:669 msg="unable to locate gpu dependency libraries"
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[libcuda.so* /run/opengl-driver/lib/libcuda.so* /nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[]
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=gpu.go:468 msg="Searching for GPU library" name=libcudart.so*
time=2025-01-31T16:10:54.719-05:00 level=WARN source=gpu.go:669 msg="unable to locate gpu dependency libraries"
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=gpu.go:491 msg="gpu library search" globs="[libcudart.so* /run/opengl-driver/lib/libcudart.so* /nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2025-01-31T16:10:54.719-05:00 level=DEBUG source=gpu.go:525 msg="discovered GPU libraries" paths=[]
time=2025-01-31T16:10:54.719-05:00 level=WARN source=amd_linux.go:60 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2025-01-31T16:10:54.720-05:00 level=DEBUG source=amd_linux.go:103 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2025-01-31T16:10:54.720-05:00 level=DEBUG source=amd_linux.go:128 msg="detected CPU /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2025-01-31T16:10:54.720-05:00 level=DEBUG source=amd_linux.go:103 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/1/properties"
time=2025-01-31T16:10:54.720-05:00 level=DEBUG source=amd_linux.go:218 msg="mapping amdgpu to drm sysfs nodes" amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties vendor=4098 device=29772 unique_id=6932302669442746357
time=2025-01-31T16:10:54.720-05:00 level=DEBUG source=amd_linux.go:252 msg=matched amdgpu=/sys/class/kfd/kfd/topology/nodes/1/properties drm=/sys/class/drm/card0/device
time=2025-01-31T16:10:54.720-05:00 level=DEBUG source=amd_linux.go:284 msg="amdgpu memory" gpu=0 total="24.0 GiB"
time=2025-01-31T16:10:54.720-05:00 level=DEBUG source=amd_linux.go:285 msg="amdgpu memory" gpu=0 available="21.8 GiB"
time=2025-01-31T16:10:54.720-05:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /nix/store/rqz0nip8bsg37rcz9zdfw102mnpfyfms-ollama-0.3.12/lib/ollama"
time=2025-01-31T16:10:54.720-05:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/bin"
time=2025-01-31T16:10:54.730-05:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /run/opengl-driver/lib"
time=2025-01-31T16:10:54.730-05:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir /nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib"
time=2025-01-31T16:10:54.730-05:00 level=DEBUG source=amd_linux.go:337 msg="rocm supported GPUs" types=[]
time=2025-01-31T16:10:54.730-05:00 level=WARN source=amd_linux.go:341 msg="amdgpu is not supported" gpu=0 gpu_type=gfx1100 library=/nix/store/v8bm9phvgnqn3yacx25snf0k43sjf815-rocm-path/lib supported_types=[]
time=2025-01-31T16:10:54.730-05:00 level=WARN source=amd_linux.go:343 msg="See https://github.com/ollama/ollama/blob/39fd89308c0bbe26311db583cf9729f81ffa9a94/docs/gpu.md#overrides for HSA_OVERRIDE_GFX_VERSION usage"
time=2025-01-31T16:10:54.730-05:00 level=INFO source=amd_linux.go:361 msg="no compatible amdgpu devices detected"
time=2025-01-31T16:10:54.730-05:00 level=INFO source=gpu.go:347 msg="no compatible GPUs were discovered"
time=2025-01-31T16:10:54.730-05:00 level=INFO source=types.go:107 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="62.7 GiB" available="39.7 GiB"

This looks like a straight up bug to me. Now, I just can’t discern if it is in packaging, ollama, or the driver stack.

try passing this along with debug HSA_OVERRIDE_GFX_VERSION = "11.0.0";

see more ollama/docs/gpu.md at 475333d533e088538e5d77c0ba65ea60833b2c77 · ollama/ollama · GitHub

3 Likes

This worked. However, I would still like to pursue this a little further because it seems like it should be correctly detected by default. Can you help me with this? I’m still a little too green with the Nix ecosystem to be confident in directing my issue here.

I think you’d have to make ollama aware of where nix would put the things it (ollama) seeks ollama/discover/amd_linux.go at 475333d533e088538e5d77c0ba65ea60833b2c77 · ollama/ollama · GitHub

1 Like

For posterity, the bug report is here: Does not recognize AMD GPUs on NixOS · Issue #8748 · ollama/ollama · GitHub

2 Likes

Hi anadon,

Could you please share your config for the ollama with deepseek?
I have RX 7800XT and it wasn’t using my GPU, instead CPU is in use.

Thank you

Unfortunately for you, your GPU will never be supported. You needed a more up to date MI Instinct, Radeon Pro, or a 7900 series card.

I was able to get deepseek-r1:8b running on a RX 7600 using the overrides in this config.

The RX 7800 XT is supported on Windows, so I think it’s only a matter of time that it will be supported on Linux, if it’s not possible already.

See ollama/docs/gpu.md at ad22ace439eb3fab7230134e56bb6276a78347e4 · ollama/ollama · GitHub for more information.

am of the same opinion, the override should only be used as an exception, quite strange. Wasted alot of time on this previously to determine why ROCM was not detected by default

In Ollama discord, one good person(pixl404) shared his config in nixos with all AMD drivers and settings.

      nixpkgs.config.rocmSupport = true;
      services = {
        ollama = {
          enable = true;

          # home = "/var/lib/ollama";
          home = "/data/ollama";
          loadModels = [
            "deepseek-r1:14b"
            "deepseek-coder-v2"
          ];

          acceleration = "rocm";

          # https://github.com/NixOS/nixpkgs/issues/308206
          # https://rocm.docs.amd.com/en/latest/reference/gpu-arch-specs.html
          rocmOverrideGfx = "11.0.1"; # gfx1101

          host = "127.0.0.1";
          port = 11434;
          openFirewall = true;
        };

        nextjs-ollama-llm-ui = {
          enable = true;
          port = 8080;
          ollamaUrl = "http://127.0.0.1:11434";
        };
      };

      boot.kernelModules = [ "amdgpu" ];
      hardware.amdgpu.opencl.enable = true;
      hardware.graphics = {
        enable = true;
        enable32Bit = true;
        extraPackages = [ pkgs.rocmPackages.clr.icd ];
      };
      
      environment.systemPackages = with pkgs; [
        rocmPackages.clr.icd
      ];

      systemd.tmpfiles.rules = [
        "L+    /opt/rocm   -    -    -     -    ${pkgs.rocmPackages.clr}"
      ];

hope this helps will someone.

2 Likes