I’m trying to set up CUDA-acceleration for the machine learning service of Immich.
This is running on a computer with an Nvidia T500, which should be supported by Immich.
GPU is set up like this (which is more or less copy-pasted from the wiki):
{ config, ... }:
{
allowedUnfree = [ "nvidia-x11" "nvidia-persistenced" ];
# Enable OpenGL
hardware.graphics = {
enable = true;
};
services.xserver.videoDrivers = [ "nvidia" ];
hardware.nvidia = {
nvidiaPersistenced = false;
# Modesetting is required.
modesetting.enable = true;
# Nvidia power management. Experimental, and can cause sleep/suspend to fail.
# Enable this if you have graphical corruption issues or application crashes after waking
# up from sleep. This fixes it by saving the entire VRAM memory to /tmp/ instead
# of just the bare essentials.
powerManagement.enable = false;
# Fine-grained power management. Turns off GPU when not in use.
# Experimental and only works on modern Nvidia GPUs (Turing or newer).
powerManagement.finegrained = false;
# Use the NVidia open source kernel module (not to be confused with the
# independent third-party "nouveau" open source driver).
# Support is limited to the Turing and later architectures. Full list of
# supported GPUs is at:
# https://github.com/NVIDIA/open-gpu-kernel-modules#compatible-gpus
# Only available from driver 515.43.04+
# Currently alpha-quality/buggy, so false is currently the recommended setting.
open = false;
# Enable the Nvidia settings menu,
# accessible via `nvidia-settings`.
nvidiaSettings = false;
# Optionally, you may need to select the appropriate driver version for your specific GPU.
package = config.boot.kernelPackages.nvidiaPackages.stable;
prime = {
offload = {
enable = true;
enableOffloadCmd = true;
};
intelBusId = "PCI:0:2:0";
nvidiaBusId = "PCI:1:0:0";
};
};
hardware.nvidia-container-toolkit.enable = true;
}
My problem is that the GPU doesn’t seem to be recognised. I test this by uploading a new image to Immich and looking at nvtop. I expect the “smart search” machine learning job to cause some load on the GPU, but never see some.
Already tried to set services.immich.machine-learning.environment.DEVICE to cuda or nvidia and users.users.immich.extraGroups = ["video" "render"];, but didn’t succeed.
Oh, it seems the package is not built with GPU support. Compare the dependencies in the package to the original project.toml.
Can someone confirm that my interpretation is correct?
I’ve gotten immich-machine-learning to work with CUDA. Here are the things I had to do. This is in addition to all the work to get nvidia drivers working. You probably want to verify that with a different app first. I used plex hardware transcoding to verify.
1.Enable cudaSupport for onnxruntime.
Ideally, I’d be able to do this via nixpkgs.config.cudaSupport but mxnet-1.9.1 with cudaSupport is marked broken. onnxruntime is a transitive dependency via at least insightface but probably also huggingface-hub so I had to use an overlay:
I read in CUDA - NixOS Wiki that adding the nix-community cache might prevent me from having to recompile things, but it didn’t (maybe because I’m not using the same nvidia driver version as others? IDK).
Point LD_LIBRARY_PATH to onnxruntime.
It seems like this might be a bug in how onnxruntime is packaged. I might file one later.
There’s a PR that’s been merged and will allow configuring this without the override:
When that’s available, if you’re using an NVIDIA card, setting services.immich.accelerationDevices to the above DeviceAllow list should enable GPU acceleration. Otherwise you’ll get an error like this:
[E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 100: no CUDA-capable device is detected ; GPU=0 ; hostname=foo ; file=/build/source/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=65 ; expr=cudaGetDeviceCount(&num_devices);