I’ve had this problem since I started using nixos – whenever I try to do something CUDA-related from a package build/patched through the nix build system, it works, blender overrides and pytorch cuda packages are an example of it.
The issue arises when I need to do things outside of patched packages or want to use the system install rather than build each software with another kernel. Or when I want to install a program that either doesn’t have a derivation or when I need to run something different that the derivation that exists, without creating a new one and doing all that wiring again.
For starters, the global install doesn’t seem to be recognized by any package at all, nothing that I’ve installed outside of the derivation was able to find the cuda libraries on my system (none of them seem to be in PATH). which ends up on either the program not finding CUDA or directly throwing a GPU not found error, while I’m using it just fine.
I’ve tried to set up the nvidia-container-toolkit to use with distrobox and podman, but distrobox enter throws an setting up CDI devices: unresolvable CDI devices nvidia.com/gpu=all error
My relevant config is the following:
nvidia.nix config
{
config,
pkgs,
...
}: {
hardware.graphics.enable = true;
services.xserver.videoDrivers = ["nvidia"];
nixpkgs.config.cudaSupport = true;
environment.systemPackages = with pkgs; [
cudaPackages.cudatoolkit
nvidia-container-toolkit
nvidia-docker
];
boot = {
blacklistedKernelModules = ["nouveau"];
kernelModules = ["nvidia_uvm"];
};
hardware.nvidia = {
modesetting.enable = true;
powerManagement.enable = false;
powerManagement.finegrained = false;
open = true;
nvidiaSettings = true;
package = config.boot.kernelPackages.nvidiaPackages.production;
};
hardware.nvidia-container-toolkit.enable = true;
nixpkgs.config.allowUnfreePredicate = p:
builtins.all (
license:
license.free
|| builtins.elem license.shortName [
"CUDA EULA"
"cuDNN EULA"
"cuTENSOR EULA"
"NVidia OptiX EULA"
]
) (
if builtins.isList p.meta.license
then p.meta.license
else [p.meta.license]
);
environment.variables = {
CUDA_HOME = pkgs.cudatoolkit;
};
}
virtualization.nix config
{...}: {
virtualisation = {
podman = {
enable = true;
dockerCompat = true;
};
};
}
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml 2>&1 | grep 'warning'
time="2025-04-28T15:52:18-03:00" level=warning msg="Failed to locate nvidia-cdi-hook: pattern nvidia-cdi-hook not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Failed to load ldcache: open /etc/ld.so.cache: no such file or directory"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate /dev/dri/controlD66: pattern /dev/dri/controlD66 not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Failed to load ldcache: open /etc/ld.so.cache: no such file or directory"
time="2025-04-28T15:52:18-03:00" level=warning msg="failed to locate libcuda.so: dlopen libcuda.so.*.* failed\npattern libcuda.so.*.* not found\nlibcuda.so.*.*: not found; using *.*"
time="2025-04-28T15:52:18-03:00" level=warning msg="Failed to load ldcache: open /etc/ld.so.cache: no such file or directory"
time="2025-04-28T15:52:18-03:00" level=warning msg="Failed to load ldcache: open /etc/ld.so.cache: no such file or directory"
time="2025-04-28T15:52:18-03:00" level=warning msg="Failed to load ldcache: open /etc/ld.so.cache: no such file or directory"
time="2025-04-28T15:52:18-03:00" level=warning msg="failed to locate libcuda.so: dlopen libcuda.so.*.* failed\npattern libcuda.so.*.* not found\nlibcuda.so.*.*: not found; using *.*"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate libnvidia-egl-gbm.so.*.*: dlopen libnvidia-egl-gbm.so.*.* failed\npattern libnvidia-egl-gbm.so.*.* not found\nlibnvidia-egl-gbm.so.*.*: not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate libnvidia-egl-wayland.so.*.*: dlopen libnvidia-egl-wayland.so.*.* failed\npattern libnvidia-egl-wayland.so.*.* not found\nlibnvidia-egl-wayland.so.*.*: not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate libnvidia-allocator.so.*.*: dlopen libnvidia-allocator.so.*.* failed\npattern libnvidia-allocator.so.*.* not found\nlibnvidia-allocator.so.*.*: not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate libnvidia-vulkan-producer.so.*.*: dlopen libnvidia-vulkan-producer.so.*.* failed\npattern libnvidia-vulkan-producer.so.*.* not found\nlibnvidia-vulkan-producer.so.*.*: not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate nvidia_drv.so: pattern nvidia_drv.so not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate libglxserver_nvidia.so.*.*: pattern libglxserver_nvidia.so.*.* not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate glvnd/egl_vendor.d/10_nvidia.json: pattern glvnd/egl_vendor.d/10_nvidia.json not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate nvidia/nvoptix.bin: pattern nvidia/nvoptix.bin not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate X11/xorg.conf.d/10-nvidia.conf: pattern X11/xorg.conf.d/10-nvidia.conf not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate X11/xorg.conf.d/nvidia-drm-outputclass.conf: pattern X11/xorg.conf.d/nvidia-drm-outputclass.conf not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate vulkan/icd.d/nvidia_icd.json: pattern vulkan/icd.d/nvidia_icd.json not found\npattern vulkan/icd.d/nvidia_icd.json not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate vulkan/icd.d/nvidia_layers.json: pattern vulkan/icd.d/nvidia_layers.json not found\npattern vulkan/icd.d/nvidia_layers.json not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate vulkan/implicit_layer.d/nvidia_layers.json: pattern vulkan/implicit_layer.d/nvidia_layers.json not found\npattern vulkan/implicit_layer.d/nvidia_layers.json not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate /nvidia-persistenced/socket: pattern /nvidia-persistenced/socket not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate /nvidia-fabricmanager/socket: pattern /nvidia-fabricmanager/socket not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate /tmp/nvidia-mps: pattern /tmp/nvidia-mps not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate nvidia-persistenced: pattern nvidia-persistenced not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate nvidia-imex: pattern nvidia-imex not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate nvidia-imex-ctl: pattern nvidia-imex-ctl not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate libnvidia-egl-gbm.so.*.*: dlopen libnvidia-egl-gbm.so.*.* failed\npattern libnvidia-egl-gbm.so.*.* not found\nlibnvidia-egl-gbm.so.*.*: not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate libnvidia-egl-wayland.so.*.*: dlopen libnvidia-egl-wayland.so.*.* failed\npattern libnvidia-egl-wayland.so.*.* not found\nlibnvidia-egl-wayland.so.*.*: not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate libnvidia-allocator.so.*.*: dlopen libnvidia-allocator.so.*.* failed\npattern libnvidia-allocator.so.*.* not found\nlibnvidia-allocator.so.*.*: not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate libnvidia-vulkan-producer.so.*.*: dlopen libnvidia-vulkan-producer.so.*.* failed\npattern libnvidia-vulkan-producer.so.*.* not found\nlibnvidia-vulkan-producer.so.*.*: not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate nvidia_drv.so: pattern nvidia_drv.so not found"
time="2025-04-28T15:52:18-03:00" level=warning msg="Could not locate libglxserver_nvidia.so.*.*: pattern libglxserver_nvidia.so.*.* not found"
I’m trying to make either the system’s cuda libraries or the nvidia ctk work for non nix packages and nix packages built without a cuda kernel built-in, but at this point I’m totally lost, and I cannot find anything else on the official docs. Any suggestions are greatly appreciated
System: nixos-unstable
Cuda version: 12.8.1
Nvidia drivers version: 570.133.07
Nvidia CTK vesion: 1.17.5