An invocation of jellyfin-ffmpeg, as generated by Jellyfin:
λ /nix/store/ad1wsd9m6jdhpwxki58wxkkv8b66r96c-jellyfin-ffmpeg-7.1.2-2-bin/bin/ffmpeg -analyzeduration 200M -probesize 1G -f matroska,webm -init_hw_device cuda=cu:0 -filter_hw_device cu -hwaccel cuda -hwaccel_output_format cuda -noautorotate -hwaccel_flags +unsafe_output -threads 1 -i file:"/path/to/file.mkv" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:1 -map 0:0 -map -0:s -codec:v:0 hevc_nvenc -tag:v:0 hvc1 -preset p1 -b:v 3616000 -maxrate 3616000 -bufsize 7232000 -profile:v:0 main -g:v:0 72 -keyint_min:v:0 72 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_cuda=format=yuv420p" -codec:a:0 aac -ac 2 -ab 256000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "fd3f7a65fe205781c9a6d0e018ae45e4-1.mp4" -start_number 0 -hls_segment_filename "/var/lib/jellyfin/transcodes/fd3f7a65fe205781c9a6d0e018ae45e4%d.mp4" -hls_playlist_type vod -hls_list_size 0 -hls_segment_options movflags=+frag_discont -y "/var/lib/jellyfin/transcodes/fd3f7a65fe205781c9a6d0e018ae45e4.m3u8"
ffmpeg version 7.1.2-Jellyfin Copyright (c) 2000-2025 the FFmpeg developers
built with gcc 15.2.0 (GCC)
configuration: --disable-static --prefix=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-jellyfin-ffmpeg-7.1.2-2 --target_os=linux --arch=x86_64 --pkg-config=pkg-config --enable-gpl --enable-version3 --disable-nonfree --disable-static --enable-shared --enable-pic --disable-thumb --disable-small --enable-runtime-cpudetect --enable-gray --enable-swscale-alpha --enable-hardcoded-tables --enable-safe-bitstream-reader --enable-pthreads --disable-w32threads --disable-os2threads --enable-network --enable-pixelutils --datadir=/nix/store/a98zssnb4ravqjzkdp9jd2yirbk2il4p-jellyfin-ffmpeg-7.1.2-2-data/share/ffmpeg --enable-ffmpeg --enable-ffplay --enable-ffprobe --bindir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-jellyfin-ffmpeg-7.1.2-2-bin/bin --enable-avcodec --enable-avdevice --enable-avfilter --enable-avformat --enable-avutil --enable-postproc --enable-swresample --enable-swscale --libdir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-jellyfin-ffmpeg-7.1.2-2-lib/lib --incdir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-jellyfin-ffmpeg-7.1.2-2-dev/include --enable-doc --enable-htmlpages --enable-manpages --mandir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-jellyfin-ffmpeg-7.1.2-2-man/share/man --enable-podpages --enable-txtpages --docdir=/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-jellyfin-ffmpeg-7.1.2-2-doc/share/doc/ffmpeg --enable-alsa --enable-amf --enable-libaom --enable-libaribb24 --enable-libaribcaption --enable-libass --enable-avisynth --enable-libbluray --enable-libbs2b --enable-bzlib --enable-libcaca --enable-libcdio --enable-libcelt --enable-chromaprint --enable-libcodec2 --enable-cuda --enable-cuda-llvm --disable-cuda-nvcc --enable-cuvid --enable-libdav1d --enable-libdavs2 --enable-libdc1394 --enable-libdrm --enable-libdvdnav --enable-libdvdread --disable-libfdk-aac --enable-ffnvcodec --enable-libflite --enable-fontconfig --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libfribidi --enable-libgme --enable-gmp --enable-gnutls --enable-libgsm --enable-libharfbuzz --enable-iconv --enable-libilbc --enable-libjack --enable-libjxl --enable-libkvazaar --enable-ladspa --enable-liblc3 --enable-liblcevc-dec --enable-lcms2 --enable-lzma --disable-metal --disable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libmysofa --disable-libnpp --enable-nvdec --enable-nvenc --enable-openal --enable-opencl --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-opengl --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-libqrencode --enable-libquirc --enable-librav1e --enable-librist --disable-librtmp --enable-librubberband --enable-libsmbclient --enable-sdl2 --enable-libshaderc --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-librsvg --enable-libsvtav1 --disable-libtensorflow --enable-libtheora --enable-libtwolame --enable-libuavs3d --enable-libv4l2 --enable-v4l2-m2m --enable-vaapi --enable-vdpau --enable-libvpl --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-vulkan --enable-libvvenc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxavs2 --enable-libxcb --enable-libxcb-shape --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxevd --enable-libxeve --enable-xlib --enable-libxml2 --enable-libxvid --enable-libzimg --enable-zlib --enable-libzmq --enable-libzvbi --disable-debug --enable-optimizations --disable-extra-warnings --disable-stripping --extra-version=Jellyfin --disable-ptx-compression
libavutil 59. 39.100 / 59. 39.100
libavcodec 61. 19.101 / 61. 19.101
libavformat 61. 7.100 / 61. 7.100
libavdevice 61. 3.100 / 61. 3.100
libavfilter 10. 4.100 / 10. 4.100
libswscale 8. 3.100 / 8. 3.100
libswresample 5. 3.100 / 5. 3.100
libpostproc 58. 3.100 / 58. 3.100
[AVHWDeviceContext @ 0x556da3b3ea40] Cannot load libcuda.so.1
[AVHWDeviceContext @ 0x556da3b3ea40] Could not dynamically load CUDA
Device creation failed: -1.
Failed to set value 'cuda=cu:0' for option 'init_hw_device': Operation not permitted
Error parsing global options: Operation not permitted
The upshot is that Jellyfin falls back to slow CPU transcoding. Years ago, I did once have hardware transcoding working on this machine, but it broke somewhere along the way. I’m unable to figure out why libcuda.so.1 is unavailable. It is present in the nix store, but I don’t know how to properly get it on the path for jellyfin-ffmpeg:
λ fd libcuda /nix
/nix/store/5c3llpsnis658z7mndrgzmc1bdyw9g2f-nvidia-x11-580.126.09-6.12.67/lib/libcuda.so.580.126.09
/nix/store/5c3llpsnis658z7mndrgzmc1bdyw9g2f-nvidia-x11-580.126.09-6.12.67/lib/libcuda.so
/nix/store/5c3llpsnis658z7mndrgzmc1bdyw9g2f-nvidia-x11-580.126.09-6.12.67/lib/libcuda.so.1
/nix/store/5c3llpsnis658z7mndrgzmc1bdyw9g2f-nvidia-x11-580.126.09-6.12.67/lib/libcudadebugger.so.1
/nix/store/5c3llpsnis658z7mndrgzmc1bdyw9g2f-nvidia-x11-580.126.09-6.12.67/lib/libcudadebugger.so.580.126.09
/nix/store/5c3llpsnis658z7mndrgzmc1bdyw9g2f-nvidia-x11-580.126.09-6.12.67/lib/libcudadebugger.so
GPU information:
λ nvidia-smi
Thu Feb 5 21:55:50 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09 Driver Version: 580.126.09 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce GTX 1650 Off | 00000000:03:00.0 Off | N/A |
| 50% 28C P8 N/A / 75W | 1MiB / 4096MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
The extent of my relevant NixOS config:
204: services.jellyfin.enable = true;
417: services.xserver.videoDrivers = ["nvidia"];
418: hardware.nvidia.open = true;
The problem is the same if I flip hardware.nvidia.open and reboot.
Transcoding settings in the Jellyfish admin UI:
I found this GitHub issue which looks similar to mine, but the transcoding settings I have seem to be in alignment with the compatibility matrix for my GPU (GTX 1650). (I’m also a bit confused how the presented error message about libcuda was actually solvable in the Jellyfin transcoding settings. Why would feeding the GPU an unsupported encoding task result in an error about a library being missing?)
Thanks for your help!