OBS "Failed to open NVENC codec"

Help
1

Hello! I’m trying to stream/record with OBS on an Nvidia card, but I keep running into the following error:

image

Failed to open NVENC codec: Operation not permitted

Please check your video drivers are up to date.

Am I missing a package or piece of configuration? I’ve tried ffmpeg-full, nvidia-cg-toolkit, and nvidia-video-sdk, which fails to download. I am on GNOME/Wayland, in case that’s relevant. I already have this for Nvidia support:

services.xserver.videoDrivers = [ "nvidia" ];
hardware.nvidia.modesetting.enable = true;
hardware.opengl.enable = true;
2 Likes
2

Also getting this error., nvidia 3060 with similar settings

2 Likes
3

Same here. Any news?

EDIT: I’ve changed to Hardware QSV and it worked:

Screenshot_20250119_222036

4

I also had this problem. This is not the problem of drivers (am using AMD). This error occurs due to misconfiguration. I fixed this by enabling gnome temporarily:

services.xserver.desktopManager.gnome.enable = true;

You rebuild your system by adding this line. Then run obs studio. You will be able to configure obs. I use hyprland and it doesn’t start many services what gnome does where the issue occurs.

5 
boot.initrd.availableKernelModules = [
    "nvidia_drm" "nvidia_modeset" "nvidia" "nvidia_uvm"
];

Make sure to enable nvidia_uvm.
https://wiki.archlinux.org/title/NVIDIA#Hardware_accelerated_video_encoding_with_NVENC