Hardware Encoding in OBS Studio with Nvidia RTX 2070 (TU106)

Help
1

I’m trying to have OBS Studio encode video with hardware so my game performance doesn’t tank. In OBS Studio, the only option for the streaming encoder is x264. It’s odd that there is no other option but I don’t really mind because I don’t care to stream. For video, there are several other options including FFmpeg VAAPI AV1, which is seemingly the only hardware encoder available for me to select. When I try to start recording with this encoder selected, I get the following error.

Starting the output failed. Please check the log for details.
Note: If you are using the NVENC or AMD encoders, make sure your video drivers are up to date.

The log says the following.

[FFmpeg VAAPI encoder: ‘advanced_video_recording’] Failed to open VAAPI codec: Function not implemented

So, I run nix-shell -p libva-utils --run vainfo to check the supported profiles and entrypoints.

Trying display: wayland
Trying display: x11
libva info: VA-API version 1.22.0
libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: VA-API NVDEC driver [direct backend]
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain12             : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileHEVCMain444            : VAEntrypointVLD
      VAProfileHEVCMain444_10         : VAEntrypointVLD
      VAProfileHEVCMain444_12         : VAEntrypointVLD

It would appear that I do not have any entrypoints for encoding, only decoding. I assume this is the problem, but I cannot figure out how to fix it.

Here is my configuration regarding Nvidia.

  services.xserver.videoDrivers = [ "nvidia" ];

  nixpkgs.config.allowUnfree = true;

  hardware.graphics = {
    enable = true;
    enable32Bit = true;

    extraPackages = with pkgs; [
      nvidia-vaapi-driver
    ];
  };

  hardware.nvidia = {
    modesetting.enable = true;
    powerManagement.enable = false;
    powerManagement.finegrained = false;

    open = true;

    nvidiaSettings = true;

    package = config.boot.kernelPackages.nvidiaPackages.stable;
  };

  nixpkgs.config.nvidia.acceptLicense = true;

Is there anything that I can do to get hardware encoding working on OBS Studio? I feel like there has to be some small configuration or package that I’m missing which would fix it immediately.

1 Like
2

Bumping, since I’m having the same issue. There’s two problems here: NVENC doesn’t appear in the list, and VAAPI simply doesn’t work for encoding.

Also of note: I have the same settings as you except I use open kernel modules.

3

You need to enable cuda support by overriding the package

programs.obs-studio.package = (pkgs.obs-studio.override {
    cudaSupport = true;
});

or globally

{
  nixpkgs.config.cudaSupport = true;
}

Relevant PR:

2 Likes
4

Thanks for your response, and sorry for my late reply.

I have applied the setting you mentioned globally and now I can see two more options for encoding: NVIDIA NVENC H.264 and NVIDIA NVENC HEVC. They both work, so now my problem is solved. Thank you so much! I hope this helps others who may be pulling out their hair like I was.