RTX 3070 sunshine nvec encoding fails

Help
1

I am trying to get game streaming working via sunshine, however there is something going wrong with encoding it seems. In the sunshine error logs it states

[2025-03-24 22:59:10.531]: Error: [AVHWDeviceContext @ 0x260e2e80] Cannot load libcuda.so.1
[2025-03-24 22:59:10.531]: Error: [AVHWDeviceContext @ 0x260e2e80] Could not dynamically load CUDA
[2025-03-24 22:59:10.532]: Error: Failed to create a CUDA device: Operation not permitted
...
[2025-03-24 22:59:10.821]: Info: vaapi vendor: VA-API NVDEC driver [direct backend]
[2025-03-24 22:59:11.067]: Error: [h264_vaapi @ 0x260df380] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2025-03-24 22:59:11.124]: Error: Could not open codec [h264_vaapi]: Function not implemented
[2025-03-24 22:59:11.125]: Info: Creating encoder [h264_vaapi]
...
[2025-03-24 22:59:28.332]: Error: [h264_nvenc @ 0x7fec103d05c0] Cannot load libnvidia-encode.so.1
[2025-03-24 22:59:28.332]: Error: [h264_nvenc @ 0x7fec103d05c0] The minimum required Nvidia driver for nvenc is 520.56.06 or newer
[2025-03-24 22:59:28.402]: Error: Could not open codec [h264_nvenc]: Operation not permitted

I have tried both open and closed drivers and am using the current beta driver.

  services.xserver.videoDrivers = ["nvidia"];

  hardware.graphics.enable = true;

  hardware.nvidia = {
    modesetting.enable = true;
    powerManagement.enable = false;
    powerManagement.finegrained = false;
    open = true;
    package = config.boot.kernelPackages.nvidiaPackages.beta;
  };

Is there a setting I’m missing somewhere? I can’t find anything about nvec / encoders on the wiki.

2

Hello
Can you also post your sunshine config.
After that try running sunshine with the right path to nvidia drivers with that:

  services.sunshine = {
    enable = true;
    # Use systemd’s wrapper feature to add the NVIDIA library paths
    serviceConfig.Environment = "LD_LIBRARY_PATH=${config.boot.kernelPackages.nvidiaPackages.beta.out}/lib:${config.boot.kernelPackages.nvidiaPackages.beta.out}/lib64";
  };

Second try installing this package:

  environment.systemPackages = with pkgs; [
    nvidiaPackages.beta.cudatoolkit
  ];

Third have you enabled AppArmor or something like this that could interfere with permissions?
Do you have any custom udev rules for nvidia?
How are you starting sunshine (gets started trough systemd or manually or from the terminal …)?

1 Like
3

Thanks for taking a look!
Below is my current sunshine config after adding your suggestion. It still doesn’t seem to support hvec though. I also tried installing cudatoolkit, but it takes forever to compile and I’d prefer not to add half an hour to my system updates if possible.

{
  config,
  ...
}:
{
  systemd.services.sunshine = {
    enable = true;
    serviceConfig.Environment = "LD_LIBRARY_PATH=${config.boot.kernelPackages.nvidiaPackages.beta.out}/lib:${config.boot.kernelPackages.nvidiaPackages.beta.out}/lib64";
  };

  services.sunshine = {
    enable = true;
    autoStart = true;
    capSysAdmin = true;
    openFirewall = true;
    applications = {
      apps = [
        {
          name = "SteamBigPicture";
          prep-cmd = [
            {
              do = "cosmic-randr mode DP-3 1280 720 --refresh 60.000 --adaptive-sync false";
              undo = "cosmic-randr mode DP-3 3440 1440 --refresh 120.000 --adaptive-sync false";
            }
          ];
          detached = [
            "setsid steam steam://open/bigpicture"
          ];
          auto-detach = "true";
          image-path = "steam.png";
        }
        {
          name = "Desktop";
          image-path = "desktop.png";
        }
    #     {
    #       name = "MoonDeckStream";
    #       cmd = "MoonDeckStream";
    #       auto-detach = "false";
    #     }
      ];
    };
  };
}
1 Like
4

Give this PR a shot and see if it helps.

2 Likes
5

Ok
Have you checked this:

Also you could try to give the user sunshine permissions. But I don’t know if sunshine even has this. But you could also check that you as the user have these groups.

  users.users.sunshine = {
    isSystemUser = true;
    extraGroups = [ "video" "nvidia" ];
  };
1 Like
6

I don’t really know how to test specific PR’s like that, but it seems to have been merged so I’ll check it out when its on unstable.
(https://nixpk.gs/pr-tracker.html?pr=392458)

7

Sorry, I forgot to mention I don’t have apparmor, and sunshine is started through its module (it has a autostart option) so I’m not really sure what that results in.

Regarding permissions, I’m in wheel so I should have permissions, and sunshine runs under my user.

8

The problem should be solved with the PR. The results could be when it would not have been an issue with the package that maybe appamour is preventing sunshine to do stuff.

9

The PR didn’t seem to fix it completely, I’m still getting the below error and it reverts back to h264. Is there a nvidia dependency that sunshine doesn’t pull in but I need in my config?

Errors:

[2025-04-03 09:26:14.886]: Error: [h264_nvenc @ 0x7fa0e03c9e80] Cannot load libnvidia-encode.so.1
[2025-04-03 09:26:14.886]: Error: [h264_nvenc @ 0x7fa0e03c9e80] The minimum required Nvidia driver for nvenc is 520.56.06 or newer
[2025-04-03 09:26:14.950]: Error: Could not open codec [h264_nvenc]: Operation not permitted
[2025-04-03 09:26:15.459]: Error: [h264_vaapi @ 0x7fa0e03c9e80] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2025-04-03 09:26:15.519]: Error: Could not open codec [h264_vaapi]: Function not implemented

Warnings:

[2025-04-03 09:25:48.589]: Warning: SSL Verification error :: ok
[2025-04-03 09:25:49.700]: Warning: SSL Verification error :: ok
[2025-04-03 09:25:49.751]: Warning: SSL Verification error :: ok
[2025-04-03 09:26:14.735]: Warning: Attempting to use NVENC without CUDA support. Reverting back to GPU -> RAM -> GPU
[2025-04-03 09:26:15.112]: Warning: Monitor  doesn't support hardware encoding. Reverting back to GPU -> RAM -> GPU
10

what Nvidia driver Version are you running?

11

I’ve tried both open and closed, version is 570.133.07

12

Can you try installing it with flatpak

13

Can you try services.sunshine.package = pkgs.sunshine.override { cudaSupport = true; };

1 Like
14

That did it, thank you!!!

1 Like
15

Doesn’t seem to work on the new cosmic-de module, I wonder why parsec can use nvec but sunshine can’t. Seems like there may be a permissions problem as sunshine can’t find cosmic-randr either.