Issues with Nvidia Prime Sync on Wayland

I am running NixOS on my Dell Precision 5530 with a Nvidia Quadro P1000. I followed this guide (Nvidia - NixOS Wiki) to set up my GPU with PRIME Sync. My GPU will work completely fine on Xorg but is simply not used on Wayland. I have tried using elements of the configuration in this repo (nixos-hardware/dell/precision/5530 at master · NixOS/nixos-hardware · GitHub) as well as importing the whole thing as suggested but I have not had any luck. I also have tried the Nouveau drivers without any success. I am relatively new to Nix OS so any suggestions are very appreciated. Bellow are my configuration.nix and hardware-configuration.nix files:

{ config, lib, pkgs, ... }:

{
 imports =
   [
     ./hardware-configuration.nix
   ];

boot = {
 loader.systemd-boot.enable = true;
 loader.efi.canTouchEfiVariables = true;
  kernelParams = [  
     # fix lspci hanging with nouveau
     # source https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1803179/comments/149
     "acpi_rev_override=1"
     "acpi_osi=Linux"
     "nouveau.modeset=0"
     "pcie_aspm=force"
     "drm.vblankoffdelay=1"
     "nouveau.runpm=0"
     "mem_sleep_default=deep"
     # fix flicker
     # source https://wiki.archlinux.org/index.php/Intel_graphics#Screen_flickering
     "i915.enable_psr=0"
     "nvidia_drm.modeset=1"
   ];

};

 services.xserver = {
   enable = true;
   # Load nvidia driver for Xorg and Wayland
   videoDrivers = ["nvidia"];
   # Enable the GNOME Desktop Environment.
   displayManager.gdm.enable = true;
   desktopManager.gnome.enable = true;
   # Configure keymap in X11
   xkb = {
   layout = "us";
   variant = "";
   };
 };

 hardware = {
   nvidia = {
     modesetting.enable = true;
     powerManagement.enable = false;
     powerManagement.finegrained = false;
     open = false;
     nvidiaSettings = true;
     package = lib.mkDefault config.boot.kernelPackages.nvidiaPackages.stable;
     prime = {
       offload.enable = false;
       reverseSync.enable = false;
       sync.enable = true;
       # Enable if using an external GPU
       allowExternalGpu = false;
       intelBusId = "PCI:0:2:0";
       nvidiaBusId = "PCI:1:0:0";
     };
   };
 };

 services = {
   #From precision 5530 configeration on github
   fwupd.enable = lib.mkDefault true;
   thermald.enable = lib.mkDefault true;
 };

 networking.hostName = "HP-Printer"; # Define your hostname.
 # networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.

 # Configure network proxy if necessary
 # networking.proxy.default = "http://user:password@proxy:port/";
 # networking.proxy.noProxy = "127.0.0.1,localhost,internal.domain";

 # Enable networking
 networking.networkmanager.enable = true;

 # Set your time zone.
 time.timeZone = "America/New_York";

 # Select internationalisation properties.
 i18n.defaultLocale = "en_US.UTF-8";

 i18n.extraLocaleSettings = {
   LC_ADDRESS = "en_US.UTF-8";
   LC_IDENTIFICATION = "en_US.UTF-8";
   LC_MEASUREMENT = "en_US.UTF-8";
   LC_MONETARY = "en_US.UTF-8";
   LC_NAME = "en_US.UTF-8";
   LC_NUMERIC = "en_US.UTF-8";
   LC_PAPER = "en_US.UTF-8";
   LC_TELEPHONE = "en_US.UTF-8";
   LC_TIME = "en_US.UTF-8";
 };

 # Enable CUPS to print documents.
   services.printing = {
   enable = true;
   drivers = [ pkgs.epson-escpr ];
  };

 # Enable sound with pipewire.
 hardware.pulseaudio.enable = false;
 security.rtkit.enable = true;
 services.pipewire = {
   enable = true;
   alsa.enable = true;
   alsa.support32Bit = true;
   pulse.enable = true;
   # If you want to use JACK applications, uncomment this
   #jack.enable = true;

   # use the example session manager (no others are packaged yet so this is enabled by default,
   # no need to redefine it in your config for now)
   #media-session.enable = true;
 };

 # Enable touchpad support (enabled default in most desktopManager).
 # services.xserver.libinput.enable = true;

 # Define a user account. Don't forget to set a password with ‘passwd’.
 users.users.andrewh = {
   isNormalUser = true;
   description = "Andrew Heuer";
   extraGroups = [ "networkmanager" "wheel" ];
   packages = with pkgs; [
   #  thunderbird
   ];
 };

 programs = {
   firefox.enable = true;
   direnv.enable = true;
   steam = {
     enable = true;
     remotePlay.openFirewall = true; # Open ports in the firewall for Steam Remote Play
     dedicatedServer.openFirewall = true; # Open ports in the firewall for Source Dedicated Server
     localNetworkGameTransfers.openFirewall = true; # Open ports in the firewall for Steam Local Network Game Transfers
   };
 };

 # Allow unfree packages
 nixpkgs.config.allowUnfree = true;
 nixpkgs.config.cudoSupport = true;

 # List packages installed in system profile. To search, run:
 # $ nix search wget
 environment.systemPackages = with pkgs; [
   #General
   libreoffice-still 
   google-chrome

   # Code
   vscode
   eclipses.eclipse-sdk
   git

   # GameDev
   unityhub
   blender

   # Messageing
   discord

   # Games
   sidequest

   # Tools
   wget
   usbimager
   openvpn
   lshw
   nvtopPackages.full

   # Misc
   logisim
 ];

 nix.settings.auto-optimise-store = true;
 nix.gc = {
 automatic = true;
 dates = "weekly";
 options = "--delete-older-than 7d --keep 3";
 };

 # Some programs need SUID wrappers, can be configured further or are
 # started in user sessions.
 # programs.mtr.enable = true;
 # programs.gnupg.agent = {
 #   enable = true;
 #   enableSSHSupport = true;
 # };

 # List services that you want to enable:

 # Enable the OpenSSH daemon.
 # services.openssh.enable = true;

 # Open ports in the firewall.
 # networking.firewall.allowedTCPPorts = [ ... ];
 # networking.firewall.allowedUDPPorts = [ ... ];
 # Or disable the firewall altogether.
 # networking.firewall.enable = false;

 # This value determines the NixOS release from which the default
 # settings for stateful data, like file locations and database versions
 # on your system were taken. It‘s perfectly fine and recommended to leave
 # this value at the release version of the first install of this system.
 # Before changing this value read the documentation for this option
 # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
 system.stateVersion = "24.05"; # Did you read the comment?

}


# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernelModules = [ "xhci_pci" "ahci" "usbhid" "usb_storage" "sd_mod" "rtsx_pci_sdmmc" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/39c3dbb8-d8a6-41e2-ba1b-738c97c9610a";
      fsType = "ext4";
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/BBF5-13B5";
      fsType = "vfat";
      options = [ "fmask=0077" "dmask=0077" ];
    };

  swapDevices = [ ];

  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
  # (the default) this is the recommended approach. When using systemd-networkd it's
  # still possible to use this option, but it's recommended to use it in conjunction
  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
  networking.useDHCP = lib.mkDefault true;
  # networking.interfaces.wlp59s0.useDHCP = lib.mkDefault true;

  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
  hardware.cpu.intel.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
}

I am not currently using home manager or flakes.

Seems like a typo, though I don’t know if your issue is related

What happens when you run Wayland? No display, unaccelerated, what? Anything interesting in dmesg or journalctl?

I don’t see hardware.graphics.enable = true; in your config; that might be part of it.

You should probably be following the official wiki page, which I think is more up to date.

Just thought I’d add I have the same issue with a 2019/20 razerblade 15 that has intel integrated graphics and a mobile GTX 1070. On wayland I cannot get it to recognize the nvidia card is there (checking via nvtop) and game performance is pitiful.

FYI saying you have a GPU issue “on wayland” is kind of meaningless, it’s generally compositor-dependent.

1 Like

Good to know, in my case it was occuring on cosmic-de and I believe hyprland, but I’d have to check hyprland again to confirm.

Judging from OP’s config the issue being diagnosed in this thread is with GNOME Shell.

1 Like

Yep, there’s a reason I replied to the other reply and not OP.

This did not fix the issue but thank you for the suggestion.

I just installed Gnome-Shell on my laptop to test as well and am having the same issue. My config is nearly identical.

When I use Wayland it just uses my integrated GPU instead of my discrete GPU. Also this message made me realize I needed to update NixOS which I did, I also added the line you suggested. I am still having the same problem though, I’m going to try to look into how to use dmesg and journalctl thing later today.

Update:

After messing with it some more I have realized that this is not a system-wide issue. My main method of checking which GPU was being used was to open Blender and use nvtop to check which GPU was showing activity. What I have now found is that when I try the same method with Steam games the correct GPU is being used in Wayland. I am still very confused about why Blender specifically uses a different GPU based on whether I am using Xorg or Wayland but my main issue has been resolved.

Sync seems to be set up with xrandr. I wonder if that setting is even supported on wayland.

1 Like

Never mind, it worked once somehow, but now it is back to only working on Xorg for everything I test…

Are you saying that there is a different way I should be setting up my GPU for Wayland or it just won’t work with Wayland right now?

xrandr - as the “x” implies - configures Xorg and therefore would only affect xwayland.

I can’t spot anything regarding prime on wayland in the nvidia docs at all, except for this in the wayland known issues:

Display multiplexers (muxes) are typically used in laptops with both integrated and discrete GPUs to provide a direct connection between the discrete GPU and the built-in display (internal mux) or an external display (external mux). On X11, the display mux can be automatically switched when a full-screen application is running on the discrete GPU, enabling enhanced display features and improved performance, but no Wayland compositors currently support this functionality.

I’m not sure if this implies that the mux cannot be used at all.

I’m probably late to the party but I can confirm that NixOS setup using NVIDIA in sync mode won’t work automagically with Gnome on Wayland as it does with Gnome on Xorg.

I pulled my hair on this one last year and concluded that Wayland was the culprit. Thanks for the link to nvidia docs @TLATER.

The only way that I found to use dGPU in Wayland (Gnome) is to use the offload method (yes, even while using sync mode).

Example:

NIXPKGS_ALLOW_UNFREE=1 nix-shell -p unigine-valley

# Running benchmark as-is will give very poor performance, and one can
# see in nvidia-settings that the dGPU is not being used.
valley

# Setting variables for offload mode will actually offload to the dGPU,
# even in sync mode. This can be confirmed (even without launching the
# benchmark) by nvidia-settings showing dGPU usage.
export __NV_PRIME_RENDER_OFFLOAD=1
export __NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G0
export __GLX_VENDOR_LIBRARY_NAME=nvidia
export __VK_LAYER_NV_optimus=NVIDIA_only
valley

Using Xorg and sync mode, there’s no need to export these 4 variables, it just works™.

Yep, I’ve since learned that wayland’s model for this (unsurprisingly) is completely different. The protocol simply makes all GPUs available, it’s up to the compositor to decide where it renders what.

This means that GNOME needs to support a “reverse sync”-like feature if that’s what you want. Apparently, to do this properly, for GNOME you can set the primary GPU with a tag from an appropriate udev rule:

TAG+="mutter-device-preferred-primary"

To get non-reverse offload, if you write a rule which matches your iGPU and adds that property GNOME will render using your iGPU. The offload env variables still work as normal according to my experimentation, so the nvidia-offload script will still do its job.

To always render with the nvidia GPU, simply apply that property to the nvidia GPU with your udev rule, I believe something like this should do it:

ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TAG+="mutter-device-preferred-primary"
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TAG+="mutter-device-preferred-primary"

NixOS however seems to start udev too late in the boot process, so I think this won’t actually work - would love to hear if it does work for anyone.

It also does not enable the power saving features of nvidia GPUs, resulting in them never fully going to sleep. See here: Nvidia never suspends - #2 by TLATER

wlroots instead uses an environment variable.

1 Like

That’s very interesting, thanks a lot for sharing your findings!

So, I tried the udev rule you mentionned, without success. Here’s what I did:

  • identify the vendor and class attributes of my dGPU with this command:
udevadm info /dev/dri/... --attribute-walk
  • define services.udev.extraRules with following content:
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TAG+="mutter-device-preferred-primary"
  • check the rule was written to 99-local.rules

I also tried to set the rule in boot.inird.services.udev.rules, without much more success because nothing changed and the content of 99-local.rules didn’t contain the rule. But the option documentation mentions that the rule will only be included in initrd, which would explain why it doesn’t work.

Lastly, I tried to manually reload udev rules (udevadm control --reload-rules and udevadm trigger) but that didn’t do anything. And since I’m on Gnome Wayland, I can’t restart the shell with Alt + F2 followed by r… Thought of signing out and back in to « restart » Wayland but that doesn’t work either.

Will take a more thorough look at your links later :slight_smile:

There’s also services.udev.rules, without boot/initrd, that will allow testing at runtime rather than boot time. I’m not sure it will trigger on boot, though, I think by then the kernel has seen the device (and as mentioned in the other post, I don’t think the initrd udev will see it either).

The gnome project itself defines it this way, but this can have inconsistent results because it depends on load order: Making sure you're not a bot!

You may also want to check if anything else already has that tag :wink: