Nvidia dgpu prime offload mode + amd igpu + wayland?

I have an Asus Proart X13 which comes with an integrated AMD GPU and a a discrete nvidia GPU. Following the instructions on the wiki, I have setup prime option A, offline mode, where everything should use the amd gpu by default and nothing should use the nvidia card unless I specifically ask.

Relevant part of config:

  # this is the latest version that still works with the nvidia drivers
  boot.kernelPackages = pkgs.linuxPackages_6_12;

  hardware.graphics = {
    enable = true;
  };

  # Load nvidia driver for Xorg and Wayland
  services.xserver.videoDrivers = ["nvidia"];

  hardware.nvidia = {

    # Modesetting is required.
    modesetting.enable = true;

    # Nvidia power management. Experimental, and can cause sleep/suspend to fail.
    # Enable this if you have graphical corruption issues or application crashes after waking
    # up from sleep. This fixes it by saving the entire VRAM memory to /tmp/ instead
    # of just the bare essentials.
    powerManagement.enable = false;

    # Fine-grained power management. Turns off GPU when not in use.
    # Experimental and only works on modern Nvidia GPUs (Turing or newer).
    powerManagement.finegrained = false;

    # Use the NVidia open source kernel module (not to be confused with the
    # independent third-party "nouveau" open source driver).
    # Support is limited to the Turing and later architectures. Full list of
    # supported GPUs is at:
    # https://github.com/NVIDIA/open-gpu-kernel-modules#compatible-gpus
    # Only available from driver 515.43.04+
    open = true;

    # Enable the Nvidia settings menu,
    # accessible via `nvidia-settings`.
    nvidiaSettings = true;

    # Optionally, you may need to select the appropriate driver version for your specific GPU.
    package = config.boot.kernelPackages.nvidiaPackages.latest;

    prime = {
        offload = {
             enable = true;
             enableOffloadCmd = true;
        };
        # Make sure to use the correct Bus ID values for your system!
        nvidiaBusId = "PCI:196:0:0";
        amdgpuBusId = "PCI:197:0:0"; # For AMD GPU
   };
  };

  nixpkgs.config.allowUnfree = true;

With this I can start X11+i3. However if I try to switch to sway, the first thing I run into is sway complaining I’m using the nvidia proprietary drivers, even though sway should be running with the amdgpu since I chose offline mode. If I give sway the flag --unsupported-gpu to make it continue anyway, then I end up at a black screen. Here’s how I tried to setup sway:

  # enable Sway window manager
  programs.sway = {
    enable = true;
    wrapperFeatures.gtk = true;
  };

  services.greetd = let
    sway-nvidia-wrapper = pkgs.writeShellScriptBin "sway-nvidia" ''
    exec ${pkgs.sway}/bin/sway --unsupported-gpu "$@"
  '';
  in {
    enable = true;
    settings = {
      default_session = {
        # sway needs the `--unsupported-gpu` whenever you use nvidia proprietary drivers
        command = "${pkgs.greetd.tuigreet}/bin/tuigreet --time --cmd ${sway-nvidia-wrapper}/bin/sway-nvidia";
        user = "greeter";
      };
    };
  };

There’s a lot of info on prime mode out there but most is for intel gpus or older driver versions. Would appreciate any insight.

Just to confirm, are those bus ids correct?
This tool would help confirm:

nix --experimental-features "flakes nix-command" run github:eclairevoyant/pcids

Appears to verify they are correct:

$ nix --experimental-features "flakes nix-command" run github:eclairevoyant/pcids
PCI:196:0:0
	NVIDIA Corporation [10de]
	AD107M [GeForce RTX 4060 Max-Q / Mobile] [28a0]
PCI:197:0:0
	Advanced Micro Devices, Inc. [AMD/ATI] [1002]
	Device [150e]
1 Like

I got this working:

  • For the CPU, newest kernel is desirable (6.14)
  • The “latest” nvidia driver in nixpkgs is not latest, you need to go to nixos-unstable for it, but you can’t selectively pick a kernel module from unstable while using the stable kernel, so I use mkDriver to install 570 directly (see in config below). In the future when newer drivers are stable you should be able to just use package = config.boot.kernelPackages.nvidiaPackages.stable;
  • You can’t use sway as your compositor. That was my main problem. I assumed the compositors wouldn’t have HW compat issues since that’s not how things usually work on X11. Under gnome, everything works fine.
    • Edit: Turns out you can use sway without disabling nvidia, as long as you tell sway to specifically use the amd card, see next post
  • To run a steam game with the nvidia card with this setup you just change your launch options to nvidia-offload %command%
  • For reasons I don’t understand gamescope only works with amd though. You can set DRI_PRIME=1 to force amd, which you may expect should be unnecessary since the point of this offload mode is to require an opt-in, but vulkan apps can choose to use whichever card they want anyway, so you may need it. For example the test app vkcube requires this.

nvidia.nix (import into configuration.nix):

{ config, lib, pkgs, ... }:

{
  boot.kernelPackages = pkgs.linuxPackages_6_14;

  # Enable OpenGL
  hardware.graphics = {
    enable = true;
    enable32Bit = true;
  };

  # Load nvidia driver for Xorg and Wayland
  services.xserver.videoDrivers = ["nvidia"];

  hardware.nvidia = {

    # Modesetting is required.
    modesetting.enable = true;

    # Nvidia power management. Experimental, and can cause sleep/suspend to fail.
    # Enable this if you have graphical corruption issues or application crashes after waking
    # up from sleep. This fixes it by saving the entire VRAM memory to /tmp/ instead
    # of just the bare essentials.
    powerManagement.enable = false;

    # Fine-grained power management. Turns off GPU when not in use.
    # Experimental and only works on modern Nvidia GPUs (Turing or newer).
    powerManagement.finegrained = false;

    # Use the NVidia open source kernel module (not to be confused with the
    # independent third-party "nouveau" open source driver).
    # Support is limited to the Turing and later architectures. Full list of
    # supported GPUs is at:
    # https://github.com/NVIDIA/open-gpu-kernel-modules#compatible-gpus
    # Only available from driver 515.43.04+
    open = true;

    # Enable the Nvidia settings menu,
    # accessible via `nvidia-settings`.
    nvidiaSettings = true;

    # Optionally, you may need to select the appropriate driver version for your specific GPU.
    # package = config.boot.kernelPackages.nvidiaPackages.latest;

    package = config.boot.kernelPackages.nvidiaPackages.mkDriver {
      version = "570.133.07";
      # this is the third one it will complain is wrong
      sha256_64bit = "sha256-LUPmTFgb5e9VTemIixqpADfvbUX1QoTT2dztwI3E3CY=";
      # unused
      sha256_aarch64 = "sha256-2l8N83Spj0MccA8+8R1uqiXBS0Ag4JrLPjrU3TaXHnM=";
      # this is the second one it will complain is wrong
      openSha256 = "sha256-9l8N83Spj0MccA8+8R1uqiXBS0Ag4JrLPjrU3TaXHnM=";
      # this is the first one it will complain is wrong
      settingsSha256 = "sha256-XMk+FvTlGpMquM8aE8kgYK2PIEszUZD2+Zmj2OpYrzU=";
      # unused
      persistencedSha256 = "sha256-4l8N83Spj0MccA8+8R1uqiXBS0Ag4JrLPjrU3TaXHnM=";
    };

    prime = {
      offload = {
        enable = true;
        enableOffloadCmd = true;
      };
      # Make sure to use the correct Bus ID values for your system!
      amdgpuBusId = "PCI:197:0:0"; # For AMD GPU
      nvidiaBusId = "PCI:196:0:0";
    };
  };
}

Follow up: you actually can run sway without disabling nvidia! But you need to tell it to explicitly use the amd card. There is an env var for wlroots based compositors for this. See the sway-wrapper here setting WLR_DRM_DEVICES. If you have the exact same laptop the pci-e address is probably the same. Claude suggested using /dev/dri/cardX but the problem with those is they could change across boots. I think the reason sway may try to use the nvidia gpu instead of amd by default despite offline mode is that at least on my system card0 is the discrete gpu (nvidia) and card1 is the integrated one (amd).

{ config, lib, pkgs, ... }:

{
  # List packages installed in system profile. To search, run:
  # $ nix search wget
  environment.systemPackages = with pkgs; [
    # sway stuff
    grim # screenshot functionality
    slurp # screenshot functionality
    wl-clipboard # wl-copy and wl-paste for copy/paste from stdin / stdout
    mako # notification system developed by swaywm maintainer
    waybar
    wlr-randr
    kitty # terminal
    blueman # bluetooth applet
    networkmanagerapplet # internet connectivity applet
    udiskie # for auto mounting
    pciutils # for lspci
  ];

  # enable Sway window manager
  programs.sway = {
    enable = true;
    wrapperFeatures.gtk = true;
  };

  services.greetd = let
    # set WLR_DRM_DEVICES to specifically be the amd card so sway
    # doesn't try to use nvidia. we use the path with an explicit
    # pci-e address rather than /dev/dri/cardN because that numbering
    # can change across boots.
    #
    # To figure out the pci address for the card use `lspci | grep -i radeon`
    sway-nvidia-wrapper = pkgs.writeShellScriptBin "sway-nvidia" ''
    WLR_DRM_DEVICES=$(realpath /dev/dri/by-path/pci-0000:c5:00.0-card) exec ${pkgs.sway}/bin/sway --unsupported-gpu "$@"
  '';
  in {
    enable = true;
    settings = {
      default_session = {
        # sway needs the `--unsupported-gpu` whenever you use nvidia proprietary drivers
        command = "${pkgs.greetd.tuigreet}/bin/tuigreet --time --cmd ${sway-nvidia-wrapper}/bin/sway-nvidia";
        user = "greeter";
      };
    };
  };

  fonts.packages = with pkgs; [
    font-awesome # contains waybar icons
  ];
}