Why there is a single X process that is using my NVidia GPU if I have prime offloading enabled?

pasqui23 · December 31, 2021, 3:00pm

nvtop reports

 Device 0 [NVIDIA GeForce RTX 3080 Laptop GPU] PCIe GEN 1@ 8x RX: 0.000 KiB/s TX: 0.000 KiB/s
 GPU 210MHz  MEM 405MHz  TEMP  42°C FAN N/A% POW  13 W
 GPU[                                       0%] MEM[                        0.006Gi/15.747Gi]
   ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
100│GPU0 %                                                                                                                                                                                                                                                                  │
   │GPU0 mem%                                                                                                                                                                                                                                                               │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
 75│                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
 50│                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
 25│                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
   │                                                                                                                                                                                                                                                                        │
  0│────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│
   └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
    PID USER DEV    TYPE  GPU        GPU MEM    CPU  HOST MEM Command                                                                                                                                                                                                         
   3217 root   0 Graphic   0%      4MiB   0%     2%    165MiB /nix/store/mxg8z2h658hgdpv6gcyyjk4xbnj601gp-xorg-server-1.20.13/bin/X -config /nix/store/h8sxpwzf8xbqsdr34i4xfjv31v6scmx1-xserver.conf -xkbdir /nix/store/mkhijsslpa19kynnhi4cyag3j27k3dw2-xkeyboard-config-2.33

This suggest that even through I am on plasmawayland and I have set

      sddm = {
        enable = true;
        settings = {
          General = {
            DisplayServer = "wayland";
            Numlock = "on";
          };
          Wayland.CompositorCommand = "${pkgs.westonLite}/bin/weston --shell=fullscreen-shell.so";
        };

as I think sddm is now capable of being run on wayland .
Where am I doing anything wrong? How can I avoid running X on the Nvidia GPU, or at all?

pasqui23 · December 31, 2021, 3:06pm

And yes,I have NVIDIA Prime set up

pasqui23 · January 2, 2022, 4:26pm

Of nota that there is only that X server process and no other process is using the GPU

georgeoshardo · January 2, 2022, 4:32pm

This thread may be of interest:

Upon freshly booting up and not accessing any GPU apps (including nvidia-smi) what does
cat /sys/bus/pci/devices/0000\:01\:00.0/power/runtime_status
return?

I added

  boot.extraModprobeConfig = "options nvidia \"NVreg_DynamicPowerManagement=0x02\"\n";
  services.udev.extraRules = ''
  # Remove NVIDIA USB xHCI Host Controller devices, if present
  ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x0c0330", ATTR{remove}="1"
  
  # Remove NVIDIA USB Type-C UCSI devices, if present
  ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x0c8000", ATTR{remove}="1"
  
  # Remove NVIDIA Audio devices, if present
  ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x040300", ATTR{remove}="1"
  
  # Enable runtime PM for NVIDIA VGA/3D controller devices on driver bind
  ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="auto"
  ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="auto"
  
  # Disable runtime PM for NVIDIA VGA/3D controller devices on driver unbind
  ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="on"
  ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="on"
  '';

to my configuration.nix and after that cat /sys/bus/pci/devices/0000\:01\:00.0/power/runtime_status returns suspended when nothing is happening, and active when I run something, such as nvidia-smi

pasqui23 · January 2, 2022, 4:44pm

It returns suspended

georgeoshardo · January 2, 2022, 4:52pm

I should have said, when running that command make sure to use the correct bus ID of your GPU.

pasqui23 · January 2, 2022, 4:59pm

That is the correct id

lspci | grep VGA     
def-matcher: command not found
01:00.0 VGA compatible controller: NVIDIA Corporation GA104M [GeForce RTX 3080 Mobile / Max-Q 8GB/16GB] (rev a1)
05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne (rev c4)

pasqui23 · January 2, 2022, 6:47pm

The problem I’m trying to solve is that powertop reports 19 W of power consumption from the ethernet controller, even when almost no packet are exchanged, suggesting it is conflating the 13W reported from nvtop

Further the power consumption does not lower even if I turn off ethernet

So my next theory would be to check Nvidia drivers

nrdxp · January 3, 2022, 7:41pm

It really seems to depend on the specific hardware is use, and there is not an exact answer thanks to the closed source nature of the driver, but I have the same thing, even on a wayland session with offload. Actually, before I enabled the nividia persistence daemon I couldn’t even get the kernel to recognize the card at all, since Turing generation cards can power completely off when not in use.

If you have the persistence daemon running, then maybe that’s it. If not, then it’s anybodies guess. I would say your best bet of getting a clearer answer would be to review the Nvidia driver docs, probably starting somewhere around here.

pasqui23 · January 3, 2022, 11:21pm

Ok so I’ve found
https://wiki.archlinux.org/title/Hybrid_graphics#Using_acpi_call
But it seems that boot.kernelModules=["acpi_call"]; is not enough to install acpi_call,even through it is in nixpkgs
https://search.nixos.org/packages?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=acpi_call
So how to add a package to the list of “modprobable” kernel modules?

pasqui23 · January 3, 2022, 11:31pm

So even trying acpi_call was a bust

pasqui23 · January 4, 2022, 12:18am

Of note that while the nvidia docs suggests

Section "ServerLayout"
    Identifier "layout"
    Screen 0 "nvidia"
    Inactive "intel"
EndSection

Section "Device"
    Identifier "nvidia"
    Driver "nvidia"
    BusID "<BusID for NVIDIA device here>"
EndSection

Section "Screen"
    Identifier "nvidia"
    Device "nvidia"
    Option "AllowEmptyInitialConfiguration"
EndSection

Section "Device"
    Identifier "intel"
    Driver "modesetting"
EndSection

Section "Screen"
    Identifier "intel"
    Device "intel"
EndSection

my config generates

; nixos-option --config_expr $flake.config --options_expr $flake.options services.xserver.config
def-matcher: command not found
Value:
''
  Section "ServerFlags"
    Option "AllowMouseOpenFail" "on"
    Option "DontZap" "on"
    
  EndSection

  Section "Module"
    
  EndSection

  Section "Monitor"
    Identifier "Monitor[0]"
    
  EndSection

  # Additional "InputClass" sections
  Section "InputClass"
    Identifier "libinput mouse configuration"
    MatchDriver "libinput"
    MatchIsPointer "on"
    
    Option "AccelProfile" "adaptive"
    
    
    
    
    
    Option "LeftHanded" "off"
    Option "MiddleEmulation" "on"
    Option "NaturalScrolling" "off"
    
    Option "ScrollMethod" "twofinger"
    Option "HorizontalScrolling" "on"
    Option "SendEventsMode" "enabled"
    Option "Tapping" "on"
    Option "TappingDragLock" "on"
    Option "DisableWhileTyping" "off"
    
    
  EndSection

  Section "InputClass"
    Identifier "libinput touchpad configuration"
    MatchDriver "libinput"
    MatchIsTouchpad "on"
    
    Option "AccelProfile" "adaptive"
    
    
    
    
    
    Option "LeftHanded" "off"
    Option "MiddleEmulation" "on"
    Option "NaturalScrolling" "off"
    
    Option "ScrollMethod" "twofinger"
    Option "HorizontalScrolling" "on"
    Option "SendEventsMode" "enabled"
    Option "Tapping" "on"
    Option "TappingDragLock" "on"
    Option "DisableWhileTyping" "off"
    
    
  EndSection



  Section "ServerLayout"
    Identifier "Layout[all]"
    Option "AllowNVIDIAGPUScreens"
    
    # Reference the Screen sections for each driver.  This will
    # cause the X server to try each in turn.
    Screen "Screen-amdgpu[0]"

  EndSection

  Section "Module"
    Load "dri2"
    Load "glamoregl"
  EndSection


  # For each supported driver, add a "Device" and "Screen"
  # section.

  Section "Device"
    Identifier "Device-amdgpu[0]"
    Driver "amdgpu"
    Option "AccelMethod" "glamor"
    
    BusID "PCI:5:0:0"
    
    
    
  EndSection

  Section "Screen"
    Identifier "Screen-amdgpu[0]"
    Device "Device-amdgpu[0]"
    

    
    

    

    

  EndSection


  Section "Device"
    Identifier "Device-nvidia[0]"
    Driver "nvidia"
    Option "AccelMethod" "glamor"
    
    BusID "PCI:1:0:1"
    
    Option "NVreg_DynamicPowerManagement=0x02"
    
    
  EndSection






''

ElvishJerricco · January 4, 2022, 12:40am

I think you need boot.extraModulePackages = [config.boot.kernelPackages.acpi_call]; as well. I thought nixos was supposed to error out at build time if you didn’t have the necessary package configured for a kernel module, so that’s kinda odd.

pasqui23 · January 4, 2022, 12:44am

I did that, and all the acpi calls from turn_off_gpu.sh failed

pasqui23 · January 4, 2022, 12:54am

Especially of note is that while nvidia suggests

The generated xconfig instead includes

pasqui23:

  Section "ServerLayout"
    Identifier "Layout[all]"
    Option "AllowNVIDIAGPUScreens"
    
    # Reference the Screen sections for each driver.  This will
    # cause the X server to try each in turn.
    Screen "Screen-amdgpu[0]"

  EndSection

pasqui23 · January 4, 2022, 12:55am

Also there is no generated section similar to this

pasqui23 · January 11, 2022, 8:55pm

So i’ve tried to run my laptop with modesetting and no nvidia drivers at all and it seems that it does not reduce power consumption.

Apparently powertop estimates that my ethernet interfaces consumes 19 W unplugged, and I’ve already configured tlp, wifi power sanving

pasqui23 · January 16, 2022, 3:17pm

Even after sudo nvidia-smi --persistence-mode=1 nvidia-smi still report being on P0

pasqui23 · January 16, 2022, 4:45pm

cat /sys/bus/pci/devices/0000:01:00.0/power_state

D3hot