I’m very happy with NixOS except for one thing: I can’t get PRIME offload to work on my laptop. I’m guessing this issue is not NixOS-specific, but I figure I will start here and see where it takes me. I want to root cause this issue and see what it would take to fix.
For reference, here are my NixOS config files: https://github.com/bonsairobo/nixos-config
They are pretty minimal at the moment, as I’ve mostly been trying to get to the point where I can use my NVIDIA GPU strictly for gaming and the AMD iGPU for all other tasks. The relevant bits are here:
# Enable AMD iGPU early in the boot process
boot.initrd.kernelModules = [ "amdgpu" ];
hardware.nvidia = {
# Drivers must be at version 525 or newer
package = config.boot.kernelPackages.nvidiaPackages.beta;
prime = {
offload = {
enable = true;
enableOffloadCmd = true;
};
# Default bus IDs should exist here:
# https://github.com/NixOS/nixos-hardware/blob/bb2db418b616fea536b1be7f6ee72fb45c11afe0/asus/zephyrus/ga503/default.nix
amdgpuBusId = "PCI:7:0:0";
nvidiaBusId = "PCI:1:0:0";
};
};
I’m mostly working off of this starting point: NVIDIA and Wayland: TL;DR: NVIDIA Linux drivers still suck and don’t work well with Wayland. This can be mitigated by using iGPU with dGPU offloading.. But I’ve changed anything to do with Intel to AMD. AFAICT, I don’t need to specify the PCI Bus IDs because there are defaults already set for my machine at the link above.
With this config, if I try to run nvidia-offload glxinfo | grep OpenGL, then I get this:
$ nvidia-offload glxinfo | grep OpenGL
OpenGL vendor string: AMD
OpenGL renderer string: AMD Radeon Graphics (renoir, LLVM 15.0.7, DRM 3.49, 6.1.55)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 23.1.7
OpenGL core profile shading language version string: 4.60
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.6 (Compatibility Profile) Mesa 23.1.7
OpenGL shading language version string: 4.60
OpenGL context flags: (none)
OpenGL profile mask: compatibility profile
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 Mesa 23.1.7
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions:
So it would seem that PRIME is not functional (this is the same output I would get without adding the nvidia-offload).
I’m not sure where to start debugging this.
Another weird thing is I don’t have nvidia-smi available and I’m not sure where to get that.
It would seem that I’ve managed to get offload working by adding this line to configuration.nix:
services.xserver.videoDrivers = [ "nvidia" ];
This gets me the following result:
$ nvidia-offload glxinfo | grep OpenGL
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: NVIDIA GeForce RTX 3070 Laptop GPU/PCIe/SSE2
OpenGL core profile version string: 4.6.0 NVIDIA 535.113.01
OpenGL core profile shading language version string: 4.60 NVIDIA
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.6.0 NVIDIA 535.113.01
OpenGL shading language version string: 4.60 NVIDIA
OpenGL context flags: (none)
OpenGL profile mask: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 535.113.01
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions:
I thought it was suspicious that xserver appears in the name of the NixOS setting while I’m trying to use Wayland. I’m not sure why it’s named that way. This change had a significant impact on the time it took to run nixos-rebuild switch, since it was compiling parts of the NVIDIA drivers.
The other strange part of this is that sway now requires that I run it with the --unsupported-gpu flag. And in fact it appears that Sway is being run on the NVIDIA GPU, which is not what I wanted:
> nvidia-smi
Fri Oct 6 21:06:00 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01 Driver Version: 535.113.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3070 ... Off | 00000000:01:00.0 Off | N/A |
| N/A 46C P3 N/A / 55W | 7MiB / 8192MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1569 G ...rxj6h-sway-unwrapped-1.8.1/bin/sway 2MiB |
+---------------------------------------------------------------------------------------+
I’m wondering if I need to do something specific to run sway with my AMD iGPU. I can see that it is possible to use the iGPU in principle, since I can do this:
$ glxinfo | grep OpenGL
OpenGL vendor string: AMD
OpenGL renderer string: AMD Radeon Graphics (renoir, LLVM 15.0.7, DRM 3.49, 6.1.55)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 23.1.7
OpenGL core profile shading language version string: 4.60
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.6 (Compatibility Profile) Mesa 23.1.7
OpenGL shading language version string: 4.60
OpenGL context flags: (none)
OpenGL profile mask: compatibility profile
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 Mesa 23.1.7
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions: