Suspend problem

Sabo · October 7, 2024, 8:49am

Hello together!

I am having problems with suspend on NixOS 24.05.
I have a NVIDIA 970. I know bad hardware choice yada yada… But I have to live with my decisions from 10+ years ago.

I read the Nvidia NixOS Wiki here: Nvidia - NixOS Wiki
But no matter how I configure it some problems persist.

  hardware.nvidia = {
    ...

    # If this on  -> suspend fails: wakes up 1 sec after suspend
    # If this off -> graphical glitches after wakeup
    powerManagement.enable = true;

    # Changing this to beta, production, ... does not help
    package = config.boot.kernelPackages.nvidiaPackages.stable;
  };

Am I missing something?
What can I change?

(Suspend was working fine on Fedora 40, so probably no hardware failure.)

TLATER · October 7, 2024, 10:42am

It’s worth knowing what that option does in detail: Chapter 21. Configuring Power Management Support (the option sets this up for you, no need to make the changes manually)

I’d also suggest manually updating the drivers to the very latest if you are on stable: dotfiles/nixos-config/hosts/yui/nvidia/default.nix at 07fdf0228a28e9de9c08a3349ba5c2e6adca5a67 · TLATER/dotfiles · GitHub (you can watch gamingonlinux’ nvidia category to stay in the loop for updates).

The difference from Fedora could be explained by kernel and driver versions. I’d consider bumping the kernel version to the latest for testing, too. Using nouveau would also change things, and I’m pretty sure that’s the default over there.

If updating the driver doesn’t help, and the nvidia docs don’t seem applicable, get dmesg, journalctl --boot and journalctl --boot -1 logs for both settings and see if you can infer something from those. Make sure to do a full reboot after changing the setting (and updating drivers/kernels).

Sabo · October 9, 2024, 2:05pm

Thank you for your help! I really appreciate this.

I tried to match the kernel and NVIDIA driver as close as possible to the working version of my Fedora instance:

Fedora (suspend working)

~ ❯ uname -r          
6.10.12-200.fc40.x86_64

~ ❯ nvidia-settings -v
nvidia-settings:  version 560.35.03

Nix

[born@nixos:~]$ uname -r
6.10.13

[born@nixos:~]$ nvidia-settings -v
nvidia-settings:  version 560.35.03

Further debugging

I also tried to suspend from TTY, to make sure not some software is causing the issue.
This did not help.

The journalctl --boot command output is pretty long and I can’t understand it well.

Do you see any obvious errors?

Atemu · October 10, 2024, 2:21am

Looking at those stack traces, we now know for sure that Nvidia is at fault.

Try using the default LTS kernelPackages rather than 6.10.

Fedora might have patches for the Nvidia crap that we don’t carry; making them compatible with 6.10.

Sabo · October 10, 2024, 9:06am

Okay, I am now on kernel 6.6.54 and NVIDIA driver 560.35.03

The issue persists - and even got a bit worse.
Now on suspend the following happens:

suspend
fans turn off
displays turn off
fans spin up again → system wakes up
displays do not turn on again

Unfortunately I cannot share the output of journalctl because I have to force reboot to recover from this situation.

TLATER · October 10, 2024, 9:10am

You can use journalctl --boot -1 to share logs from the previous boot.

That said, yeah, looks very much like an nvidia bug. Not much that can be done about it, unless you find out what the difference between the fedora and NixOS kernels is, and apply the same patches to yours.

Sabo · October 10, 2024, 9:40am

The suspend now is very inconsistent, sometimes it suspends and wakes up again immediately, sometimes it fails to suspend altogether, sometimes suspend even worked.

Here is one example: Okt 10 11:41:14 nixos systemd-logind[776]: The system will suspend now!Okt 10 - Pastebin.com

Here is another example of journalctl --boot -1 - maybe not as helpful as not all messages are not saved because of the force shutdown:

Okt 10 11:30:19 nixos systemd-logind[776]: The system will suspend now!
Okt 10 11:30:19 nixos sudo[3320]: pam_unix(sudo:session): session closed for user root
Okt 10 11:30:19 nixos systemd[1]: Starting NVIDIA system suspend actions...
Okt 10 11:30:19 nixos systemd[1]: Starting Pre-Sleep Actions...
Okt 10 11:30:19 nixos systemd[1]: pre-sleep.service: Deactivated successfully.
Okt 10 11:30:19 nixos systemd[1]: Finished Pre-Sleep Actions.
Okt 10 11:30:19 nixos systemd[1]: Reached target Sleep.
Okt 10 11:30:20 nixos .cosmic-comp-wrapped[1198]: Failed to submit rendering: Failed to submit result for display
                                                  
                                                  Caused by:
                                                      0: The underlying drm surface encountered an error: DRM access error: Page flip commit failed on device `Some("/dev/dri/card0")` (Permission denied (os error 13))
                                                      1: DRM access error: Page flip commit failed on device `Some("/dev/dri/card0")` (Permission denied (os error 13))
                                                      2: Permission denied (os error 13)
Okt 10 11:30:20 nixos systemd[1]: nvidia-suspend.service: Deactivated successfully.
Okt 10 11:30:20 nixos systemd[1]: Finished NVIDIA system suspend actions.
Okt 10 11:30:20 nixos systemd[1]: nvidia-suspend.service: Consumed 509ms CPU time, 453.2M memory peak.
Okt 10 11:30:20 nixos systemd[1]: Starting System Suspend...

I think I am not technical enough to compare kernel patches.
So probably my NixOS journey stops here.
Sad to see it! I loved the declarative approach, so much easier than traditional distros.

Thank you for your help again!

If anyone still finds something I am very happy to try it though!

Atemu · October 10, 2024, 10:46am

From your logs I gather that you use cosmic. Did you also use it on Fedora?

Sabo · October 10, 2024, 10:57am

Yes, using the new cosmic alpha on both fedora and nix.

But I also tried Gnome and pure TTY on nix in case that might be the issue (did not help)

TLATER · October 23, 2024, 6:14pm

So, there’s a new driver: NVIDIA 565.57.01 Beta has Wayland and HDR improvements, plus DXVK and VKD3D optimizations | GamingOnLinux

It lists this bug as having been fixed, maybe this is what you’re looking for? To be clear, this would mean you need powerManagement.enable, but that’s kind of how this is supposed to work anyway.

@abl030 might also be interested.

Sabo · October 24, 2024, 8:28am

It seems to not be on the beta branch of nixos yet. So I need to specify the driver manually. Where can I find/obtain the sha256 etc?

package = config.boot.kernelPackages.nvidiaPackages.mkDriver {
 version = "565.57.01";
 sha256_64bit = ?
 sha256_aarch64 = ?
 openSha256 = ?
 settingsSha256 = ?
 persistencedSha256 = ?
};

TLATER · October 24, 2024, 9:00am

Just set them to "" and it should give you the shas one by one.

Sabo · October 24, 2024, 10:08am

Exactly what cheksums are for

For anyone wondering: here is what worked for me:

package = config.boot.kernelPackages.nvidiaPackages.mkDriver {
  version = "565.57.01";
   sha256_64bit = "sha256-buvpTlheOF6IBPWnQVLfQUiHv4GcwhvZW3Ks0PsYLHo=";
   sha256_aarch64 = "";
   openSha256 = "";
   settingsSha256 = "sha256-H7uEe34LdmUFcMcS6bz7sbpYhg9zPCb/5AmZZFTx1QA=";
   persistencedSha256 = "";
};

Sadly this did not improve the situation.

I tried the following combinations with, which all failed. All of them failed.

Kernel	Display protocol
6.11	Wayland Gnome
6.11	X11 Gnome
6.6	Wayland Gnome
6.6	X11 Gnome

I doubt it helps but here is an example output of NVIDIA 565.57.01, kernel 6.11, Wayland Gnome; still exceptions all over the place!

TLATER · October 24, 2024, 5:07pm

Yeaaah, you still get guarantees of subsequent evaluations being the same, though, i.e., ssh server cert style (aka TOFU).

Nix unfortunately does use an unorthodox method (and character sets) for hashing, so sadly the hashes published upstream can’t be converted. This is the best you get in the nix world.

Have you tried using the open driver?

Sabo · October 24, 2024, 5:22pm

No, I never tried to because it is mentioned on the nixos-hardware repo that maxwell GPUs do not support the open driver.

abl030 · October 25, 2024, 12:28am

Hey Tlater, thanks for the @.
I had enough of it TBH and bought myself an Intel A310 off ebay.
After years of glitches on suspend across all distros I was just amazed at how everything now ‘just works’ since the drivers are in the kernel.
Even things like gnome remote desktop are perfect now which I could never get to work with the two pascal cards I have here.
Thanks again for your help.