Nix builds freeze on desktop with nvidia gpu

Help
1

When building with Nix (but seemingly not any other activity), my desktop sometimes freezes. On a laptop with an extremely similar configuration (but importantly, no gpu) there are never freezes.

For concreteness, consider the following derivation, but it happens on other packages as well.

nix build github:stephen-huan/maipkgs#triton-cpu

It builds for a few seconds, but then the screen freezes & the computer becomes unresponsive (to keyboard inputs, mouse, trying to switch to a virtual terminal, etc.). Sometimes i3’s bar continues to update but eventually that stops as well, at which point I power off and power on again. I don’t think it’s just a display issue (i.e. the monitor isn’t updating but the computer is still working) because the exact same build takes ~10 minutes on my laptop and builds generally happen a lot faster on this desktop when they work, so if it was processing in the background I should be able to wait for it to finish and have it be built when I log back in, but this is never the case, no matter how long I wait.

Note that although I suspect the issue is ultimately caused by nvidia drivers, the above build doesn’t use cudaSupport at all (since it’s a flake build, it’s hermetic without --impure).

Desktop is a Dell Precision 3680 tower, cpu: Intel i7-14700K (28) @ 5.500GHz, dGPU: NVIDIA GeForce RTX 4070, iGPU: Intel Raptor Lake-S GT1 [UHD Graphics 770]. Laptop is a Tuxedo Pulse 15, cpu: AMD Ryzen 7 4800H with Radeon Graphics (16) @ 2.900GHz.

Both NixOS, nixos-unstable, configuration here (desktop has hostname tetuya, laptop is hima).

Things I’ve tried

  • I don’t think it’s a simple load issue, because I haven’t experienced this freezing under any other circumstances that isn’t a nix build. For example, I tried stressing the system with a giant matrix multiplication on cpu (bottom shows all the cores are at 100% for a good amount of time). I haven’t experienced the freezing using the computer on a day-to-day basis either (although since it’s a desktop I haven’t tried suspending / hibernating at all).
  • Although the build doesn’t take much memory (checking with bottom), I added a swapfile
    swapDevices = [{
  device = "/var/lib/swapfile";
  size = 32 * 1024;
  randomEncryption.enable = true;
}];
    The computer has 32 GB RAM + 32 GB swap, which is more than enough.
  • I am on impermanence, but (1) my /tmp is persisted, which is where the nix builds happen (so it isn’t a matter of running out of tmpfs space) and (2) the build works fine on my laptop, which is also nixos + impermanence (but with no gpu).
  • sudo journalctl -f doesn’t show anything interesting (the nix-daemon, then the freeze happens without a message, then I shutdown and the next message is the next boot).
    Jan 18 20:32:58 tetuya sudo[11138]:   slhuan : TTY=pts/3 ; PWD=/home/slhuan ; USER=root ; COMMAND=/run/current-system/sw/bin/journalctl -f
Jan 18 20:32:58 tetuya sudo[11138]: pam_unix(sudo:session): session opened for user root(uid=0) by slhuan(uid=1000)
Jan 18 20:33:03 tetuya nix-daemon[1570]: accepted connection from pid 11144, user slhuan (trusted)
-- Boot ec1beeb7902e49d8b3474775cd8442e5 --
Jan 18 20:36:41 tetuya kernel: Linux version 6.6.71 (nixbld@localhost) (gcc (GCC) 14.2.1 20241116, GNU ld (GNU Binutils) 2.43.1) #1-NixOS SMP PREEMPT_DYNAMIC Fri Jan 10 13:31:36 UTC 2025
  • Using the open-source nvidia module with the below configuration. I tried replacing "nvidia" with "nouveau" in the below but then SDDM doesn’t even start at all (it kicks me to the virtual terminal). I also tried removing the line services.xserver.videoDrivers = [ "nvidia" ]; entirely, and I can log in, but the desktop is very broken (can’t switch desktops on i3, etc.). This seems brittle, which is why I suspect the freezing caused by nvidia drivers.
    {
  hardware.nvidia = {
    modesetting.enable = true;
    open = true;
  };
  services.xserver.videoDrivers = [ "nvidia" ];
  nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
    "nvidia-persistenced"
    "nvidia-settings"
    "nvidia-x11"
  ];
}
  • Builds seem to succeed more often if I do them from a virtual terminal instead of from a graphical environment, although they still fail sometimes. Maybe they would always work if I did them from the virtual terminal and disabled all nvidia kernel modules, I’m not sure.
  • I’ve never experienced a build freeze when building my NixOS configuration. I have no idea why, or what makes certain packages freeze. It’s not deterministic, which makes it hard.
  • I updated the firmware with fwupd. This did not fix the issue.
2

Duplicate of Nix build ate my RAM 😭.

More cores means more RAM used, and nix by default uses all cores (0). I’m not sure why exactly my laptop is more responsive at oom compared to the desktop, and I guess that’s some bad interaction between linux oom, swap, and nvidia drivers.

{
  services.earlyoom = {
    enable = true;
    enableNotifications = true;
    # kill as soon as swap is used
    freeSwapThreshold = 100;
    freeSwapKillThreshold = 100;
  };
  # https://discourse.nixos.org/t/nix-build-ate-my-ram/35752
  systemd = {
    slices.nix-daemon.sliceConfig = {
      ManagedOOMMemoryPressure = "kill";
      ManagedOOMMemoryPressureLimit = "50%";
    };
    services.nix-daemon.serviceConfig = {
      Slice = "nix-daemon.slice";
      OOMScoreAdjust = 1000;
    };
  };
  nix.settings.cores = 16;
}