`nvidia-smi` Doesn't display Tesla Series GPU

I have a recent install of NixOS
Both nvtop and nvidia-smi are unable to detect my Tesla-series Nvidia card

Outputs:
$ nvtop
No GPU to monitor.
$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

I have tried a few different options in my configuration.nix file

here is the output of lspci | grep -vga:

Output:
04:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
05:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
0b:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)

I believe these to be the relevant config settings:

nvidia.nix
{ config, pkgs, ... }:
{
  nix.settings = {
    substituters = [ "https://cuda-maintainers.cachix.org" ];
    # ?
    # unsure if this is needed
    trusted-public-keys = [
      "cuda-maintainers.cachix.org-1:0dq3bujKpuEPMCX6U4WylrUDZ9JyUG0VpVZa7CNfq5E="
    ];
  };

  environment.systemPackages = with pkgs; [
    cudatoolkit
    linuxPackages.nvidia_x11
    libGLU libGL
  ];

  # OpenGL
  hardware.graphics = {
    enable = true;
    enable32Bit = true;
  };

  hardware.nvidia = {
    modesetting.enable = true; # I have also set this to false based on some suggestions I've seen
    powerManagement.enable = true; # this has been both true and false
    powerManagement.finegrained = false;
    open = false; # this has been both true and false
    nvidiaSettings = true;
    package = config.boot.kernelPackages.nvidiaPackages.production;
    
    # from https://discourse.nixos.org/t/nvidia-drivers-not-loading/40913/12
    #?
    forceFullCompositionPipeline = true;
    #?

    prime = {
      offload.enable = true;
      reverseSync.enable = true;
      #nvidiaBusId = "PCI:05:00:0";
      intelBusId = "PCI:0b:00:0";
      nvidiaBusId = "PCI:04:00:0";
    };
    # enabling this fails to build
    # datacenter.enable = true;
  };

  # uncommenting this leaves the screen at an uninteractable screen
  #services.xserver.videoDrivers =  [ "nvidia" ];
  # boot.initrd.kernelModules = [ "nvidia" "i915" "nvidia_modeset" "nvidia_uvm" "nvidia_drm" ];
  boot.kernelParams = [ "nvidia_drm.fbdev=1" ];
}

There are (obviously) other configuration set, but they are across multiple *.nix files

An important note to add - I am using xfce as a desktop environment on this server.
I’ve never used a DE on a server before this, but I’m not sure if that is entirely relevant.

Overall, I’m having a lot of trouble finding consistent information related to CUDA drivers, and almost no information related to server GPUs specifically.

Nvidia’s site indicates that the proper driver for my card should be the 550 driver, which seems to be the same one available in the production package.

I’m also having trouble finding any information related to GPUs that account for multiple PCI slots.

I’m fairly new to NixOS, so if there’s anything obvious that I’m missing, please let me know - or if there’s anything that may be breaking this that I’m just not accounting for.

In conversation with a friend directly, some questions were raised, notable:

What is the error message that comes from uncommenting the datacenter.enable = true; line?

Output with hardware.nvidia.datacenter:
# nixos-rebuild switch
building Nix...
building the system configuration...
error:
       … while calling the 'head' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35741:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `system.build.toplevel':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix':

       … while evaluating the option `warnings':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix':

       … while evaluating the option `systemd.services.nvidia-fabricmanager.serviceConfig':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/hardware/video/nvidia.nix':

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: lib.meta.getExe': The first argument is of type set, but it should be a derivation instead.
Above with --show-trace
# nrs --show-trace
building Nix...
building the system configuration...
error:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/default.nix:1:277:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35596:
       … while calling the 'head' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35741:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27631:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27748:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:9031:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `system.build.toplevel':

       … while evaluating the attribute 'mergedValue'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34959:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34979:
       … while evaluating the attribute 'values'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34766:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34612:
       … while calling the 'any' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34615:
       … while evaluating the attribute 'values'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:38484:
       … while calling the 'concatMap' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:38493:
       … while calling the 'concatMap' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34231:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34242:
       … while calling the 'map' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34255:
       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix':

       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34372:
       … while calling 'dischargeProperties'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37119:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37128:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:24665:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix:1:2760:
       … while calling 'showWarnings'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/trivial.nix:1:16786:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/trivial.nix:1:16791:
       … while calling 'foldr'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:1920:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:2069:
       … while calling 'fold''
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:1973:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:1984:
       … while calling the 'length' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:1946:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix:1:2773:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27631:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27748:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:9031:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `warnings':

       (10 duplicate frames omitted)

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix':

       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34372:
       … while calling 'dischargeProperties'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37119:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37128:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:24665:
       … while calling the 'concatLists' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13423:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:24457:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:24463:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13473:
       … while calling the 'concatLists' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13724:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13753:
       … while calling 'optional'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:14530:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:14536:
       … in the left operand of the AND (&&) operator
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13781:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13515:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27631:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27748:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:9031:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `systemd.services.nvidia-fabricmanager.serviceConfig':

       … while evaluating the attribute 'mergedValue'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34959:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:35055:
       … while calling 'merge'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:19811:
       … while calling the 'mapAttrs' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:19825:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:19851:
       … while calling 'filterAttrs'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15891:
       … while calling the 'removeAttrs' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15900:
       … while calling the 'filter' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15917:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15925:
       … in the argument of the not operator
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15933:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15933:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:19867:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15943:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:19902:
       … while evaluating the attribute 'optionalValue'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:35609:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:35631:
       (8 duplicate frames omitted)

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/hardware/video/nvidia.nix':

       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34372:
       … while calling 'dischargeProperties'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37119:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37128:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:20091:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/hardware/video/nvidia.nix:1:30380:
       … while calling 'getExe'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/meta.nix:1:8960:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/meta.nix:1:8963:
       … while calling 'getExe''
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/meta.nix:1:10272:
       … in the condition of the assert statement
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/meta.nix:1:10279:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/meta.nix:1:10286:
       … while calling 'assertMsg'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/asserts.nix:1:700:
       … in the right operand of the OR (||) operator
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/asserts.nix:1:710:
       … while calling the 'throw' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/asserts.nix:1:713:
       error: lib.meta.getExe': The first argument is of type set, but it should be a derivation instead.

The idea to use hardware.nvidia.datacenter.enable = true; came from this post from 2022:

A bit more specifically, the idea to use nvidia-container-toolkit came from there, and the datacenter option was explicitly mentioned when hardware.nvidia-container-toolkit.enable = true; is set, as having that and running nixos-rebuild switch results in the following:

Output with nvidia-container-toolkit:
# nixos-rebuild switch
building Nix...
building the system configuration...
error:
       … while calling the 'head' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35741:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `system.build.toplevel':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix':

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error:
       Failed assertions:
       - `nvidia-container-toolkit` requires nvidia datacenter or desktop drivers: set `hardware.nvidia.datacenter.enable` or add "nvidia" to `services.xserver.videoDrivers`
Above but with --show-trace
# nrs --show-trace
building Nix...
building the system configuration...
error:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/default.nix:1:277:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35596:
       … while calling the 'head' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35741:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27631:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27748:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:9031:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `system.build.toplevel':

       … while evaluating the attribute 'mergedValue'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34959:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34979:
       … while evaluating the attribute 'values'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34766:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34612:
       … while calling the 'any' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34615:
       … while evaluating the attribute 'values'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:38484:
       … while calling the 'concatMap' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:38493:
       … while calling the 'concatMap' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34231:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34242:
       … while calling the 'map' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34255:
       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix':

       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34372:
       … while calling 'dischargeProperties'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37119:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37128:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:24665:
       … while calling the 'throw' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix:1:2659:
       error:
       Failed assertions:
       - `nvidia-container-toolkit` requires nvidia datacenter or desktop drivers: set `hardware.nvidia.datacenter.enable` or add "nvidia" to `services.xserver.videoDrivers`

I still see this as an option here:

But I’m not sure how much of an authority that is