`nvidia-smi` Doesn't display Tesla Series GPU

[ParkedHampster]

In conversation with a friend directly, some questions were raised, notable:

What is the error message that comes from uncommenting the datacenter.enable = true; line?

Output with hardware.nvidia.datacenter:

# nixos-rebuild switch
building Nix...
building the system configuration...
error:
       … while calling the 'head' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35741:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `system.build.toplevel':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix':

       … while evaluating the option `warnings':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix':

       … while evaluating the option `systemd.services.nvidia-fabricmanager.serviceConfig':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/hardware/video/nvidia.nix':

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: lib.meta.getExe': The first argument is of type set, but it should be a derivation instead.

Above with --show-trace

# nrs --show-trace
building Nix...
building the system configuration...
error:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/default.nix:1:277:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35596:
       … while calling the 'head' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35741:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27631:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27748:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:9031:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `system.build.toplevel':

       … while evaluating the attribute 'mergedValue'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34959:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34979:
       … while evaluating the attribute 'values'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34766:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34612:
       … while calling the 'any' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34615:
       … while evaluating the attribute 'values'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:38484:
       … while calling the 'concatMap' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:38493:
       … while calling the 'concatMap' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34231:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34242:
       … while calling the 'map' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34255:
       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix':

       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34372:
       … while calling 'dischargeProperties'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37119:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37128:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:24665:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix:1:2760:
       … while calling 'showWarnings'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/trivial.nix:1:16786:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/trivial.nix:1:16791:
       … while calling 'foldr'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:1920:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:2069:
       … while calling 'fold''
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:1973:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:1984:
       … while calling the 'length' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:1946:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix:1:2773:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27631:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27748:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:9031:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `warnings':

       (10 duplicate frames omitted)

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix':

       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34372:
       … while calling 'dischargeProperties'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37119:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37128:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:24665:
       … while calling the 'concatLists' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13423:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:24457:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:24463:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13473:
       … while calling the 'concatLists' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13724:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13753:
       … while calling 'optional'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:14530:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:1:14536:
       … in the left operand of the AND (&&) operator
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13781:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/systemd.nix:1:13515:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27631:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27748:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:9031:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `systemd.services.nvidia-fabricmanager.serviceConfig':

       … while evaluating the attribute 'mergedValue'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34959:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:35055:
       … while calling 'merge'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:19811:
       … while calling the 'mapAttrs' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:19825:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:19851:
       … while calling 'filterAttrs'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15891:
       … while calling the 'removeAttrs' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15900:
       … while calling the 'filter' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15917:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15925:
       … in the argument of the not operator
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15933:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15933:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:19867:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:15943:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:19902:
       … while evaluating the attribute 'optionalValue'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:35609:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:35631:
       (8 duplicate frames omitted)

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/hardware/video/nvidia.nix':

       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34372:
       … while calling 'dischargeProperties'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37119:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37128:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/types.nix:1:20091:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/hardware/video/nvidia.nix:1:30380:
       … while calling 'getExe'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/meta.nix:1:8960:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/meta.nix:1:8963:
       … while calling 'getExe''
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/meta.nix:1:10272:
       … in the condition of the assert statement
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/meta.nix:1:10279:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/meta.nix:1:10286:
       … while calling 'assertMsg'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/asserts.nix:1:700:
       … in the right operand of the OR (||) operator
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/asserts.nix:1:710:
       … while calling the 'throw' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/asserts.nix:1:713:
       error: lib.meta.getExe': The first argument is of type set, but it should be a derivation instead.

The idea to use hardware.nvidia.datacenter.enable = true; came from this post from 2022:

How to use NVIDIA V100/A100 GPUs?

AFAIU the current NVIDIA driver packaged is 495.44. However this driver doesn’t support important “data center” GPUs, esp. V100’s and A100’s. For example, nvidia-smi works just fine on an g4dn.xlarge instance (T4), but fails to detect the GPU on a p3.2xlarge instance (V100): ❯ nvidia-smi No devices were found How does one use V100s/A100s in NixOS? It looks like there are drivers readily available here.

A bit more specifically, the idea to use nvidia-container-toolkit came from there, and the datacenter option was explicitly mentioned when hardware.nvidia-container-toolkit.enable = true; is set, as having that and running nixos-rebuild switch results in the following:

Output with nvidia-container-toolkit:

# nixos-rebuild switch
building Nix...
building the system configuration...
error:
       … while calling the 'head' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35741:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `system.build.toplevel':

       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix':

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error:
       Failed assertions:
       - `nvidia-container-toolkit` requires nvidia datacenter or desktop drivers: set `hardware.nvidia.datacenter.enable` or add "nvidia" to `services.xserver.videoDrivers`

Above but with --show-trace

# nrs --show-trace
building Nix...
building the system configuration...
error:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/default.nix:1:277:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35596:
       … while calling the 'head' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:35741:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27631:
       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:1:27748:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:9031:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:33591:
       … while evaluating the option `system.build.toplevel':

       … while evaluating the attribute 'mergedValue'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34959:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34979:
       … while evaluating the attribute 'values'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34766:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34612:
       … while calling the 'any' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34615:
       … while evaluating the attribute 'values'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:38484:
       … while calling the 'concatMap' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:38493:
       … while calling the 'concatMap' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34231:
       … while calling anonymous lambda
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34242:
       … while calling the 'map' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34255:
       … while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix':

       … from call site
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:34372:
       … while calling 'dischargeProperties'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37119:
       … while evaluating a branch condition
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:37128:
       … while evaluating the attribute 'value'
         at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:1:24665:
       … while calling the 'throw' builtin
         at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix:1:2659:
       error:
       Failed assertions:
       - `nvidia-container-toolkit` requires nvidia datacenter or desktop drivers: set `hardware.nvidia.datacenter.enable` or add "nvidia" to `services.xserver.videoDrivers`

I still see this as an option here:

hardware.nvidia.datacenter.enable - MyNixOS

But I’m not sure how much of an authority that is