How to use GPU during buildtime?

Here is the minimal reproducible example flake:

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
  outputs = { self, nixpkgs }:
  let
    system = "x86_64-linux";
    pkgs = import nixpkgs {
      inherit system;
      config = {
        allowUnfree = true;
        cudaSupport = true;
      };
    };
  in
  {
    packages.x86_64-linux.default = pkgs.stdenv.mkDerivation {
      name = "print gpu info";
      cudaSupport = true;
      unpackPhase = ''
        echo "unpacking"
      '';
      buildPhase = ''
        mkdir -p $out/bin
        nvidia-smi
      '';
    };
  };
}

Basically what i truly need is buildtime GPU access. But i cant seem to get any connection to drivers.

Normally nvidia-smi works and torchWithCuda works inside a devshell too. But i couldnt make it work in build phase. Just making the nvidia-smi example work is enough for me actually. I believe i can go from there.