Putting this here for other people: It is not xorg that is using the GPU, it is nvidia-smi itself. Nvidia-smi turns on the GPU inorder to get information, and then reports itself as xorg for some reason.
You should use cat cat /proc/driver/nvidia/gpus/(PCI BUS ID)/power to check if the GPU is actually powered on or suspended. My output:
Runtime D3 status: Enabled (fine-grained)
Video Memory: Off
GPU Hardware Support:
Video Memory Self Refresh: Supported
Video Memory Off: Supported
S0ix Power Management:
Platform Support: Supported
Status: Disabled
Related post: How to use Nvidia PRIME offload to run the X server on the integrated board - #4 by boxofrox