Enabling NVIDIA makes graphics slower

I use NixOS on a Lenovo Thinkpad P1 Gen 2 (equivalent to X1 Extreme G2) with hybrid graphics (nvidia and intel).
When i activate the nvidia gpu with prime sync (or offloading, doesn’t change anything weirdly), in my configuration.nix like this:

 services.xserver.videoDrivers = ["nvidia"];

  hardware.nvidia = {

    # Modesetting is required.
    modesetting.enable = true;

    # Nvidia power management. Experimental, and can cause sleep/suspend to fail.
    # Enable this if you have graphical corruption issues or application crashes after waking
    # up from sleep. This fixes it by saving the entire VRAM memory to /tmp/ instead 
    # of just the bare essentials.
    powerManagement.enable = false;

    # Fine-grained power management. Turns off GPU when not in use.
    # Experimental and only works on modern Nvidia GPUs (Turing or newer).
    powerManagement.finegrained = false;

    # Use the NVidia open source kernel module (not to be confused with the
    # independent third-party "nouveau" open source driver).
    # Support is limited to the Turing and later architectures. Full list of 
    # supported GPUs is at: 
    # https://github.com/NVIDIA/open-gpu-kernel-modules#compatible-gpus 
    # Only available from driver 515.43.04+
    # Currently alpha-quality/buggy, so false is currently the recommended setting.
    open = false;

    # Enable the Nvidia settings menu,
	# accessible via `nvidia-settings`.
    nvidiaSettings = true;

    # Optionally, you may need to select the appropriate driver version for your specific GPU.
    package = config.boot.kernelPackages.nvidiaPackages.stable;
  };

  hardware.nvidia.prime = {
    #sync.enable = true; # sync mode, gpu will stay active
    offload = {
      enable = true;
      enableOffloadCmd = true;
    };
    # Make sure to use the correct Bus ID values for your system!
    intelBusId = "PCI:0:2:0";
    nvidiaBusId = "PCI:1:0:0";
  };

The graphics overall get slower, gnome is a bit sluggish and hovering over fields changes their shade slower. The overall UX just feels worse. The GPU does get utilized tough. If i run nvidia-smi and run a 4k video in Firefox (which runs fine) i see GPU utilization and temp go up.

Altough if i disable the nvidia gpu by commenting out:

#  services.xserver.videoDrivers = ["nvidia"];

Gnome will feel faster and nicer to use which does not make sense to me.

Does anybody have some experience with this and could help me set this up to run smooth with both gpus? Help would be really appreciated.

Here is my full config:
configuration.nix: My nixos configuration.nix · GitHub
hardware-configuration.nix: hardware-configuration.nix · GitHub
nvidia-smi output: nvidia-smi · GitHub
System specs: neofetch · GitHub