How to change xorg.xorgserver build flags without triggerring rebuild of all depedents on xorg.*?

Help
1

So I needed to have some options enabled in xorg.xorgserver package, which I couldn’t do individually, instead I had to change whole xorg group in overlay:

_: self: super:

{
  xorg = super.xorg // {
    xorgserver = super.xorg.xorgserver.overrideAttrs (old: {
      configureFlags = old.configureFlags ++ [ "--enable-suid-wrapper" ];
      postInstall = old.postInstall + ''
        mkdir -p $out/etc/X11
        echo "needs_root_rights=yes" >> $out/etc/X11/Xwrapper.config
      '';
    });
  };
}

because if I try to do it as xorg.xorgserver = super.... it makes all the xorg.* disappear except xorg.xorgserver from pkgs.

And since I edited the xorg all the packages dependent on xorg.* (which maybe irrelevant to xorg.xorgserver) such as window-managers/desktop-environments, all goes on rebuild… Which is really annoying, its like all the packages going full on CPU and my system goes in Gentoo mode…

Is there any better way, or the grouping of nixpkgs flawed by design for accommodating changes in build flags?