Cannot change refresh rate to more than 60Hz at full res

I currently have the following problem. I have a new laptop (ASUS Zephyrus G16) hooked up to a Thunderbolt 3 Dock.
My problem is that at full resolution (5120x1440) of my monitor only the 60Hz rate is working. Everything else will make the screen go black. The setup has worked on my previous laptop just fine and it also is working on the current one when I boot into windows.
It does not matter if I set the mode via gui or directly with

randr --output DP-1-1-2 --mode 5120x1440 --refresh 120

My GPU is in mode AsusMuxDgpu (according to supergfxctl). It does look like some kind of bandwith issue though because if I lower the resolution higher rates are possible.

xrandr gives me the following output

Screen 0: minimum 8 x 8, current 7680 x 1600, maximum 32767 x 32767
HDMI-0 disconnected (normal left inverted right x axis y axis)
DP-0 connected 2560x1600+5120+0 (normal left inverted right x axis y axis) 16mm x 10mm
   2560x1600    240.00*+  60.00  
DP-1 disconnected (normal left inverted right x axis y axis)
DP-1-1 disconnected (normal left inverted right x axis y axis)
DP-1-2 disconnected (normal left inverted right x axis y axis)
DP-1-3 disconnected (normal left inverted right x axis y axis)
DP-1-4 disconnected (normal left inverted right x axis y axis)
DP-1-1-1 disconnected (normal left inverted right x axis y axis)
DP-1-1-2 connected primary 5120x1440+0+0 (normal left inverted right x axis y axis) 1190mm x 340mm
   5120x1440    119.97 + 100.00    59.98* 
   3840x1080    119.97 +  99.96    59.97  
   2560x1440     59.95  
   2560x1080    120.00   119.88   100.00    60.00    59.94  
   1920x1080    120.00   100.00   119.88    60.00    60.00    59.94  
   1680x1050     59.95  
   1600x900      60.00  
   1280x1024     75.02    60.02  
   1440x900      59.89  
   1280x800      59.81  
   1152x864      75.00  
   1280x720      60.00    59.94  
   1024x768      75.03    70.07    60.00  
   832x624       74.55  
   800x600       72.19    75.00    60.32    56.25  
   640x480       75.00    72.81    66.67    60.00    59.94  
   720x400       70.08  
DP-1-1-3 disconnected (normal left inverted right x axis y axis)

boltctl also sees the dock connected correctly

 ● Belkin International, Inc. Thunderbolt 3 Dock - Pro
   ├─ type:          peripheral
   ├─ name:          Thunderbolt 3 Dock - Pro
   ├─ vendor:        Belkin International, Inc.
   ├─ uuid:          c4010000-0080-8c1e-03d5-bc288c809025
   ├─ generation:    Thunderbolt 3
   ├─ status:        authorized
   │  ├─ domain:     43338780-f093-9ea9-ffff-ffffffffffff
   │  ├─ rx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  ├─ tx speed:   40 Gb/s = 2 lanes * 20 Gb/s
   │  └─ authflags:  none
   ├─ authorized:    Fr 30 Aug 2024 23:01:38 UTC
   ├─ connected:     Fr 30 Aug 2024 23:01:38 UTC
   └─ stored:        Fr 30 Aug 2024 20:23:46 UTC
      ├─ policy:     iommu
      └─ key:        no

I have been sitting here for quite some time now but have run out of ideas by now. Would be great if someone had an idea what else I could try. If there is any more information I can provide that could be helpful I will gladly do so

What I can say so far. The issue happens in wayland as well as with x11. Also neither prime offload nor dgpu only behaved differently.
My current guess is that for some reason the bandwith seems to be limited but I do not understand why.

Unfortunately not too much progress. What I can say so far

Left USB-C/Thunderbolt 4 port and connected dGPU

  • works with 5120x1440@60Hz
  • works with 3840x1080@120Hz
  • 5120x1440@>60Hz results in black screen

Right USB-C port

  • works with 5120x1440@60Hz
  • 3840x1080@>60Hz causes crash back to login screen

The problems occur with X11 as well as wayland sessions. In Windows the monitor is working just fine. I am using NVIDIA Prime Offload but was unable to get the monitor to work with sync as well.
I am not entirely sure I the port does not output at all or if whatever it is outputting cannot be handled by the monitor.
I also tried to lower the bpc as well as setting boradcast rbg to limited hoping that would lower the bandwith usage (if bandwith availability is even part of the problem). All without any luck.

Here`s the verbose output of xrandr for the display port. Maybe somebody could point me in a direction

DP-1-2 connected primary 5120x1440+0+160 (0x6e6) normal (normal left inverted right x axis y axis) 1190mm x 340mm
        Identifier: 0x6e2
        Timestamp:  18916
        Subpixel:   unknown
        Gamma:      1.0:1.0:1.0
        Brightness: 1.0
        Clones:    
        CRTC:       0
        CRTCs:      1 0 2 3
        Transform:  1.000000 0.000000 0.000000
                    0.000000 1.000000 0.000000
                    0.000000 0.000000 1.000000
                   filter: 
        _KDE_SCREEN_INDEX: 1 
        EDID: 
                00ffffffffffff004c2d9c0f00000000
                2b1c0104b57722783aa2a1ad4f46a724
                0e5054bfef80714f810081c08180a9c0
                b3009500d1c02ad20050f03840401810
                3a00a9504100001a000000fd0032781e
                be61000a202020202020000000fc0043
                3439524739780a2020202020000000ff
                004831414b3530303030300a20200234
                02031ef046105a405b3f5c2309070783
                010000e305c000e60605018b7312565e
                00a0a0a0295030203500a9504100001a
                584d00b8a1381440f82c4500a9504100
                001e1a6800a0f0381f4030203a00a950
                4100001af4b000a0f038354030203a00
                a9504100001a00000000000000000000
                00000000000000000000000000000028
                701279000003013c57790188ff139f00
                2f801f009f055400020009006c370108
                ff139f002f801f009f05450002000900
                33b70008ff139f002f801f009f052800
                02000900000000000000000000000000
                00000000000000000000000000000000
                00000000000000000000000000000000
                0000000000000000000000000000f390
        HDCP Content Type: HDCP Type0 
                supported: HDCP Type0, HDCP Type1
        Content Protection: Undesired 
                supported: Undesired, Desired, Enabled
        max bpc: 12 
                range: (6, 12)
        Broadcast RGB: Automatic 
                supported: Automatic, Full, Limited 16:235
        audio: auto 
                supported: force-dvi, off, auto, on
        link-status: Good 
                supported: Good, Bad
        CTM: 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 
                0 1 
        CONNECTOR_ID: 289 
                supported: 289
        non-desktop: 0 
                range: (0, 1)
  5120x1440 (0x6e3) 966.000MHz +HSync -VSync +preferred
        h: width  5120 start 5168 end 5200 total 5280 skew    0 clock 182.95KHz
        v: height 1440 start 1443 end 1453 total 1525           clock 119.97Hz
  3840x1080 (0x6e4) 538.020MHz +HSync -VSync +preferred
        h: width  3840 start 3864 end 3880 total 3920 skew    0 clock 137.25KHz
        v: height 1080 start 1083 end 1093 total 1144           clock 119.97Hz
  5120x1440 (0x6e5) 797.250MHz +HSync -VSync
        h: width  5120 start 5168 end 5200 total 5280 skew    0 clock 150.99KHz
        v: height 1440 start 1443 end 1453 total 1510           clock 100.00Hz
  5120x1440 (0x6e6) 469.000MHz +HSync -VSync *current
        h: width  5120 start 5168 end 5200 total 5280 skew    0 clock  88.83KHz
        v: height 1440 start 1443 end 1453 total 1481           clock  59.98Hz
  3840x1080 (0x6e7) 453.000MHz +HSync -VSync
        h: width  3840 start 3888 end 3920 total 4000 skew    0 clock 113.25KHz
        v: height 1080 start 1083 end 1093 total 1133           clock  99.96Hz
  3840x1080 (0x6e8) 266.500MHz +HSync -VSync
        h: width  3840 start 3888 end 3920 total 4000 skew    0 clock  66.62KHz
        v: height 1080 start 1083 end 1093 total 1111           clock  59.97Hz