I have transmission-gtk installed.
I’m NOT running the NixOS transmission service (not sure whether I had a reason not to once, or just didn’t know about it until now).
I have port 57766 open in my firewall, although that might not be relevant.
I’m using Mullvad too. I used to have ports open on their servers, but I don’t any more since it’s no longer an option.
Transmission seems to be working for me. Fingers crossed.
So I guess try removing the part of your config that describes the transmission service?