Pipewire pulse server not working anymore

I’m using pipewire on my system which generally works satisfactory. One of my use-cases is that I sometimes su to another user and also want to use audio from there. To achieve this, I added the following section to my pipewire config:

services.pipewire.config.pipewire = {
    context.modules" = [
        {
          name = "libpipewire-module-protocol-pulse";
          args = {
            "server.address" = [
              "unit.native"
              "tcp:4713"
            ];
          };
        }
      ];
    }

The other user account has PULSE_SERVER=tcp:127.0.0.1:4713 in his shell environment. This has been working until recently, but now I cannot play audio anymore on that other user. I’ve checked the usual suspects, the port is open and the connection is not blocked by the firewall. Here is what pw-mon shows me when I run pactl info as the other user:

added:
	id: 78
	permissions: rwxm
	type: PipeWire:Interface:Client (version 3)
 	properties:
 		pipewire.protocol = "protocol-native"
 		pipewire.sec.pid = "3012"
 		pipewire.sec.uid = "1000"
 		pipewire.sec.gid = "100"
 		pipewire.sec.label = "kernel"
 		module.id = "2"
 		object.id = "78"
 		object.serial = "82"
 		client.api = "pipewire-pulse"
 		pulse.server.type = "tcp"
 		pipewire.client.access = "restricted"
 		application.process.id = "19976"
 		application.process.user = "dude"
 		application.process.host = "nixbrett"
 		application.process.binary = "pactl"
 		application.name = "pactl"
 		application.language = "en_US.UTF-8"
 		window.x11.display = ":0"
 		application.process.machine-id = "e837084745db43e1b68f524b1110210a"
 		config.name = "pipewire.conf"
 		core.daemon = "true"
 		core.name = "pipewire-0"
 		default.clock.max-quantum = "1024"
 		default.clock.min-quantum = "256"
 		default.clock.quantum = "256"
 		default.clock.rate = "48000"
 		link.max-buffers = "16"
 		log.level = "2"
 		cpu.max-align = "64"
 		default.video.width = "640"
 		default.video.height = "480"
 		default.video.rate.num = "25"
 		default.video.rate.denom = "1"
 		clock.power-of-two-quantum = "true"
 		mem.warn-mlock = "false"
 		mem.allow-mlock = "true"
 		application.process.session-id = "2"
 		core.version = "0.3.42"
 		pipewire.access = "restricted"

It seems as if the connection is marked as restricted. How can I allow this connection to play audio? There seems to be all kinds of media-session-managers around that are probably related to this, but I don’t really understand yet how all of this plays together. But I’d like to play audio in this scenario again. Has anyone got this running and can provide some configuration here? I am using 21.11.