Right, let me explain what’s going on, so maybe you understand where the mismatch is.
So, modern Linux DEs have now universally adopted wayland. X11 is still available, but it’s in the middle of being phased out.
Some desktop applications have however not yet caught on. This is because the second- or third-to-last ubuntu LTS is practically the universal development target for applications that don’t care about Linux support, and that means there is about a 4-8 year lag between what applications like discord do and what the Linux desktop actually needs. It also generally means that these applications are horribly insecure and shouldn’t be used.
This is alright in wayland land, because wayland has this integration layer called xwayland which sits in-between and translates X11 stuff to wayland. For the most part, this works well, except when wayland behaves fundamentally differently (notably, scaling on hidpi displays, font scaling, window resizing, window positioning between multiple screens, etc.). X11 just doesn’t have all the features we need for modern computer display purposes, so there’s no good way forward without fully abandoning it - but yeah, that’ll take a decade.
What does this have to do with screensharing? Well, wayland doesn’t permit applications to just randomly get output buffers from other applications. This is to prevent security issues; on traditional X11 systems it’s stupid easy to drop keyloggers n stuff which just capture everything happening on your desktop. It also allows wayland to optimize things a little under the hood.
For this reason, to be able to screenshare, an application needs to ask for permission on wayland. Applications that run on xwayland are completely incapable of this, because there is no concept like this in X11. You need a wayland-compatible application - or this xwaylandvideobridge, apparently, which I guess will just share buffers with any application the way X11 does, but asks for permission first. I wouldn’t be surprised if it’s a bit wonky and doesn’t always work.
Once we have an application that correctly asks for permission, wayland itself does not implement a UI of any kind, so we need some additional tool to show a popup and ask you if (and which) you want to share a window. The “xdg portal” of various desktop environments implement a protocol which wayland uses for this purpose, and pipewire is used underneath all of this to then actually share the buffee contents instead of directly giving things access to wayland memory.
So this leads to where we are here. You’re using the plasma desktop env, which by default sets up its xdg portal if you use the NixOS module for it. You should remove any additional portal configuration to minimize the amount of potentially incompatible things you have configured.
Next, you should confirm that whatever applications you’re trying to screenshare with actually use wayland. For this, I’d try disabling xwayland - not sure how to do that with plasma. If your browser doesn’t start anymore after that, it’s incompatible.
Once you’ve confirmed you have a browser that renders with wayland, and screensharing still does not work, we then will need to check the logs of pipewire and your xdg desktop portal to see if pipewire fails to send things around or if your portal fails to ask for permission.