I’ve been using cgroups to try to work around similar problems. The magic formula being:
systemd-run --scope -p MemoryLimit=20000M firefox &
Unfortunately firefox currently seems to have a bug in that situation. I’ve opened a new discussion opened in a new topic. .
Tab unloading does help with my situation (see some of the linked Firefox bugs) but does not solve it.