Oh it looks like I could actually have home-manager manage XFCE instead, of using Debian’s package: Letting home-manager handle the X session on Debian
But I don’t really understand how things interact between ~/.xinitrc, ~/.xserverrc, ~/.xsession, ~/.xsessionrc (!), ~/.profile and ~/.xprofile and the DM. Why are there so many different configuration files?