I’m on Firefox 77.0.1 with firefox-wayland from nixos-unstable now and it appears that VAAPI is indeed used, where it applies, according to intel_gpu_top measurements.
Waiting for Firefox 78 hitting the shelves now, as there have been important performance fixes [1] gone into this release, especially around stuttering/flickering with UI-elements on-screen.
About the Window Protocol: This is just a guess, but perhaps VAAPI is indeed used to decode the frames, but Firefox still copies the decoded frames from the GPU to the CPU and back again without “wayland/drm”.
Anyhow, @primeos is correct in this context, as FF from unstable has now added libva to the wrapper libs, also, this whole FF hardware acceleration topic is still under heavy development and today still is dependend on many other factors.
@colemickens There have been issues [2] around opengl in firefox lately, which webrender is dependend upon.
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1619882
[2] Firefox no WebGL on Intel hardware: FEATURE_FAILURE_GLXTEST_FAILED · Issue #81514 · NixOS/nixpkgs · GitHub