I beleive the issue is with the inner with plugins; statement. It needs to reference an attribute set of packaged grammars, with pkgs.tree-sitter-grammars; which has exactly that works.
Additionally you’ll want to remove the ensure_installed = "all" from the plugin config as nix will have taken care of the installation of the grammars. Leaving that config will cause errors as the plugin will try to install them and requires a compiler to do so.
The .so files will end up in the nix store and the path provided to neovim so you wont find them in ~/.local/share/nvim/site/parser anymore, you may even want to delete that directory after you have the nix config working as it may conflict with the nix installed grammars. You can verify the grammars are installed with nix using :TSInstallInfo once you’ve switched to the new configuration.