You may be able to use ssh match blocks to configure the user and port, though doing so will also mean the flake cannot be evaluated outside your specially configured environment. I don’t think you can condense it more than that, you’ll still need to use ssh+git.
You can’t use the registry for this, as nix no longer supports using the registry in flake inputs. Of course you can still override an input to use the registry on the command line, but that feels even clunkier to me.