Help configuring Git and SSH with HomeManager

Help
1

Hello, I have Git and SSH configured as seen in the code attached below. It works well except for one issue. On a fresh boot, if I do ssh git@github.com or git clone git@github.com:foo/bar.git I am correctly prompted for my SHH passkey and can commit / push to the repository like normal. However, on a fresh boot if I enter a repository that I cloned on a previous session and attempt to make a commit, I get the error error: Couldn't find key in agent?. In order to make commits in a repository that I previously cloned, I have to manually run 'ssh git@github.com` once on every boot. Although it is not too annoying to do this I do not remember my old Arch config having this issue and I would like to know if it is possible to improve my NixOS config to fix this. Thanks!

{ hostOptions, ... }:
{
    programs = {
        git = {
            enable = true;
            userName = hostOptions.user.name;
            userEmail = hostOptions.user.email;

            signing = { 
                signByDefault = true; 
                key = hostOptions.user.publicSSHKey;
            };

            extraConfig = {
                init.defaultBranch = "main";
                gpg.format = "ssh";
            };
        };

        ssh = {
            enable = true;
            addKeysToAgent = "yes";

            extraConfig = ''
                Host *
                    IdentityFile ${hostOptions.user.privateSSHKey}
            '';
        };
    };

    services.ssh-agent.enable = true;
}