It was a timing issue between pcscd and scdaemon using the internal ccid driver. More details are provided here: Troubleshooting - Nitrokey Documentation
I was able to resolve this issue by adding this to my Home Manager configuration:
programs.gpg = {
enable = true;
scdaemonSettings = {
disable-ccid = true;
};
};