While working to mount the root file system during boot, the system is in what’s called “stage 1” or “initrd”. Userspace software is run in a tmpfs root until it mounts the root fs at a subdirectory, at which point switch_root is used to make it the root mountpoint and the “stage 2” OS is started.
All this is to say that for ZFS’s keylocation=file://... to work, the path you give it has to be there during stage 1, in the stage 1 tmpfs. NixOS does include a convenient way to do this though. The boot.initrd.secrets option tells NixOS that when it’s setting up the boot loader and the menu entries for all the generations, it should also copy some extra files into the initrd archive so they’re in the stage 1 tmpfs.
boot.inird.secrets."/foobar" = /path/to/foobar;
This says that while setting up the boot loader, the file at /path/to/foobar should be copied into the initrd so that stage 1 has it at /foobar in the tmpfs. Then ZFS will be able to find it for keylocation=file:///foobar.
Do be aware that this of course means that the key file is written unencrypted to your ESP /boot partition. If you want it to get the key file from a removable USB drive, that would involve some custom code to get it to mount the USB drive in stage 1, get the key file out of it, and unmount it.