After doing some test, I can finally get openai-whisper-cpp to work with CUDA.
Used a command rg -- -lcuda in nixpkgs repo and find a similar package openai-triton which use substituteInPlace to substitute some paths.
And I also found this post useful.
For my machine, The problem is solved by replacing "-lcuda " with "-lcuda -L${pkgs.linuxPackages.nvidia_x11}/lib " in Makefile with the substituteInPlace command, then it can be bulit with no error. I will post an overlay here later.