Thanks!
Now I refactor my flake only using nixos-25.05 (before it have nixos-unstable and master branch), and removing the applyPatch here (which cost about 30s evaluation time), now my flake only evaluate in 1 minutes 15 seconds.
Thanks so much for the advice!