Setting environment variables that way should work in buildPhase just fine (try echoing them there).
I suspect your issue is rather
Which is probably caused by $HOME being set to non-existing /homeless-shelter by default for reproducibility reasons and cargo cannot write there.
Try calling export HOME=$TEMPDIR before running cargo build.