I’ve been having the same issue. Switching to the beta binary cache increased my download speed by an order of magnitude