Requesting review before starting NixOS mirror in China

Announcements
1

(Was originally posted at https://github.com/NixOS/nixpkgs/issues/32659#issuecomment-574530418 and reposted here because the thread was old and probably won’t get attention.)

(TL;DR: We are thinking of mirroring channels and cache.nixos.org again because of renewed interest. Please review this gist on how we are planning it: About https://github.com/tuna/tunasync-scripts/pull/49 · GitHub)

There has been some renewed interest in a mirror in the Chinese Nix community. (Even since the switch to Fastly, cache.nixos.org is still not ideal (still much slower than other distros), and the improvement seems have occurred only in random regions, probably depending on the ISP.)

We have created a mirroring script and a supporting environment, and currently TUNA and USTCLUG, hosters of two prominent open source software mirror sites in China (TUNA’s mirror, USTC’s mirror) are interested in running such a mirror for Nix packages. (TUNA is having other issues at the moment and is unable to do so in the short term.)

We are ready for some initial testing. But we would like to request official review from the NixOS team before we do so. Mirroring entire channels is no small task and we hope that communication will help it to go smoothly. Also, we just might be doing something wrong, and we really hope someone knowledgeable can point it out.

Information regarding how we are planning to mirror, a few links, and other things can be found in this gist, and we will keep it up to date. We request a review of what will be doing in order to minimize risk of mistakes and surprises. Also, please tell us if you need additional information about our mirrors before we start.

Thanks for creating the amazing Nix ecosystem, and thanks in advance for your assistance.

15 Likes
2

Wish you the best of luck, nixpkgs is a non-trivial amount of data to handle. :slight_smile:

3

@thoughtpolice: I suppose we can’t improve that just by some trivial action at Fastly?

@dramforever: does Debian work well in there? I thought they were mainly mirrored by Fastly, too. (But I certainly haven’t looked much into that.)

4

How much diskspace does cache.nixos use right now if you’d happen to know? (not building a cache, only curious)

5

As a Chinese NixOS user, I am really looking forword to seeing the mirror available in China! :grinning:
Fastly doesn’t work well in my place and the average download speed of cache.nixos.org is around 10kb/s, although still a little bit faster than THE mirror of Fedora Silverblue :joy:

1612171914
16121719142000×1500 1020 KB

1 Like
6

Nice to see that someone is working on being able to mirror (parts) of
the binary cache! :slight_smile:

Just curious if you ever thought about just running a regional caching
proxy in front of the S3 bucket or fastly? Maybe that would allow you to
have a high enough cache hit ratio and no need to mirror everything.

That would also help with those paths that might not have been part of
the channels that you are mirroring.

7

I’m not really experienced with these infrastructure stuff, but it seems that all Fastly based services suffer from slowness here in China. It’s probably going to be a while before action is taken on the Fastly side, if they ever do it.

8

One of the providers (namely TUNA) by policy will not host a caching proxy, so that’s kind of a no-go.

9

:thinking: I see their (geographically) closest PoPs are in Hong Kong and Japan. That doesn’t sound that bad to me (to explain speeds like 10 kbps), but I don’t really know how these things work, especially in China… so your experience certainly carries much more weight.