Slow cache.nixos.org

Development Infrastructure
1

I’ve reported this problem at #infra:nixos.org Matrix room, but haven’t heard from them.

Following is the video demonstrating the problem:

https://asciinema.org/a/506447

Following is the output from the diagnostics script:

http://ix.io/40Rt

From my observation, it seems like if the file is not cached at local cache.nixos.org server(s), then the fetch from origin is slow. Once the file is cached on cache.nixos.org server, it downloads on the client just fine. And from my use it seems like I’m the only nixos user in my region, as I seem to be the one downloading all of these files for first time here :).

If there is anything you folks can do to resolve this seemingly congestion between fastly and cache.nixos.org origins, that’ll be really great.

EDIT: Update the link to the asciinema recording

6 Likes
2

I’m afraid the origin, namely aws s3, is known for being painfully slow.

3

Thanks for the reply, though I only started experiencing this in the last 2-3 weeks, before that it worked well for ~2 years, so something must have changed.

4

FWIW, I have seen this recently too, 2~3 weeks sounds about right. Just for certain downloads / items, seemingly at random, when updating along the unstable channel.

2 Likes
5

+1 on exact same experience. I wasn’t sure on what is going on, because I saw this behavior on different ISPs in different cities, although arguably both in same region.

1 Like
6

I’ve also recently noticed specific storepaths are occasionally VERY slow from the cache. I never used to see that. I think I first saw it about a month and a half ago, though. (That’s a vague memory. Don’t put too much stock in it)

7

Looks like there is an upstream issue that could be related to this:

https://github.com/NixOS/nixos-org-configurations/issues/212

2 Likes
8

Eelco reverted a change. Could someone confirm it’s fixed?

2 Likes
9

Last night the NixOS GNOME ISO was downloading very slow for me less than 1 MB/s on my gigabit internet in Las Vegas.

I just tried it again now and it is getting around 10 MB/s, so significantly better, but still slower than I remember before.

1 Like
10

I can confirm there’s an improvement, but still not back to old days as ryantm says. I’m in India if it makes any difference.

11

AFAICS and tell, the problem is still there:

https://asciinema.org/a/u9gzwyK1JG1rh9fnwDiLymsZ6

I used the URL from the issue.

HTH

12

Did a test again, it’s worse:

https://asciinema.org/a/y58Gulk1W1PEswPSEjSCCEJSP

13

Possibly-related, I’m seeing a bunch of cache misses for packages that are known and seemingly exist (because there are signatures for them). Perhaps some kind of negative caching or incomplete cache population.

14

Update from here:

In my test today on 10 Gbit/s,

wget -O /dev/null https://cache.nixos.org/nar/1w0nk8lhf3vbna1cl07qs835f13xj7w60mx7ny3xx3rxk6waxk1r.nar.xz

runs at 5 MB/s. That’s faster than 1 MB/s but still atrocious.