Connection to cache.nixos.org seems awful

I tried to build a flake, pulling some binaries from cache.nixos.org, as I usually do, but today the download speed seems so bad that it could take days to download what I previously didn’t even notice. My everyday download speed is at about ~10Mbps, but when I pull from cache.nixos.org it does not go over 10kbps… I have tried from several computers, with wifi or Ethernet, nothing changes. Download speed is normal for every other website I tried.
I have found a diagnostic script, which gave this result, not sure if that’s very helpful. I have tried directly downloading several archives directly with curl (I tried, for instance, curl -vvv -o ghc.tar.gz https://cache.nixos.org/nar/0qwrky7iz1lgaqs03q0zzi9rj20h80966mahaz57s00gpfkdzvnl.nar.xz) but got exactly the same result (a speed that often did not exceed 5kbps).
I am kind of desperate because this makes building or upgrading my system impossible.

1 Like

Now it just works at the normal speed. However, I would like to find out what exactly caused this issue because I would like it very much not happening ever again. Does anyone have an idea?

1 Like

> mtr -c 20 -w -r cache.nixos.org
    Start: 2022-03-05T22:03:59+0100
    HOST: cthulhu                                         Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- 2a01:e0a:58a:330::1                              0.0%    20    2.2  26.1   2.1  91.4  38.7
      2.|-- 2a01:e02:1:f836:ae82::ffff                       0.0%    20   34.8  46.2  25.3 141.9  34.9
      3.|-- 2a01:e02:1:1735::ffff                           35.0%    20  182.3  95.5  46.1 255.8  66.7
      4.|-- ???                                             100.0    20    0.0   0.0   0.0   0.0   0.0
      5.|-- ???                                             100.0    20    0.0   0.0   0.0   0.0   0.0
      6.|-- ???                                             100.0    20    0.0   0.0   0.0   0.0   0.0
      7.|-- be4204.ccr21.par04.atlas.cogentco.com           90.0%    20  125.8 127.2 125.8 128.5   2.0
      8.|-- be2103.ccr42.par01.atlas.cogentco.com           65.0%    20  252.8 166.5 122.4 273.7  66.6
      9.|-- be2922.rcr21.b032899-0.par01.atlas.cogentco.com 25.0%    20  126.2 145.0 123.3 308.9  47.6
     10.|-- 2001:978:2:19::ca:2                             15.0%    20  125.0 151.2 118.4 263.8  48.6
     11.|-- 2a04:4e42:1d::729                               25.0%    20   64.2  98.4  63.1 273.4  62.1

As you can see in this mtr output there is 25% packet loss (the packet loss at intermediary routers usually doesn’t matter) at the last hop. When you have packet loss with TCP your speed drops significantly, since it waits for and retries to retrieve packets, that it didn’t receive.

If this happens regularly during certain daytimes it might be due to congestion on a particular link that leads you to cache.nixos.org. I’m especially not fond of Cogent, but it’s hard to say if they are at fault, since what you are not seeing is the reverse path, the one that the cache takes to you. Packet Loss could easily be on the reverse path as well, and it’s tricky to debug that then.

1 Like

If your diagnostic is correct, it seems like it’s impossible to do something about it…

is there a way to see the other path?

cache.nixos.org is backed by Fastly. I also often (once every 3 months) run into trouble with Fastly. Go to https://www.fastly-debug.com/ and https://v6.fastly-debug.com/ and send the result to Fastly support. They usually are able to fix the issue quickly.

3 Likes

I had the same thing happen to me. In my case, the output of mtr was fine. What did help is… reboot :confused: But then it happened again after a few minutes :man_shrugging: