From my observation, it seems like if the file is not cached at local cache.nixos.org server(s), then the fetch from origin is slow. Once the file is cached on cache.nixos.org server, it downloads on the client just fine. And from my use it seems like I’m the only nixos user in my region, as I seem to be the one downloading all of these files for first time here :).
If there is anything you folks can do to resolve this seemingly congestion between fastly and cache.nixos.org origins, that’ll be really great.
Thanks for the reply, though I only started experiencing this in the last 2-3 weeks, before that it worked well for ~2 years, so something must have changed.
FWIW, I have seen this recently too, 2~3 weeks sounds about right. Just for certain downloads / items, seemingly at random, when updating along the unstable channel.
+1 on exact same experience. I wasn’t sure on what is going on, because I saw this behavior on different ISPs in different cities, although arguably both in same region.
I’ve also recently noticed specific storepaths are occasionally VERY slow from the cache. I never used to see that. I think I first saw it about a month and a half ago, though. (That’s a vague memory. Don’t put too much stock in it)
Possibly-related, I’m seeing a bunch of cache misses for packages that are known and seemingly exist (because there are signatures for them). Perhaps some kind of negative caching or incomplete cache population.
The problem still persists for me, and even seems to have drastically increased.
About 1.5 years ago, I could download from https://cache.nixos.org with almost my full 1GB/s.
Now it’s only around 1MB/s - that’s over a 100x slowdown!
I ran several test:
I tested the download speeds from a different network in my hometown. This network only has a 50Mb/s connection, which is fully saturated: yay!
I then logged into my server at home with a 1GB/s down-link. This resulted in a slow download speed of 1.17MB/s, that’s substantially slower than my previous test (1.)! Even though I’m connecting to the same server, since the (redacted) https://cache.nixos.org IPs match to my previous test (1.).
$ ssh vesta
Last login: Tue Oct 7 13:44:17 2025 from 10.0.0.2
$ wget -O /dev/null https://cache.nixos.org/nar/1w0nk8lhf3vbna1cl07qs835f13xj7w60mx7ny3xx3rxk6waxk1r.nar.xz
--2025-10-07 13:45:17-- https://cache.nixos.org/nar/1w0nk8lhf3vbna1cl07qs835f13xj7w60mx7ny3xx3rxk6waxk1r.nar.xz
Resolving cache.nixos.org (cache.nixos.org)... [REDACTED IPv6], [REDACTED IPv4]
Connecting to cache.nixos.org (cache.nixos.org)|[REDACTED IPv6]|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96081900 (92M) [application/x-nix-nar]
Saving to: ‘/dev/null’
/dev/null 100%[========================================>] 91.63M 1.17MB/s in 71s
2025-10-07 13:46:28 (1.29 MB/s) - ‘/dev/null’ saved [96081900/96081900]
To verify that I don’t have any network issues myself, I followed up by downloading a large file from https://huggingface.co, this worked nicely and saturated my 1GB/s connection.
What am I to do? I do a lot of NixOS config hacking with frequent rebuilds, this significantly slows down my workflow.
I used to host my own binary cache, however I only have an upload speed of 50Mb/s, therefore rebuilds remain slow when I’m not home. And it was quite tedious always passing --substituters based on where I was…
Slow speeds likely mean that whatever you’re loading isn’t cached and needs backfilling from S3 at this time. This can happen after large channel bumps like staging merges.
I’d suggest waiting an hour or two and then retrying.