As of yesterday the NixOS cache is now hosted by Fastly.
Fastly is a big supporter of open source projects and now NixOS is one of them! Fastly provides us with CDN capability, which previously was running on AWS CloudFront.
Big thanks go to Fastly, in particular Tom Denniston and Elaine Greenberg, our friends at Infor, Packet.net, and Rob Vermaas.
I’m getting download errors on nearly every NixOS upgrade since the cache moved to Fastly. The build will fail in most cases. After trying again a few times those errors vanish most of the time. This is very annoying though.
For instance:
warning: unable to download 'https://cache.nixos.community/fhacan05grksvb2k5v4qas19yf6q99qy.narinfo': Error in the HTTP2 framing layer (16); retrying in 261 ms
warning: unable to download 'https://cache.nixos.org/nar/0jk4391qd9biw5w0f0pvsyylh2zvrdsz5qfnanw4qzhwysy72cz7.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 312 ms
warning: unable to download 'https://cache.nixos.org/nar/0j8x0sqsnjszn53r916nrvr0p36ji6y2r7wld26ih5s98z515rv8.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 250 ms
warning: unable to download 'https://cache.nixos.org/nar/1l4a4j8di41rrvangf2ix0rgrc8rwzg4ikrmskgjp2c2nisaccrj.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 271 ms
warning: unable to download 'https://cache.nixos.org/nar/1df7whm4rj1bgqr609czmvhqixnmhxgr8jd56hl4nmifzzbq6lzw.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 296 ms
error 9 while decompressing xz file
Any chance we can find out what’s going on here?
There have been some other 503 errors showing up lately, according to some other users. I have created an issue here, so we can gather all issues, and use that to contact Fastly support.
I will also see if we can enable some logging to see if we can find out more.
@fpletz Could you gather as much info as possible for your problem and post it in the linked issue?
From context I get that the S3 that was “behind” still remains the same, more or less?
Yes, the only thing that has changed is that Cloudfront has been replaced by Fastly.
Is there design documentation on how cache.nixos.org is supposed to work in concert with a client and something uploading data to it? S3 is not an ordinary file system.
Just for the record, I’m still getting this cache errors when updating right now. (And they do get away after retries, so not critical, even though slightly annoying).
Once again, just for the record: @zimbatm told me on Mastodon that these issues may be IPv6-related, and indeed, I was downloading the updates from an IPv6 network.
I am also experiencing problems when installing from home, which is an IPv6 connection.
e.g.:
warning: unable to download 'https://cache.nixos.org/nar/11gcpmqadnwk1s8q0v5ygh7v497fdx4ms3szihq6px4fbf60wzkr.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 311 ms
warning: unable to download 'https://cache.nixos.org/nar/1mx1gwd4srdqwnipp94zv0cax0mhbyzjhvhfqvppr2zpl68nfac9.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 342 ms
warning: unable to download 'https://cache.nixos.org/nar/080240p3cbs25znrv9bcfvxn9cmv4q8zv20bfxh4kap6hwrc1jgk.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 327 ms
Is there something that can be done about it?
I’m seeing the exact same error
As of January 2019 this problem still persists, however these errors are not resolving themselves as for others. These packages are exclusively .nar files, all other archives from nix.cache appear to copy without problem.
I have the same issue.
I have the same issue. From reading this thread I gather that there’s no solution/workaround for this available? I’m not able to upgrade my system, which is a real bummer.
The work around is to keep trying. For me two attempts are typically enough. --keep-going helps, too.
Hey everyone,
Sorry for the quiet on this issue.
I’ve been working with Fastly support and have made a configuration change which should eliminate the 503 problem. They’ve told me a lot of the 503 errors are due to a low connect timeout between Fastly and S3. I’ve changed this value from 1000ms to 2000ms. We’ll start with this change and see how the number of reports over time changes.
I can report that it’s definitely still happening. Maybe a bit less, though.
Are ICMP packages blocked somewhere on the line?
To my (limited) understanding, IPv6 packages are discarded if their MTU is too big for any unit in the chain. This happens normally in conjunction with an ICMP packet, which tells the sender too lower the MTU and resend. If ICMP pakets are filtered, this won’t work. But, as one stream of packets is not taking always the same route, sometimes the MTU is not too big for any unit, so that the packet will be delivered. On my university we had quiet a lot of problems with IPv6; on the one hand the universities firewall filtered some ICMP packages (which basically breaks IPv6), on the other hand some consumer grade routers filtered them too.
I see the kinds of errors documented by @fpletz quite frequently.
Generally, without many repeated attempts, I’m unable to use the haskell tool, stack, to instantiate a nix shell with all necessary dependencies.
Nodejs- and ghc-related packages, in particular, trigger these kinds of errors.
Note that the problem is still there. Is there a different cache I can use that works?