The NixOS Cache is now hosted by Fastly!


#1

As of yesterday the NixOS cache is now hosted by Fastly.

Fastly is a big supporter of open source projects and now NixOS is one of them! Fastly provides us with CDN capability, which previously was running on AWS CloudFront.

Big thanks go to Fastly, in particular Tom Denniston and Elaine Greenberg, our friends at Infor, Packet.net, and Rob Vermaas.


#2

I’m getting download errors on nearly every NixOS upgrade since the cache moved to Fastly. The build will fail in most cases. After trying again a few times those errors vanish most of the time. This is very annoying though.

For instance:

warning: unable to download 'https://cache.nixos.community/fhacan05grksvb2k5v4qas19yf6q99qy.narinfo': Error in the HTTP2 framing layer (16); retrying in 261 ms
warning: unable to download 'https://cache.nixos.org/nar/0jk4391qd9biw5w0f0pvsyylh2zvrdsz5qfnanw4qzhwysy72cz7.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 312 ms
warning: unable to download 'https://cache.nixos.org/nar/0j8x0sqsnjszn53r916nrvr0p36ji6y2r7wld26ih5s98z515rv8.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 250 ms
warning: unable to download 'https://cache.nixos.org/nar/1l4a4j8di41rrvangf2ix0rgrc8rwzg4ikrmskgjp2c2nisaccrj.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 271 ms
warning: unable to download 'https://cache.nixos.org/nar/1df7whm4rj1bgqr609czmvhqixnmhxgr8jd56hl4nmifzzbq6lzw.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 296 ms
error 9 while decompressing xz file

Any chance we can find out what’s going on here?


#3

There have been some other 503 errors showing up lately, according to some other users. I have created an issue here, so we can gather all issues, and use that to contact Fastly support.

I will also see if we can enable some logging to see if we can find out more.

@fpletz Could you gather as much info as possible for your problem and post it in the linked issue?


#4

From context I get that the S3 that was “behind” still remains the same, more or less?


#5

Yes, the only thing that has changed is that Cloudfront has been replaced by Fastly.


#7

Is there design documentation on how cache.nixos.org is supposed to work in concert with a client and something uploading data to it? S3 is not an ordinary file system.


#8

Just for the record, I’m still getting this cache errors when updating right now. (And they do get away after retries, so not critical, even though slightly annoying).


#9

Once again, just for the record: @zimbatm told me on Mastodon that these issues may be IPv6-related, and indeed, I was downloading the updates from an IPv6 network.


#10

So is fastly IPv6 CDN a bit flaky?


#11

I am also experiencing problems when installing from home, which is an IPv6 connection.
e.g.:

warning: unable to download 'https://cache.nixos.org/nar/11gcpmqadnwk1s8q0v5ygh7v497fdx4ms3szihq6px4fbf60wzkr.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 311 ms
warning: unable to download 'https://cache.nixos.org/nar/1mx1gwd4srdqwnipp94zv0cax0mhbyzjhvhfqvppr2zpl68nfac9.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 342 ms
warning: unable to download 'https://cache.nixos.org/nar/080240p3cbs25znrv9bcfvxn9cmv4q8zv20bfxh4kap6hwrc1jgk.nar.xz': HTTP error 200 (curl error: Failure when receiving data from the peer); retrying in 327 ms

Is there something that can be done about it?


#12

I’m seeing the exact same error