The NixOS Foundation's Call to Action: S3 Costs Require Community Support

[endgame]

@domenkozar

• Keeping the budget, but garbage-collecting it. We could shrink probably shrink the costs a lot (realistically up to 70%), but there are heavy tradeoffs at play here since the data in the cache is sometimes very valuable.

I’m with @vs49688 : we probably want to hold all source archives and patches, but in general, we will have to be very careful when doing this. I’m sure there are other ways the cache has cached derivations from nixpkgs which cannot be rebuilt from scratch. (It took a long time before people noticed that fetchZip/postBuild changes broke many fonts).

@jade

I have a suggestion with respect to how to get it out, although it has the potential risk of costing even more money in the process: can we deduplicate the data by stuffing it into a content addressed store on top of s3 before pulling that deduplicated content address store out?

If you set up a VPC with an S3 Gateway Endpoint (free), you then get free transfers for S3<->VPC. So you could shove a bunch of machines in there (which you still pay for, of course) to do this without paying to egress the entire contents of the bucket.

@graham33

@domenkozar @ron et al, I probably missed it above, but just wondering if we have Intelligent Tiering (Amazon S3 Intelligent-Tiering Storage Class | AWS ) on the buckets?

Every time I’ve looked at S3 Intelligent Tiering in my own work, the $0.0025 per 1,000 objects automation fee makes me nervous. According to @edolstra, there are 667M objects in the cache.nixos.org bucket, so you’re paying $1667.50/month in automation fees, and 3/4 of the bucket is already in Infrequent-Access tier by some mechanism or other. So Intelligent Tiering needs to move a lot of stuff to smarter storage classes to come out ahead (or we only turn it on for large NARs, or something).