Upcoming Garbage Collection for cache.nixos.org

Announcements
1

Dear NixOS Community,

we write to share important news regarding an upcoming garbage collection process scheduled for end of February on cache.nixos.org.

This initiative is driven by our commitment to optimizing the repository and improving overall performance, while also addressing the substantial costs associated with the storage of our build artifacts. As part of our ongoing efforts, we aim to reduce these costs by implementing strategic measures such as garbage collection.

In this case, a garbage collection is the process of deleting store paths from cache.nixos.org. In the history of the cache, we always have taken an approach trying to preserve as much as possible all our history and all our store paths built once. (see below for details and implications)

Context: Navigating the Challenges of the Binary Cache

The NixOS Cache Working Group was formed in the fall last year with a bold idea: what if we could use chunk deduplication to reduce the costs of storage? Depending on the amount of deduplication achievable, this would allow us to reduce the ongoing costs, while also keeping the whole build history. Compressing/Deduplicating in place would also make it cheaper to egress from AWS in case we needed to change our hosting provider (for reference, the current extraction cost would be $25k).

The alternative was always that if the savings were not significant enough, we would have to resort to removing old build artifacts. We wanted to avoid doing so, because being able to checkout a 10 year old nixpkgs and still being able to run binaries from there without having to build is a neat property. Especially for scenarios such as reproducible science.

The efforts so far produced a lot of valuable tooling, such as a parquet index of all the NARInfo files, a bucket log processing pipeline, and better tooling to understand dependencies between store paths in the binary cache. All these tools are open-source, and we plan to make the data more available more widely in the near future too.

On the cost saving side, while the deduplication would reduce the amount of data, we unfortunately didn’t anticipate an additional point in the list of AWS charges - the cost of PUT requests to re-write the data in chunked fashion.
$0.0005 per 1000 requests is cheap, until you have 800M objects. If you also use per-blob chunking to deduplicate similar files, this number could easily be multiplied by 4-8.

Factoring in these costs, we’d be in a range that is higher than the extraction costs just to deduplicate things in-place, greatly diminishing the immediate benefits.

While this could have been worked-around by yet more engineering effort (“packing” chunks together and providing an index), we decided against it at this time, and instead to proceed with plan B. This is mostly to relieve some pressure, as the foundation currently has to pay an additional $6,000 and rising per month (on top of the $9000 cost offset by Amazon). This should give some more headroom for further cache compaction mechanisms.

Garbage Collection policy and implications

The goal is to remove enough objects to reduce costs to a sustainable level, while minimizing the impact for people using supported NixOS releases. We will have a more precise policy down the line.

As part of this work, we want to preserve all fixed output derivations to the best of our knowledge. That means that old nixpkgs checkouts might rebuild, but the source code needed will still be available from the cache. This is an important details as old projects often disappear.

We also want to keep store paths from current NixOS release channels (but might prune store paths that are part of older channel bumps of ancient NixOS releases).

If you are not using ancient store paths, you should not be affected by those operations.

If you are using ancient store paths, you might have to rebuild them from scratch. Even if the upstream sources were deleted, due to the fixed output derivations being kept in the cache, you should be able to rebuild.

Technical details of the process

We are currently working on a script/policy w.r.t. what to delete and what to keep, and will adjust parameters depending on the storage (and costs) saved.

As described above, we want to preserve all store paths that have no references (in the NAR jargon), which is a superset of almost all fixed output derivations.

We also want to trace the list of NixOS channel bumps and the store paths contained therein, and keep them for active releases (and keep the store paths for the last channel bump for each EOL’ed release).

Call to Action: Save Your Data Responsibly

  1. If there is specific data on cache.nixos.org that you wish to be preserved, we kindly request you to reach out to us directly at foundation@nixos.org.
    We are more than willing to assist you in safeguarding the data without causing unnecessary strain on the cache. Pulling the data from the CDN without coordination can lead to cache thrashing, decreased quality for everyone, and increased operational costs for us, which we aim to avoid.

  2. If cache availability is important to you, please also consider donating to https://opencollective.com/nixos/projects/s3-cache-long-term.

We want to thank all the individuals and companies that donated already.

We also want to thank Amazon that are working with us to offset the invoices by $9000.- every month.

Stay Informed

We will provide regular updates on the progress of the garbage collection in the Announcements category on Discourse.

Thank you for being an integral part of the NixOS community.

65 Likes
2

Do you think it would be possible to only delete objects that we know we can rebuild?
By the way, considering the outrageous egress costs, rebuilding some objects from scratch may be a cheaper option.

3

As it is written:

Theoretically, any derivation as long as their FODs are available should be rebuildable. The other way to ascertain this is to rebuild everything again now without those FODs and filter things, but… well, that’s another cost.

4

So, to be clear, this includes all source tarballs needed to build the final packages, right?

5

Yes, they are fixed output derivations, they are not the input of something else.
If they were input, we could rederive them from their dependencies, e.g. fixed output derivations.

5 Likes
6

Two questions:

  1. Will it be feasible to archive the current state somewhere, e.g. glacier or just manually downloading everything that will be deleted and putting it on a disk somewhere?
  2. Should we expect more GCs in the future, or will it be possible to do the deduplication moving forward?
4 Likes
7

We would love to be able to do this. The issue is how to make this happen with the amount of focus and budget we have.

S3 pricing is surprisingly tricky to navigate with 550TB of data and 800M objects. If we are not careful, we can easily spend 20-50k. For example, did you know that transition costs are charged $0.01 per 1,000 requests? That’s $8k to put all the data in Glacier. Now the storage only costs $2.2k per month instead of $11k or so. But then getting the data out now has a $0.03 per GB and $10.00 per 1,000 requests cost.

If you, or somebody else reading this, have experience modelling this, or would like to get involved, I would love your input. Please contact me so I can organize your access to the data.

Alternatively, if your company depends on the cache, helping cover the costs is also super helpful, and allows us to keep more of the history: S3 Cache Long Term - Open Collective

Probably both. We want to pursue dedupe once we get the cache back to a manageable level of spend. That being said, nixpkgs keeps growing. Even with the dedupe compression, at some point, we have to come up with a GC policy.

15 Likes
8

Could the collectable objects be deny-listed first, so that the impact of deletion on the community can be assessed before the point of no return?

Actually seeing what fails could motivate potential cache sponsors.
If it’s truly gone, there won’t be anything to sponsor anymore.

12 Likes
9

Yes, it’s a good idea and something we have considered. We explored using ACLs for this; it doesn’t scale to the number of objects we have in mind. If you or anybody else wants to help, we will explore those options.

Another thing to know is that part of the condition for Amazon to sponsor us is that we reduce our running costs. Which makes sense. They extended the help when we found ourselves in dire need and didn’t necessarily want to do so forever.

At the end of the day, how much cache we can keep depends on sponsorship, and how effectively we can use it. But it has to be on a sustainable level. We can’t depend on hypothetical donors. Also, thanks to Mercury for donating 10k to the cause. Hopefully, more companies will show up!

7 Likes
10

Maybe send narinfo requests to a lambda that checks against the deny list in a DynamoDB table, or a redis instance (instances?) or would that be even worse? I don’t know the numbers. Maybe Fastly k/v store could be of help?
Regardless, this would be a temporary experiment for like 3 months tops, I suppose.

Unfortunately I don’t have time to spare now. I hope someone else finds this interesting. They may have a significant positive impact on the project.

You’re right that we can’t depend on hypothetical donors. The experiment has to convert them into real donors, and be cancelled if it doesn’t reach a threshold after a certain deadline.

2 Likes
11

You can consider enabling versioning on the bucket. Delete requests in a versioning-enabled bucket hide the object but don’t delete the data. You can purge the deleted versions when ready by creating a lifecycle rule to expire non-current versions. I believe both deletion and expiry are free.

9 Likes
12

I am wondering what’s the impact on Devbox guys offering historic versions of packages like this - Nixhub.io | A Nix Packages Registry

1 Like
13

Great thanks for all work on this problem. I think it is the right move on a path to overall sustainability of the project.

1 Like
14

But I would not expect 800M PUT, this is 50% narinfo (400M), if the idea is to use a more aggregated format less than that.

Others 50% are nar files (400M), but deduplication algorithm may generate less than 400M files. If there is any chance that we are overwriting the same file, we can do HEAD before PUT that will cost the same as GET.

Depending on the migration plan, we do not have to pay all ops in the same month. And depending on how much dedupe from previous month reduced in the total, we can be more aggressive next month.

080M PUT requests x 0,000005  = 0,400.00 USD
160M PUT requests x 0,000005  = 0,800.00 USD
240M PUT requests x 0,000005  = 1,200.00 USD
320M PUT requests x 0,000005  = 1,600.00 USD
640M PUT requests x 0,000005  = 3.200,00 USD

800M GET requests x 0,0000004 = 0,320,00 USD
15

Regarding the policy w.r.t. what to delete and what to keep: how about keeping some cache of older nixos versions? (in addition to FODs of course!)

All NixOS versions since 20.03 supports being used as flake inputs (19.09 also supports being evaluated in pure mode with flake=false;). So it would make sense to find flakes which reference these versions as inputs. Building the flakes from scratch would be prohibitive for most users, but if core packages like stdenv and common libraries are kept in the cache, it would be far easier to reproduce research code and the like.

My idea is twofold for old NixOS versions:

  1. only cache the builds from the final nixos-xx.yy git revision. This means a simple nix flake update will automatically select the supported revision, and we won’t have to keep around duplicate builds per nixos version.
  2. Don’t cache leaf packages and libraries with a just a single reverse dependency. Users should expect to have to build something themselves.

TLDR:

# all outputs built by hydra are available in cache
inputs.nixpkgs-edge.url = "github:NixOS/nixpkgs/nixos-unstable";
inputs.nixpkgs-2311.url = "github:NixOS/nixpkgs/nixos-23.11";
inputs.nixpkgs-2305.url = "github:NixOS/nixpkgs/nixos-23.05";
# only the final git ref is available in cache, sans leaf packages and single-use libraries
inputs.nixpkgs-2211.url = "github:NixOS/nixpkgs/nixos-22.11";
inputs.nixpkgs-2205.url = "github:NixOS/nixpkgs/nixos-22.05";
inputs.nixpkgs-2111.url = "github:NixOS/nixpkgs/nixos-21.11";
inputs.nixpkgs-2105.url = "github:NixOS/nixpkgs/nixos-21.05";
inputs.nixpkgs-2009.url = "github:NixOS/nixpkgs/nixos-20.09";
inputs.nixpkgs-2003.url = "github:NixOS/nixpkgs/nixos-20.03";

The strategy could also be inverted: instead of starting with everything and pruning leaf packages, start from zero and add only the load bearing packages to the whitelist (release-small perhaps?)

5 Likes
16

I know there’s a lot of valid reasons, but the simple deletion of the historical cache data really does not sit right with me. I would definitely support some way to archive the old cache data, even if it is not easily accessible and requires a special foundation request.

It’s true that it’s never been guaranteed, but a lot of the value I see is in that history and I wouldn’t want it just gone.

Some more logistical questions:

  1. Has a list been made of what paths will be deleted? How much space/spend does this save?
  2. Of those paths, have any tests been run that things are safely rebuildable? It would be good to randomly sample things and actually make sure that things can be rebuilt.
  3. What does “superset of almost all fixed output derivations” mean? Are all FODs going to be preserved?
3 Likes
17

@zimbatm any news? How it went? How is going?
I hope you planned to run it on Feb 30th. :sweat_smile:

18

No concrete update yet. @edef is working on automating the narinfo index, and that will be used by the GC script.

I will let you know once we’re ready to do the first run.

1 Like
19

I definitely think there needs to be some more discussion on the GC criteria. It sounds like the index is a prerequisite to determining that, but I don’t think it’s time to delete anything yet without examining the index carefully and seeing what will go.

20

Previous discussions made it quite clear that the cost of egress data is a blocker for moving data out of the cache and to another storage system. But what about store paths that are already distributed among people? This will be more like for recent channel bumps, but I’m also thinking of ISO/install media people have tucked away in drawer etc.
It might be worthwile to keep narinfos of GC’d paths in case another storage solution arises in the future and people are willing to dig up their old storage media.