### Author:  [@Atemu](https://discourse.nixos.org/u/Atemu)
#### Post date: [June 4, 2023, 1:34pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/82 "2023-06-04T13:34:19Z")
> [@nh2](#):
>
> Another concrete suggestion on how to reduce the $32k cost:
>
> Do deduplication using a deduplicating backup software such as `bup` or `bupstash` , before the egress.
That’s impressive deduplication and I’m sure you could even improve it even further by stripping store paths (obviously keeping a tiny record to put them back) but I’m yet to come across a deduplicating archiver that’s _fast_. I highly doubt something like `bup` would be fast enough to serve data in real-time which is a requirement for any binary cache.
---
### Author:  [@NickCao](https://discourse.nixos.org/u/NickCao)
#### Post date: [June 4, 2023, 1:34pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/83 "2023-06-04T13:34:24Z")
I think we can postpone the migration process, and instead have all historical data archived in S3 Glacier Deep Archive for now, it would cost us less than 500USD/mo for 500TB of data. Meanwhile we can have hydra pushing new paths to R2 or other cheaper alternatives and call it a day. This would indeed cause a loss of access to historical data for a brief period, but given the timeline, still reasonable.
Edit: the retrieval free for glacier would still be, massive, but that’s how aws works ¯\_(ツ)\_/¯
---
### Author:  [@NickCao](https://discourse.nixos.org/u/NickCao)
#### Post date: [June 4, 2023, 1:44pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/84 "2023-06-04T13:44:41Z")
And speaking of historical data and its research potentials, we can contact research facilities that have the motivation and ability to store and serve them for us, namely CERN, they did use nix for a while with LHCb[1]. And they are by all means experts in handling HUGE amounts of data for a extended time span.
[1] [lhcb-nix · GitLab](https://gitlab.cern.ch/lhcb-nix)
---
### Author:  [@sternenseemann](https://discourse.nixos.org/u/sternenseemann)
#### Post date: [June 4, 2023, 2:30pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/85 "2023-06-04T14:30:24Z")
> [@domenkozar](#):
>
> Keeping the budget, but garbage-collecting it.
I want to suggest that we leave the possiblity of garbage colleccting the binary cache out of the discussion—at least as far as the short term solution is concerned. Garbage collecting the binary cache is a big problem in two ways:
1. It is a question of policy: Do we want to garbage collect the cache? What do we want to keep? What can be deleted? This should be a community decision which can’t be achieved in a month’s time.
2. It is an engineering problem, i.e. how do we determine what store paths are to be deleted according to our established policy.
Additionally, garbage collecting the binary cache is risky, as we may wind up intentionally or erroneously deleting data we may miss—be it in 2 months or 10 years.
For the policy discussion one problem is that there is not a lot information about the cache available (although this has recently [improved](https://discourse.nixos.org/t/nixos-foundations-financial-summary-a-transparent-look-into-2022/28107/16)). Also it’d invariably involve looking to the future, i.e. how will our cache grow, how will storage costs develop etc.
For the engineering side, the big problem is that the store paths correspond to build recipes dynamically generated by a turing complete language, making it all but trivial to determine all store paths in the cache stemming from a specific nixpkgs revision. Assuming all store paths in the binary cache have been created by `hydra.nixos.org` (is this true for all time?), we have the `/eval//store-paths` endpoint available (from which `store-paths.xz` is generated) as a starting point. That will of course never contain build time only artifacts or intermediate derivations that never show up as a job in Hydra—among those, though, are the most valuable store paths in the binary cache, i.e. patches, sources etc. Even though we have tools to (reliably?) track those down, it becomes more difficult to do so for every historical nixpkgs revision. Additionally there is the question of the completeness of the data Hydra has (what happens to evals associated with deleted jobsets?). (If we were to garbage collect the binary cache, I think we should probably try figuring out what is _safe_ to delete rather than determining what to keep.)
Garbage collection is a long term solution if at all.
* * *
It seems to be that finding a way to deduplicate the (cold) storage or archiving little used data more cost effectively is a similar amount of engineering effort, but less risky and comparatively uncontroversial—while still offering a sizeable cost reduction.
---
### Author:  [@deliciouslytyped](https://discourse.nixos.org/u/deliciouslytyped)
#### Post date: [June 4, 2023, 2:53pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/86 "2023-06-04T14:53:40Z")
See also the apparent difficulty of getting an offline version of nixpkgs for a given version, or whatever was in this thread exactly: [Using NixOS in an isolated environment](https://discourse.nixos.org/t/using-nixos-in-an-isolated-environment/3369) .
---
### Author:  [@matthewcroughan](https://discourse.nixos.org/u/matthewcroughan)
#### Post date: [June 4, 2023, 2:57pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/87 "2023-06-04T14:57:51Z")
It is far from complete, or good, but I made [GitHub - nix-how/marsnix: Taking Nix Offline](https://github.com/nix-how/marsnix) prior to NixCon 2022 to work on Nixpkgs offline, and it worked great. It fetches all of the inputs (FODs) so that it’s possible to recompute the outputs (input-addressed derivations) entirely offline.
---
### Author:  [@patricksjackson](https://discourse.nixos.org/u/patricksjackson)
#### Post date: [June 4, 2023, 4:42pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/88 "2023-06-04T16:42:31Z")
The bandwidth is a smaller part of the cost thanks to the CDN, but there’s engineering work in the Nix client that can be done to help reduce it.
Improving Nix’s ability to handle multiple binary caches would make it easier for users to use a local/alternative cache. Right now companies need to setup their own caching server using external tools, and explicitly workaround issues with Nix to make it usable. For example: [Setting up a private Nix cache for fun and profit](https://www.channable.com/tech/setting-up-a-private-nix-cache-for-fun-and-profit)
I started some of that work in [Allow missing binary caches by default by arcuru · Pull Request #7188 · NixOS/nix · GitHub](https://github.com/NixOS/nix/pull/7188), maybe I’ll go pick it back up.
---
### Author:  [@matthewcroughan](https://discourse.nixos.org/u/matthewcroughan)
#### Post date: [June 4, 2023, 5:23pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/89 "2023-06-04T17:23:55Z")
There is also cachecache made by @cleverca22, which acts as a transparent http caching proxy. Each time a request is made to [cache.nixos.org](http://cache.nixos.org) that output will be cached on the LAN on the server running cachecache. Now imagine if everyone ran a cachecache on 127.0.0.1, that’d be a cachecachecache and it’d be almost peer to peer. Now imagine if people recursively looked up eachother’s cachecache’s like dnsmasq recursive resolving.
> **[GitHub - cleverca22/cachecache](https://github.com/cleverca22/cachecache/)**
>
> Contribute to cleverca22/cachecache development by creating an account on GitHub.
---
### Author:  [@Solene](https://discourse.nixos.org/u/Solene)
#### Post date: [June 4, 2023, 5:51pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/90 "2023-06-04T17:51:39Z")
This only help the distribution part, which is already solved by the free CDN from fastly.
You still need to store all the stuff upstream.
---
### Author:  [@nh2](https://discourse.nixos.org/u/nh2)
#### Post date: [June 4, 2023, 6:19pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/91 "2023-06-04T18:19:25Z")
> I’m yet to come across a deduplicating archiver that’s _fast_
From my link, `bupstash` deduplicates at 500 MB/s. It’s multi-threaded Rust, vs `bup` being Python.
This means 500 TB would take 11 days.
> I highly doubt something like `bup` would be fast enough to serve data in real-time which is a requirement for any binary cache.
Note this is not what I’m suggesting in this thread. While I’m interested in using deduplication for binary cache storing in general at a later time, here I am only suggesting to use it to reduce the amount of one-time data egress from S3.
---
### Author:  [@dinvlad](https://discourse.nixos.org/u/dinvlad)
#### Post date: [June 4, 2023, 6:38pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/92 "2023-06-04T18:38:55Z")
I think we should also keep in mind that
1. S3 charges per the number of requests, at $0.4/1M requests in Standard and $1/1M for Infrequent Access and
2. S3 request IOPS can be slower than with a regular disk.
If deduping makes multiple requests to the same S3 objects, that could quickly add up costs. Currently there’re 667M objects, meaning just reading all objects once would likely cost in the proximity of $500-$1000.
So both to minimize the cost and time, it would probably be beneficial to do it on a large instance with tons of EBS storage, which would “cache the cache” for all subsequent computation.
---
### Author:  [@nh2](https://discourse.nixos.org/u/nh2)
#### Post date: [June 4, 2023, 9:47pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/93 "2023-06-04T21:47:51Z")
> [@dinvlad](#):
>
> just reading all objects once would likely cost in the proximity of $500-$1000
Yes.
> [@dinvlad](#):
>
> it would probably be beneficial to do it on a large instance with tons of EBS storage, which would “cache the cache” for all subsequent computation
This should not be necessary; modern dedup backup tools can do the dedup directly against a remote server.
---
### Author:  [@dinvlad](https://discourse.nixos.org/u/dinvlad)
#### Post date: [June 4, 2023, 10:18pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/94 "2023-06-04T22:18:15Z")
> [@nh2](#):
>
> This should not be necessary; modern dedup backup tools can do the dedup directly against a remote server.
I think we agree on that, my point is more about the time and cost saved though, especially if the dedup tools (like `bupstash` or even `restic`) can access the same object multiple times.
---
### Author:  [@zhaofengli](https://discourse.nixos.org/u/zhaofengli)
#### Post date: [June 4, 2023, 10:31pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/95 "2023-06-04T22:31:39Z")
> [@Atemu](#):
>
> I highly doubt something like `bup` would be fast enough to serve data in real-time which is a requirement for any binary cache.
NAR reconstruction can be simple to do (streaming the concatenation of each chunk one by one) and the resulting NAR can be cached as well so subsequent accesses can be as fast as the current setup.
I’m currently doing this on my personal Attic setup, with a Cloudflare Worker controlling the caching of the reconstructed NARs. The NAR reconstruction itself can even be done on those edge FaaS services with CDN integration like Cloudflare Workers or Fastly Compute@Edge, so there are fewer potential bottlenecks.
---
### Author:  [@ron](https://discourse.nixos.org/u/ron)
#### Post date: [June 5, 2023, 4:43am UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/96 "2023-06-05T04:43:40Z")
While we keep the general discussion here and on the matrix channel, we are starting to collect all ideas/options/etc. in a more structured format for review in the github issue below. Splitting them between the near term options and the longer term options.
[[Short Term Strategy and Priorities] Migration of S3 Bucket Payments to Foundation · Issue #82 · NixOS/foundation (github.com)](https://github.com/NixOS/foundation/issues/82)
[[Long Term Strategy and Priorities] Migration of S3 Bucket Payments to Foundation · Issue #86 · NixOS/foundation (github.com)](https://github.com/NixOS/foundation/issues/86)
---
### Author:  [@twoolie](https://discourse.nixos.org/u/twoolie)
#### Post date: [June 5, 2023, 7:18am UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/97 "2023-06-05T07:18:06Z")
I haven’t had the time to read every comment, but it seems clear that there will need to be some trimming of the working set. IMHO the easiest to delete would be historical install/live ISOs. They’re big, trivial to rebuild from the cache, and almost never substituted from cache.
---
### Author:  [@Federico](https://discourse.nixos.org/u/Federico)
#### Post date: [June 5, 2023, 1:43pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/98 "2023-06-05T13:43:23Z")
Backblaze is excellent. I’ve used B2 at work for storage of over 100 TB of data, but I’ve not tested actually serving those files to the public from the buckets.
I forgot that they have a [specific offer](https://www.backblaze.com/b2/solutions/datatransfer/cloud-to-cloud.html) for this kind of transfer:
> Backblaze covers data transfer and any legacy vendor egress fees on \>50TB committed contracts and all sizes of B2 Reserve bundles.
See also [How to Migrate From Block to Object Storage: Solving SaaS What-Ifs](https://www.backblaze.com/blog/how-to-migrate-from-block-to-object-storage-solving-saas-what-ifs/).
---
### Author:  [@alper](https://discourse.nixos.org/u/alper)
#### Post date: [June 5, 2023, 4:31pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/99 "2023-06-05T16:31:10Z")
> [@ron](#):
>
> Splitting them between the near term options and the longer term options.
Reading through the options, a lot of the suggestions are wildly unproven. The main considerations with a volunteer organization like this should be that when you move, you move for a good period of time and the place you move to is as easy to operate as possible (within your budget).
With the financial reserves, the short term solution (3-6 months) should probably be to stay on S3 with some tweaks and eat the 9k/month. If that 9k can be significantly reduced with some of the measures discussed here, that becomes even more attractive.
For longer term I would suggest doing a lot of due diligence before actually moving anywhere.
---
### Author:  [@ac-fpost](https://discourse.nixos.org/u/ac-fpost)
#### Post date: [June 5, 2023, 7:54pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/100 "2023-06-05T19:54:56Z")
Hello all,
To be up front I work for the company I am going to suggest. American Cloud, [https://americancloud.com](https://americancloud.com), does not charge egress fees and we offer an Object Storage service similar to the S3 experience. We also would be more than happy to assist in the migration for free. If you are interested please email at [frank@americancloud.io](mailto:frank@americancloud.io) and I will get a meeting scheduled with our CEO, COO and CTO.
---
### Author:  [@UefiPls](https://discourse.nixos.org/u/UefiPls)
#### Post date: [June 5, 2023, 9:06pm UTC](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/101 "2023-06-05T21:06:34Z")
Maybe it’s time to really consider investing in IPFS. It was announced a while ago [Obsidian Systems is excited to bring IPFS support to Nix - #47 by Ericson2314](https://discourse.nixos.org/t/obsidian-systems-is-excited-to-bring-ipfs-support-to-nix/7375/47) but I assume it wasn’t important until now. There are probably quite a few people who wouldn’t mind hosting or spinning up an instance to host a part of the cache.
If it were as simple as something like the config below, I’d do it. I can easily set aside BTRFS volume of a few gigabyte to contribute to the cache.
```nix
nixpkgsCache = {
contribution = {
enable = true;
backend = "ipfs";
amount = lib.percentage "100";
location = "/mnt/nixpkgsDisk/";
}'
};
```
It could also be an option to have an option that allows sharing only the packages one has installed in order to save disk-space. I’d be up for that. It would also make it interesting to organizations that want to spin up a multiple instances of nixos machines in different datacenters.
I think it would also make it much easier for the community to contribute: “activate this flag and you contribute” vs “please donate money into this pot, it’s probably going where we say it is”.
[Previous page](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672.md?page=4)
[Next page](https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672.md?page=6)