The NixOS Foundation's Call to Action: S3 Costs Require Community Support

Announcements
136

There’s definitely valid arguments here, but I think long-term it would be better to have something like IPFS seeding combined with a smarter nix-store that works with deduplicated blocks instead of with nar files.

I’m confident that this would result in a severalfold reduction in stored size and bandwidth, and hosting doesn’t even need to happen in the cloud (like with bittorrent).

(note that due to the store path hashes, lots of duplicate blocks aren’t detected by sliding window dedupers. So the store paths need to be removed entirely (so variable length doesn’t matter) from the files before deduplicating, and restored after reduplication. See here for more.)

4 Likes
137

I really wish I had more to offer than I do. I don’t know what a solution is going to look like. However, I know what is not a solution, and it is using too much energy in these discussions.

BitTorrent is not a solution as it simply does not address the problem in any meaningful way. BitTorrent addresses distrobution, not storage. Look at @balazs.lengyel answer just above me. He suggests that Nix maintain a seeder. The only way to do that is to have a copy of the seed. Thus you are not addressing the storage issue. You are only addressing a distrobution issue.

I think where people went sideway suggesting BitTorrent may have gone like this:

  • Come up with a buzzword that sounds good - decentralization
  • What is a good decentralized solution? - BitTorrent
  • Suggest BitTorrent as a solution

The issue with that is of couse if you have to maintain a seed to ensure the item remains available, you now have to store both the stuff to be seeded, and the torrent metadata itself. In otherwords, you actually increase your storeage requirement rather than reduce it.

While BitTorrent is a great solution to the problem of distrobution, that is not the issue at hand. Storeage is the problem that really needs a solution.

5 Likes
138

How does this address storage? A cache is a temporary copy of something that must already exist. It does not in any way releave the need for Nix to store the original. You are providing a distrobution solution to a storage issue.

1 Like
139

The way I see it, the main problem is actually three problems.

  1. Storage.
  2. Bandwidth.
  3. Growth.
    Currently, cache.nixos.org has to handle all this by itself and is charged by AWS.

Regarding bandwidth:
Using a peer-to-peer system allows for the distribution of bandwidth across multiple systems. This directly reduces the bandwidth load on cache.nixos.org and therefore reduces costs.

Regarding growth:
As the Nix community grows, so does the demand for the cache. This is a direct relationship and, if left without a solution, could cause failure in our success. Ballooning costs are not what anyone wants to see. But with a peer-to-peer system, as the Nix community grows, so does the cache as more and more people opt in to share their resources.

Finally, to storage:
The cache is not a life-and-death storage system for the Nix community. If a Nix system needs a package which is unavailable at cache.nixos.org, it can build it itself. This is how it comes into existence. However, it is generally faster to download a pre-built package from the cache than compile everything from the source code.
The cache is also very dynamic and is subject to change in packages that are in demand by the community. And the more a cache entry is in demand in a peer-to-peer system, the more copies will be kept on many nodes/mirrors. The more nodes/mirrors holding a package, the more bandwidth is now available for that package.
And visa versa, as a package becomes obsolete, the less nodes/mirrors will hold copies of it until no one holds any copies. This is called organic garbage collection. This directly reflects the community’s needs, instead of the Nix Foundation guessing what package should be garbage collected and which to keep.
Also, a package can return from the dead by nodes/mirrors rebuilding it and sharing again. This is also dynamically and organically done according to the community’s needs.

EDIT: I forgot to mention that all cache.nixos.org needs to maintain is the hashes of builds to verify a build out in the peer-to-peer system is valid. And a hash does not take up much storage. So it is not a pure peer-to-peer system, rather a hybrid validator/peer-to-peer system. This will require the Foundation to choose a solution officially. There are already a few peer-to-peer Nix Binary Cache built, but for this to work, the Foundation must ultimately make the executive decision. We don’t want a fragmented solution. (the Foundation might be able to create an OFFICAL API standard that different peer-to-peer solutions can utilize.)

Finally, I believe a peer-to-peer system is a natural solution for a distributed cache system. If we were talking about a much more sensitive and critical part of the infrastructure, I would tend to agree with you, but in this case, I am fully convinced of my point and have yet to see a genuine counter-point that brings me concern.

Yes, many people think and use decentralisation as a buzzword without knowing the tradeoffs. But here, the tradeoffs are all in the Foundation’s and the Community’s favour.

I prefer that the Nix Foundation spend its donations on talented people to help manage the Nix ecosystem instead of raw compute resources. A far better use of actual money donations if you ask me. I am sure we can all agree leadership/management is what Nix really needs to move from niche to mainstream. (Not a bloated cache system)

I appreciate the criticism of my suggestion, which allowed me to spell out my thoughts more clearly. So thank you, @T313C0mun1s7, for your comments. :heart:

1 Like
140

I think an OFFICIAL working solution is urgent that has direct endorsement from the Foundation. We don’t want a fragmented peer-to-peer system, as that will be troublesome for the community.
Optimization can be looked at later. (unless the implementation is easy to do from the start.)
But I am definitely on-board with the optimization of the cache system!!!
Excellent points.
But for all those who are experimenting with distributed cache systems, I thank you.
You are unofficially named the Nix Scientists! :grin:

141

I just realized there is already a working group on the task.

If you have more questions regarding a peer-to-peer solution, I suggest we continue the discussion there.

2 Likes
143

This perspective on storage neglects two important facts:

  1. Most builds’ hashes are computed from the hashes of their inputs. Only content-addressed and fixed-output derivations will be safe to trust from other builders.
  2. Upstream tarballs can disappear, making rebuilding impossible. Recent example: Remove roy mirror by endgame · Pull Request #237693 · NixOS/nixpkgs · GitHub
2 Likes
144

To adress the non-content-addressed issue, we need a trust db, which maps input hashes to output hashes. It doesn’t matter if the output hash is itself a CA or not, what matters only is that you trust the mapping.

Right now this trust is implicit in the nix binary cache. By moving it to a separate db, each user can have different trust providers, and any binary serving solution can be used.

As an added bonus, you can insert “fake” mappings, like mapping a native compile build input to the output hash for a cross-compiled build. I imagine that would be nice for embedded platforms.

9 Likes
145

According to @matthewcroughan we have a separate URL for tarballs, and the storage is not that big.

It is sad to see a tarball disappear, but this is life. Sometimes stuff is lost.
We just need to ask ourselves one question: Is cache.nixos.org a cache or an archive?
If it is an archive, then my peer-to-peer suggestion doesn’t make sense at all.
If it is a cache, a temporary file storage, then my suggestion fits perfectly.

But to have it be both a cache and an archive is just scope creep and makes any attempt at optimization impossible. Because a cache requires high bandwidth and dynamic temporary storage, but an archive requires high permanent storage and low bandwidth. We must be clear that cache.nixos.org is not for permanent tarball storage. (Even one of the official solutions proposed was to do some garbage collection on cache.nixos.org. That means data was going to be discarded.)

If a tarball disappears but the source code remains, a new tarball can be created, then verified by Hydra, and then a dead package can be rebuilt. (With new hashes, but we will always have packages needing the hashes updated…)

2 Likes
146

Yes, I am a fan of hashing both inputs and outputs. I think a centralized trust db is just what we need. An API validator for all publicly shared builds. So that the trust db can be queried by any Nix system wanting to confirm a build’s validity.
A hash repository shouldn’t take too much storage/bandwidth.

1 Like
147

Using AWS at beginning was mistake. There are much cheaper options out there

1 Like
148

AWS at the beginning was literally free (for us). It’s as cheap as mathematically possible.

5 Likes
149

I read a few people suggesting IPFS, if there is any path for it, Scaleway just opened an IPSFS service: /en/ipfs-pinning/

1 Like
150

Hey Adam,

If you require an official working solution instead of a P2P system to reduce bandwidth cost, then Protocol Labs can help you. I am working on the Decentralized CDN, called Saturn, which has been built by Protocol Labs. We can help you reduce your retrieval cost by around 60% with a very powerful community-run CDN (>2500 nodes with 10 Gbit/s connections globally). As long as your origin files are content-addressed and have a CID, they are automatically available via Saturn if announced on IPNI or DHT using IPFS.
As it seems that your bandwidth cost are much higher than the storage cost, a first step could be to make the origin data content-addressable (you could even keep them on AWS for now) and then retrievable via Saturn.

4 Likes
151

Absolutely. If you use IPFS and store a build in a CAR file, then all the blocks will be deduplicated and can be automatically served by the Saturn CDN which cache misses to any IPFS or Filecoin servers.

2 Likes
152

Hmmmm. I think paying the AWS bill sounds better than using a Ponzi network.

3 Likes
153

I am for stoj as an option, could it be possible for people to run stoj nodes to help fund NixOS as well? Say people have nodes they could host but can’t donate the money directly? We could even make a NixOS module that enables this and sets disk size and bandwidth limit. This would reduce the difficulty to contribute.

Amendment: For note, I’m merely suggesting ways this could be made an even more feasible option.

3 Likes
154

It could actually be interesting in more ways than one, though due to the relative urgency of the matter the mode of “direct S3-compatible replacement” could IMHO be feasible first, especially if Storj were to give favourable conditions.

For the far(ther) future, one could technically consider for Nix (Foundation) to host their own satellites (coordinators of a storj network), and each nix user could (on a voluntary basis) run a modified storagenode that would serve part of the /nix/store (i.e. the ones that are “commonly indexed”) and in parallel storage space that the volunteer contributes and where other peers would upload more “historical” store contents.

Of course the latter idea would require significant implementation effort, but the idea that many participants would by default have a relatively “modern” nix store would be extremely fitting since the majority of traffic is likely generated for upgrades/maintenance, and this part would be by default peer-to-peer when using storj (or a derivative). This would also basically be a “self-seeding” system, in that “the first users of new packages” would get a cache-miss and build it, and after say x (some tens of) people have done so the availability of the package at those peers will be known to the satellite and there would be a cache hit and the package would be downloaded in parallel in shards from mentioned x people.

For (very) old nixpkgs the volunteered extra space would come in, which would be pushed by the nixpkgs cache maintainers (automatically) via storj uplink to said peers, and be accessible in the same way as in the above example.

The modification of storagenode would be necessary to make sure the local /nix/store is published to the satellite in the same way the storagenode’s own storage (where other peers store data) is.

A further precondition for this whole concept to work is that the volunteer’s system has to be permanently online and have a “flat-rate” or at least significant monthly bandwith.

3 Likes
155

And hopefully symmetric bandwidth. Lots of people (like me) have, say, 400Mbps down and 100Mbps up (which, of course, would be serving the files).

1 Like
156

Hopefully, sure, but that’s just a nice-to-have, and I think a large majority of (home) users doesn’t have symmetric. But I’m a pretty heavy duty user (small team dev infrastructure supporting remote work) with asymmetric (250/40 !) and it all works decently. But indeed the proposed concept would work well/better with a significant number of users (thousands), which should spread upload demand well enough.

2 Likes