Solving the cache.nixos.org problem

Development
1

Hi,

good news everybody.
There is now a fully decentralized, cheap and fast storage network that we can utilize to create a paid, super fast, 100% uptime cache.

It utilize the power of sui, to coordinates storage nodes that store red-stuff encoded chunks of data.
Sui, is more a programmable object database with functions to implement behavior that runs on a worldwide DAG. It basically scales to infinite TPS, depending on the data structures used and has a finality of ~400ms. And cheap, tx cost usually around 0.2$ and can be paid by a third party.

In Walrus, you store data over epochs (1 day). For example, you want to 500 gb for 30 days, then you pay a certain amount of WAL tokens into the storage objects, and the system ensures the data is available and missing chunks restored.

Those who run a walrus node, earn WAL tokens for providing storage.
The data is highly distributed, but can be reassembled very CPU efficient Only 1/3+1 chunk need to arrive for reconstruction.

Data can be stored as read-only or encrypted.
Write is expensive, storage is cheap, read is for free.

  1. Define the storage objects we need and how the blobs and signatures should be stored.
  2. Create a SUI smart contract that accept any coins, uses a aggregator to convert them into WAL tokens and pays the archive.
  3. Create a fast rust daemon that can serve the archive.
    • Add local and s3 caches interface
  4. Create tooling, allow to run custom archives, …

Walrus is currently in testnet

https://tusky.io/ is a dropbox based on walrus and many more are comming.

You can test it out, and get some test sui / test walrus token.

I think this would solve the cache.nixos.org problematic perfectly.

You can run your own gateway in your local network and get a fully saturated link.
NixOS foundation can then work with a CDNs to serve cache.nixos.org distributed

2

Read until there.

Your experimental decentralised data storage (read: extremely complex to get anywhere near usable) that is based around some shitcoin that is very likely just a grift to make someone rich is not a practical solution.

31 Likes
3
1 Like
4

Also check out Arweave.
Odysee is migrating their storage to them. Not sure how much it’ll help

5

Seems like a similar “token” system:

The protocol is secured by an open and decentralized network of miners, who store and replicate data in exchange for AR token rewards.

6

Arwave does not scale and is slow. You pay for 100 or 200 years upfront.
Sia and Filecoin are both slow. Here a even longer list

With Sui and Walrus, we can actually implement cache dropping of packages that have not been requested of a certain amount of time.

Even cooler, we can decentralize the build as well.
We could utilize ika which will go live in the next months to sign packages.
Build Queue is in Sui. People can run their a builder, that gets their jobs from the queue.
Each job is distributed to set of different builders, each builds and signes the output hash of the build and uploads the chunks to walrus.
Once enough identical results are uploaded, the result is signed with a key that resides in the MPC network.

This way, the signing key is distributed among hundreds of entities and can only be used through a contract.

We can build faster through better parallelization.

7

First, it is not mine.

Shitcoin, sure.
I find it fascinating how the whole class of byzantine fault tolerant systems is somehow the evil. I suspect this overreaction comes mostly from people that ignored BTC multiple times and are now angry to have missed out.

That walrus has a token, is the only logical mechanic. You need stake to prevent attacks, and you want participants to earn for providing storage. The WAL token will represent the value of storage of data on a global scale. This way you decouple the price from unrelated values which is a good thing in my opinion.

Not all tokens/coins are a scam - the largest Ponzi schema is inflationary money. Have you never wondered why your parents could build a house and you can’t ?

If you don’t believe in the value of a coin, you can send USDC or other stable coins in 0.5s worldwide for sub cents.
If you prefer to feed the cooperate overlords that constantly attack and harass people for no good reason like paypal,…

8

If you want to understand what they are building and why, very good interview

There are reasons they win research prices nearly every month and ship…

9

I feel arweave is mostly for permanent (long term) storage and walrus is for faster large size files in a blockchain.
What a year to atleast know all these existing :fire:, Walrus, Tigerbeetle, Arweave, Nix …
I hope 2025 helps me explore more :fire:

1 Like
10

Yes, but you can build a permanent storage via a smart contract that feeds the storage objects, which is a much better design.
You could for example stake, collect the rewards and pay with the rewards the storage pool.

Walrus will open source once mainnet launches, which is planned for ~ January. I guess the rust lib will then also be available.

1 Like
11

I’m normally beyond skeptical of ‘crypto’, but a system for rewarding owners for their machines doing actual useful work, e.g. storage, doesn’t seem necessarily inherently cursed … the devil would definitely be in the technical detail though, I don’t have the brain to evaluate that.

If nothing else, given the likely engineering resource required, you’d want to be reasonably sure any given platform had a long-term future.

Having said that, proof of concepts are always good, if anyone wants to get stuck in?

1 Like
12

I can understand doubt about the crypto community. But tokenization is the only way to monetarily incentivize a system without loosing decentralization.

1 Like
13

In fact, I think tokenization of communities can create a very strong binding character.
Since the community controls the token mechanics, it can incentivize participation.

Answering forum questions, committing code, developing code, compiling packages, etc could all be rewarded with the community token.

Then you vote about development, and you burn tokens for voting power. Or you stake your token to earn voting power and the longer you stake, the higher your voting power gets. Something like MIN(tokenyears, token4) - this way being invested long term gives you more power.

Move is actually a pretty nice language to develop in. Most APIs I have seen in blockchains are like eating glass.

We will need to wait until walrus gets open sourced since it contains the rust libs. They say January.

I love crypto, but I’m not a fan of PoW L1 chains like bitcoin. It has advantages, but the waste of energy because the algorithms are so simple is just sad.
I know people that mine bitcoin ethically, like living autonomous and when the batteries are full, they turn on the miner. Or using mining to stabilize the grid in overproduction cases.
I prefer PoS networks, since they are efficient and binding capital as security and inflationary element makes sense. Liquid staking token is your friend…

From what I can see, MystenLabs really tries to build the best distributed development platform on the planet, and their speed of development and research is really impressive. protocol updates for example.
It will also become more distributed in the next months and DOS resistant (SCION).

14

Or it can drive away anyone who has not been exposed to the cryptocurrency ecosystem, has no idea what any of the first-class concepts mean and has little desire to find it out because it looks daunting.

As someone who has been a part of technology selection processes on buying and selling side (completely unrelated to NixOS), I don’t think you picked the right pitching technique.

When one wants to make a choice between technologies, it’s important to focus on the question “what problem are we trying to solve” and (depending on the complexity and comparability of the solutions), produce some metrics that would make the comparison between solutions easy.

It’s also important to add context to technical details when surfacing them to the top level of the comparison. Few people care about the material from which the bearings are made, people do care that the thing has wheels.

Qualitative notions (cheap, expensive) don’t matter without context. An industrial lathe is probably expensive if one is buying it for their hobby use but cheap if it’s just another addition to a machine shop at a factory.

To illustrate, quoting the original post:

You introduce the concepts of the “network”, “sui”(as a programmable(why does it matter in this context?) object database) and “cheap”(unquantified except for a reference to “transaction” but without defining “transaction”) and “walrus”. Is walrus a business entity? Is it a technology? A combination?

Paid

Who pays whom? For what specific action?

super fast

How fast is fast? What specifically is fast? I’m not being facetious, but if we think about which actors are involved in a package cache use, we have:

  1. Entity(1) that downloads and builds packages
  2. Entity(1) uploads packages to durable storage (2)
  3. Packages from storage get distributed to users (3) which could be humans or machines

The “users(3)” is a very heterogenous set (e.g. geography). Would everyone get very download speeds regardless of their location? Would users in the physical vicinity of their nearest “gateway” get good download speeds? How is this achieved?

Write is expensive, storage is cheap, read is for free.

Again, without qualifying those the numbers do not really make sense.

  1. Define the storage objects we need and how the blobs and signatures should be stored.

The “define” part, it’s probably “I need 10T of data accessible at Y speed”, but do the definition knobs actually expose what is needed for cache.nixos.org?

…how the blobs and signatures should be stored.

How should they be stored? What are blobs in this context? What is signed? Who is validating the signatures? Mentioning concepts like this is again exposing implementation details that produces more questions than answers.

  1. Create a SUI smart contract that accept any coins, uses a aggregator to convert them into WAL tokens and pays the archive.

Smart contract? WAL tokens? “Pays the archive”? What is the archive?

  1. Create a fast rust daemon that can serve the archive.
    Add local and s3 caches interface

This sounds very much like “draw the rest of the owl”.

  • Create tooling, allow to run custom archives, …

And this sounds like “draw some friends for the aforementioned owl”.

I’m in favor of being open minded and giving ideas fair shots, but IMO this is not the way to pitch a technology.

14 Likes
15

Things like Seti or Tor relays have existed long before tokens and the only motivation needed for people is to be part of something bigger and maybe the ranking.

In addition attaching voting power to it, is very undemocratic.

5 Likes
16

While it is true that those projects have existed for a long time without monetary incentives does not mean that they are sustainable and can lead to adverse effects like secret services running a significant portion of the TOR network. In our case it is questionable if the required bandwith and storage would be available in all regions, something that a monetary incentive would at least alleviate.

I think for an incentive free decentralized approach the barrier to entry (cost, setup complexity) has to be very low to succeed, like peer to peer serving of the store content of nixos machines.

That being said having hardware belonging to the foundation in a co-location datacenter might be just as cost effective with less technological overhead.

3 Likes
17

Aside from critiquing the jargon-y presentation of the original post (like most crypto schemes seem to use), the main aspect that bears stating is, like any cryptocurrency, its inherent value is 0; all value comes from speculation. Using stablecoin as a defense here doesn’t work, because “stablecoin” isn’t even stable, either. And of course, crypto exchanges have control to misuse the funds stored in said wallets, by say, coercively converting all funds into a currency that they have full control over. Throwing “smart contracts” (which is a fancy way of saying “ad-hoc code that mediates your funds”) into the mix will only serve to open up a new class of vulnerabilities with potentially unfortunate legal, financial and technical implications (which can include things like transferring or deleting all your funds).

At least, with traditional currencies, if a bank cannot pay you back, you have some assurance from the federal governments of most countries where this cache would operate; with crypto, losses are simply your own problem, and you can be stolen from via malicious code, code vulnerabilities, by the exchanges or by the same regulatory features that control most any financial transaction. I think it’s disingenuous to claim that crypto is decentralised in most countries in 2024, nor does it appear to improve the financial position of the NixOS Foundation to throw tens of thousands of euros into yet another such scheme backed by closed-source code.

In any case, it seems like it’s forgotten that there already is a (nearly 5-year old, now) mirror of the nixpkgs cache with (afaik) zero monetary incentive: Announcing TUNA Nix mirror

Sustainability is good to think of; regardless it seems NixOS has a lot to learn from other distros/projects in terms of distribution methods, rather than creating new problems for ourselves with novel, unproven methods.

13 Likes
18

Thank you @poelzi for your suggestion - I think it’s worth to think about decentralized storage (and the nix community already did) - but please stay humble and realistic.

Suggesting a technology that hasn’t even reached mainnet for a large-scale project that needs stable infrastructure - I think we can talk about that in 1 year - earliest… When some other, more experimental projects have adopted the technology, and the network has proven itself in terms of stability and longevity.