And hopefully symmetric bandwidth. Lots of people (like me) have, say, 400Mbps down and 100Mbps up (which, of course, would be serving the files).
1 Like
Hopefully, sure, but that’s just a nice-to-have, and I think a large majority of (home) users doesn’t have symmetric. But I’m a pretty heavy duty user (small team dev infrastructure supporting remote work) with asymmetric (250/40 !) and it all works decently. But indeed the proposed concept would work well/better with a significant number of users (thousands), which should spread upload demand well enough.
2 Likes
Perhaps tailscale and/or tor could be leveraged as well.
Approach governments. French have rolled out Nixos in some education establishments. Russia has banned Windows. Must be state players and educational institutions who see the importance of Nixos. You could offer a teaching resource in exchange for support. Try Nigeria or somewhere. (I am just throwing it out there)
5 Likes
I was just fed an ad for this service. Might be an option. No egress fees. $0.01/GiB/month for 100TiB to <1PiB.
Not sure about public access, etc.
https://www.rsync.net/pricing.html
1 Like
Not sure about a plan for the egress costs to move the data, but Wasabi is a great S3 storage solution for hosting the data. Its $6.99 per TB per month, which at 400 would be roughly $2300 a month.
Wasabi doesnt charge for egress or API calls
For the hosting side, F4 Network and ByteHosting, both offer better hosting prices than Linode, D.O, etc:
Hope this information helps.
1 Like
I haven’t seen much talk in this thread about growth. What can be done to make less data in the future?
Could there be weekly or monthly archived releases of the packages in the rolling channels, with everything in between being thrown away?
I wonder if there’s any deduplication possibilities left with the archival data. Maybe some new file format could be created that’s just a bunch of pointers to binary chunks, that the client can reassemble, and old stuff could be converted over time.
I am a big fan of BitTorrent in general, and it could solve some user side issues even if it doesn’t address storage.
Do you have any reference to the above?
Hi Jos,
Well I saw an article on it; I’ll have to search a bit for it though.
This might stir some interest though:
1 Like
Why not just use Backblaze B2 for object storage? They have an S3-compatible API and their pricing is insanely good: Cloud Storage Pricing Comparison: Amazon S3 vs Azure vs B2 (backblaze.com).
1 Like
For the amount of storage NIX uses, Wasabi S3 would be better suited as their pricing would be $72 per TB per year compared to Blazes $180 per TB per year. Wasabi doesnt charge for egress or API usage.
So, with the amount of storage NIX currently has @ 425 TiB:
Wasabi - $30,549/yr
BB - $76,500/yr
that large amount adds up significantly. And this reply isnt to bash anything, just merely showing the advantage wasabi would have for them as their s3 storage, and even for anyone else needing s3 storage.
I thought that Backblaze B2 cost $5 per TB? At least, that’s what I got from their pricing page. So if you had 425 TiB that’d be $2125/yr? Did I misread the pricing page?
You are correct about the $5 per TB per Month, but you calculated that price for 1 month. The price they give you is per year. So to upload and download 1TB per month for Wasabi, would be $7, backblaze would be $15 or $180 per year per TB. The numbers are obviously based on the data uploaded and downloaded per month which is not going to be a one for one, But this is a good example regardless as touching the data with backblaze costs, where wasabi it will not. They could literally export all 425TiB one month and it wouldnt cost them a penny to do so. Ingress prices always look great, until you add egress and api call pricing, then it adds up.
If we’re serious about the reproducibility goal, the archiving part is definitely important for the cases where the upstream source disappears or is volatile. Packages can depend on each other, not being able to rebuild one package exactly can have a larger effect.
However, I think it makes sense to distinguish both the archiving and the caching goals, since they might be optimized differently.
For the archiving part, it may also be worth trying to work together with one of the existing archiving projects like https://archive.org.
5 Likes
Or better Software Heritage, now that they’re expanding their artifacts archival. It sounds like it would be a research project of its own to figure out the deduplication etc…
4 Likes
This issue would be mitigated by the recent announcement of AWS waiving egress fees when moving out of AWS.
13 Likes
This is a game changer. This changes the entire equation in my opinion.
4 Likes
Note that the free egress is limited to 100GB/month, if we need more (…very likely…) we have to request from Customer Service first:
If you need more than 100 gigabytes of data transfer out per month while transitioning, you can contact AWS Support to ask for free DTO rates for the additional data. It’s necessary to go through support because you make hundreds of millions of data transfers each day, and we generally do not know if the data transferred out to the internet is a normal part of your business or a one-time transfer as part of a switch to another cloud provider or on premises.
2 Likes