This could be done gradually by package maintainers wishing to help clean up the cache if it’s a pressing matter.
Perhaps documenting the process could be great for preservation efforts.
Wasn’t someone working on an analyser to list dead sources in cache ?
If this storage network loses the last copy of an artifact, it has failed. Rebuilding derivations that are bit-for-bit reproducible so that the hydra.nixos.org signature still validates is a neat trick, and might be used in a sophisticated way to set the redundancy target of different artifacts (either as fun tuning and tinkering after everything else is working, or as a response to volunteer capacity being too low to support the full archive at full redundancy), but I think we should first attempt to find/assemble a mechanism that Just Works as a simple storage/serving service.
I was kind of hoping that we could make a thing out of:
I was also hoping the IPFS community would just have a component that does #1, either ready to go or in development, but I don’t see one. I bet the IPFS community would love to have such a component. Maybe we could build it together.
Edit: There is an IPFS component for #1: ipfs-cluster! Specifically, the collaborative clusters feature.
So maybe the only new functionality we need to implement is IPFS fetching and parallel fetching in nix? See also Obsidian Systems is excited to bring IPFS support to Nix
Could Tahoe-LAFS be suitable? It is designed to handle sharding, distribution, redundancy and more. Although, probably it does not scale to a world-wide network. (Managing redundant shard distribution for resiliency and performance.) That problem might be solvable by using region-based mirrors of Tahoe filesystems, which would provide even more resilient redundancy.
Tahoe-LAFS is designed with a relatively stable set of nodes in mind. We have another ongoing effort to self-host the cache on bare metal and we might use it there.
That’s a bit of the problem regarding availability on a full P2P network; if you can’t reason about the probability of the nodes going down, the only way to compensate is to increase the number of duplicates, which then increases the total cost of storage proportionally.
Where P2P could truly shine is by becoming part of the distribution network. Even if the discovery only happens on the local network, it could help organizations and clusters to both get the NAR files faster and also save up on Internet bandwidth. This is something Microsoft is also using to distribute Windows updates.
The scheme doesn’t need to be too fancy either; have hosts discover each other through rendez-vous on the local network, and then query all of the hosts with a set of requested hashes. It’s boring, and probably quite effective.
I only just noticed this thread. Here is my two-cents in reply to another post page.
TLDR: I fully believe in a hybrid validator/peer-to-peer solution.
The cache.nixos.org is the validator which keeps copies of official build hashes.
The peer-to-peer system dynamically supplies the storage and bandwidth.
In the end we need a real-world experiment to start doing this and collect data on how well the system runs.
Seeders easily opt into the swarm thanks to a new
services.nix-serve-p2p.enable: bool = falseNix option enabling a service to join the swarm. This could be added as a comment in the generated config to raise awareness and help it more easily gain traction.
I think first we should start with an experimental feature: services.experimental.nix-serve-p2p.enable: bool = false
Real-world testing can help us focus our efforts in the correct direction.
In the short-term cache.nixos.org will have to keep holding on to all its data until we can see how the peer-to-peer system really works in the real world. A slow transition is most likely the only way forward. But it is nice to see we have 12 months to breathe easy:
NixOS S3 Short Term Resolution!
I believe there is enough community support to start the experiment.
We can easily talk about it, but a lot of issues raised might not even be much of a problem when the rubber meets the road…
I just had a thought regarding the storage guarantee.
Perhaps cache.nixos.org should monitor the health of a given package in the peer-to-peer system. A dynamic garbage collection model.
If there are less than X nodes holding a given package, then cache.nixos.org must retain a full copy.
But if the nodes for a given package are really high, then cache.nixos.org can do a garbage collection on itself for that package, as it is confident that it is fully available in the wild.
Then if the node counts start to drop, it can require that package for archiving purposes.
Yes, there is a small risk of losing a package for good, but that requires losing everything: the cache, the source URL going down and the original source code being lost and no one has an old copy to reupload to a new URL. Such a failure cannot really be our fault. This is just life. Sometimes things get lost with no way of recovering them. In this modern age were everything “needs to be preserved for the historical record” can be at times simply sacrificing the future for the past. Not a good way to live. Not a good way to run a cache. (A temporary file storage).
We are not the archive of the world.
Regarding IPFS, I’d link some info about the last experiment I’m aware of:
https://github.com/NixIPFS/nixipfs-scripts/issues/11#issuecomment-1520711673
TLDR: I fully believe in a hybrid validator/peer-to-peer solution.
The cache.nixos.org is the validator which keeps copies of official build hashes.
The peer-to-peer system dynamically supplies the storage and bandwidth.
I like this model very much. It makes sense to not try to solve the problems around distributing storage and trust at the same time. If we keep the “centralized trust” model of cache.nixos.org, much of the infrastructure (ie Hydra) can remain basically the same. Once the p2p distribution of nar-files (based on their hashes) is working, nothing is stopping us from also work on distributing the trust in some way (like CA-derivations, trusted builders, Trustix, etc).
I’d like to offer up the idea of a Trust DB here. Right now the mapping from input hash to NAR file is implicit in the nixos cache.
However, by simply publishing a mapping of input hash → content hash, a user can choose to trust a certain publisher, and then it doesn’t matter what cache system you use, the hash can be verified locally.
This is orthogonal to any other solutions. Having the outputs be CA is nice but not required. You don’t need to know what the input hash means, what attribute makes it, etc.
However, by simply publishing a mapping of input hash → content hash, a user can choose to trust a certain publisher, and then it doesn’t matter what cache system you use, the hash can be verified locally.
Is this not just the narinfo-files? When substituting, Nix already works in two phases. First it fetches the narinfo file for a store path from a substituter (cache.nixos.org). The narinfo contains the content hash of the nar and the url where the nar can be found. In the second phase, Nix fetches the nar-file itself. The url is always assumed to be a sub-path of the substituter’s domain, but you could imagine just querying some p2p network for the content hash in that second phase instead.
Nix currently has 2 ways to address a single “thing”.
In general, from what I understood, trustix wanted to provide the software and infrastructure to provide a decentralised “web of trust” for IA->CA mappings, but when I asked in the matrix how to set up things, it was pointed out to me, that the project is mostly dead.
Well, the narinfo provides the content hash (called the “nar hash”). This is just the sha256sum of the nar file. The narinfo can also contain a signature which is based on the store path, the nar hash, the referred store paths and the private key of the substitutor. So I’d argue that cache.nixos.org is already a trust database. Just fetch the narinfo for the store path you are interested in, verify its signature and then you are free to fetch the nar file corresponding to the nar hash from whatever storage provider you want. Of course, you then also have to validate that the nar hash matches what you receive.
Yes, it definitely is a trust DB, regardless of considering CA substitution or not. Though a definitiv problem in this scenario is, that this trust is coupled to substitution, and there is currently no way to ask one authority for a mapping, and another for the content.
The substitution process doesn’t allow to seperate these concerns from my understanding.
In general the substitution process is rather limited from what I can tell, and really only considers substitutes. It does not ask configured remote builders if they have a “product” available.
Instead, if everything goes south, remote A will happily build from scratch what builder B already has, just because of a race in the queue…
Not quite related to the topic of the thread though… Or perhaps it is, and the substitution and remote and local build should be unified into a single abstraction rather than 5?
It doesn’t explicitly allow to separate these concerns today, but it is already making two separate requests, one for the narinfo and one for the nar. So adding support for additional “nar providers” wouldn’t be a big thing, conceptually. The signature verification and configuration of trusted keys would remain as it is today.
In general the substitution process is rather limited from what I can tell, and really only considers substitutes. It does not ask configured remote builders if they have a “product” available.
Yeah this not optimal, and somewhat confusing. However, the trust model Nix uses for remote builders is different than the one for substituters. When using a remote builder it is generally the coordinating machine that handles signing of the resulting store paths. So while builds might be available on remote builders, they could be missing the required signatures. I would love to see some unification in that area…
Not quite related to the topic of the thread though… Or perhaps it is, and the substitution and remote and local build should be unified into a single abstraction rather than 5?
Yes! Internally in Nix, these things are actually already somewhat unified, with the Store abstraction. However, I want to see that abstraction also on the outside, so you could mix and match different “stores” used for building, fetching, storing. See this Nix issue: Separate stores for evaluation, building and storing the result · Issue #5025 · NixOS/nix · GitHub
I must say I didn’t consider narinfo files, they indeed contain all the information for a trust DB.
Now I wonder what their overhead is, currently. Suppose we have an efficient way to distribute trust DB deltas, and you can join multiple providers, and each provider signs the deltas, how much local storage would you need to store these mappings?
And supposing we keep the trust DB mostly as an online thing, it looks like each narinfo is a separate HTTP request, would it make sense to allow bulk requests?
Has a working group been established? From what I understand John Ericoson did a lot of important work with content-addressable derivations and presented it about 3 years ago.
I’m not sure what’s left to do, but from a novice perspective, it seems like the ground work has been done to provide a service agnostic backend to retrieve packages from peers be if over TahoeLAFS, IPFS, bittorrent, or something else.
I don’t think it has. In all channels (including this one) I’ve seen just what I’d call initial brainstorming.
I believe we need official guidance from the Foundation to continue with this goal.
Otherwise we remain fragmented with different solutions.
A member of the Foundation should be appointed to oversee the direction of this working group.
Just need someone to read all the comments and review the current proposed solutions, then choose the next path forward.
Without an executive decision we are just going to continue pursuing different solutions.
But multiple fragmented peer-to-peer solutions is sub-optimal for the community and the mission.
A narinfo can refer to NAR’s via an absolute URL, thus allowing one to host the contents somewhere else. It is currently limited to a single URL, but the format allows for extension to support “URLs” or “Mirrors”.