Given the complexity IPFS brings to Nix (redesigning quite a bit of internals as well as APIs), I think we should make a case that it has to come with significant benefit to at least outweigh that.
I read something somewhere about people looking into moving cache resolvers - or whatever it is, out of nix core. I think core work like this should be higher priority than it is and would enable a lot more experimentation with interesting backends. Nix is way overdue for some serious refactoring.
All criticisms aside, it will still be interesting to see what you can come up with prototypewise! And then we can reevaluate.
I assume that “trustless” here means that the build server doesn’t have to trust the build clients? The clients still has to trust the build server, right?
Could you shine some light on this quote from your proposal:
What data are you talking about here?
Overall, I find the proposal very interesting. I’m not sure if it is a good idea or not to tie Nix so close to IPFS, but I welcome any experimentation with these things.
The “trust maps” for derivation -> outputs you talk about in your proposal seems similar to what I have implemented for nixbuild.net, where I have to be able to represent several differing builds for the same store path in a safe way. The issue of “trusting the client’s Nix cache” I’ve handled by simply signing all uploads with a user-specific Nix signing key, in addition to storing any signatures provided during the upload. All signatures are then used as a “trust map” when finding out if a path is valid or not for a user. A user can freely decide on which signing keys that should be trusted. Of course, I don’t have to maintain a proper local nix store at the same time, so things are probably a bit easier for me.
Are you planning to publish some design documents somewhere?
The difference using P2P technology is that any downloader can easily become a mirror and mirrors are found automatically. In the case of nix, it would be very nice to share packets in the LAN.
I was already very excited about the idea of using APT with Bittorrent. Unfortunately this has never been adopted.
I think Nix + IPFS is a perfect fit to try the idea out in practice.
I assume that “trustless” here means that the build server doesn’t have to trust the build clients? The clients still has to trust the build server, right?
Exactly.
What data are you talking about here?
The output path, which is computed from hashModuloDrv on the client currently. The trick is to let the server do the final step of taking the imput drv hashes and computing the output path.
The “trust maps” for derivation -> outputs you talk about in your proposal seems similar to what I have implemented for nixbuild.net
Yes, I think the intentional store will be a huge boon for nixbuild.net :). With the intensional store, not only can one choose whether to trust a single build, they can also audit what dependencies (especially non-referenced build-time-only ones) were used by the build.
Are you planning to publish some design documents somewhere?
Of course! We’ll be writing multiple RFCs as the details get fleshed become clear from our initial implementation work.
If Nix archives can be stored as Git object trees, it will inherently allow much more de-duplication of content. => cheaper storage, less bandwidth usage and easy hard-linking. The distribution mechanism is sort of orthogonal to this and it could be applied to zeroconf types of setup as well.
I’m really really excited about this! Nice to see this being tackled!
I agree it might be necessary to refactor some parts in Nix first, and making “cache resolvers/remote store backends” more pluggable in general was something I thought about too.
I wouldn’t say this is a blocker before being able to do any work (at least not for a POC), but I’d consider IPFS, some avahi/zeroconf backend or other delivery mechanisms (maybe Cachix too) as an opportunity to shape the design of such an interface.
This already works today by the way. I currently have both builders and cache set up using zeroconf. However the way nix consults caches isn’t ideal; it seems to be optimised for “there i just 1 cache” and starts to get annyoing when you have like 10
I’ve written about how build results are reused between untrusted users in nixbuild.net, maybe it is of interest to some readers of this thread: Build Reuse in nixbuild.net
Read it; looks good! Yeah CA derivations basically mean the same thing, but since path is content addresssed now the drv<->output mapping must be in the extra table, instead of the input-addressed path <->narhash (or other content address) mapping.
Your system should continue to work with this new stuff, since it already works with fixed-output derivations, and you still retain the advantage of delivering those benefits to the Nix derivations that exist today :).
As a data point, at some point I saw some network with their own cryptocoin supplying distributed storage. - the name of which I don’t remember. It might be interesting to look at how that worked out. I doubt it’s more cost effective than say using some cloud based solution, so this has to stand on it’s own regardless of - or despite storage considerations. *Edit: I.e. people would join the network, supply storage, and get coin for uptime or whatever.
@Ericson2314 do you have a derivation you test with?
I’m adding a new “host” to my nixcfg that is a test VM with an ipfs-enabled nix package. However, my first naive attempt failed: (collapsed now for brevity)
builder for '/nix/store/8zs09r0qy444w2y9d5w8dwnvqwj67c99-nix-2.3.7.drv' failed with exit code 2; last 10 log lines:
unpacking source archive /nix/store/ijd7yg3g642wija13lsz00avwn552258-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
build flags: -j8 -l8 SHELL=/nix/store/6737cq9nvp4k5r70qcgf61004r0l2g3v-bash-4.4-p23/bin/bash profiledir=\$\(out\)/etc/profile.d
GEN Makefile.config
/nix/store/6737cq9nvp4k5r70qcgf61004r0l2g3v-bash-4.4-p23/bin/bash: ./config.status: No such file or directory
make: *** No rule to make target 'config.h', needed by 'precompiled-headers.h.gch'. Stop.
Hi @Ericson2314, I have a few questions about the implementaion that you may be able to clear up.
Is the intent that a significant portion of of Nix installations would also have an IPFS daemon running alongside to provide IPFS connectivity?
Is there any provision in the current roadmap to allow substitution via an IPFS gateway such as the cloudflare gateway?
When running an IPFS daemon and pushing nix store paths into IPFS, is the path added with --nocopy (filestore backed) or are the contents duplicated into ipfs chunk storage? Is this why the store path gets a nix temporary GC root?
When substituting paths from IPFS, are the paths fetched to the ipfs filestore and symlinked/hardlinked into the nix store? Would this create a ipfs pin to prevent cleanup from IPFS?
Regarding the following statement:
Trustless remote building: Nix users shouldn’t need to be trusted to utilize remote builders.
The way I currently understand building with nixpkgs, most derivations are built by a trusted source (hydra) and only packages you’re customizing or developing yourself need to be built locally or a remote builder. For these packages, you don’t know what the result CID will be until you’ve built it for the first time, and if you’ve built it to get the CID, you no longer need a remote builder to do the build for you.
Is my understanding here wrong, or am I missing some use-case that makes trust-less remote building more useful?
That would be nice! But we’re not trying to get ourselves in a network effects chicken-or-egg situation either. In particular, we ought to be able to share source code with upstream devs, other distros, etc., so we are just one part of the wider network.
That isn’t currently implemented, mainly due to time/budget constraints as the gateway and regular interface are different. But there’s no reason all the reading/downloading (not writing/uploading, of course) couldn’t also work.
Duplicated. I had not actually heard of --nocopy, but my inclination is trying to share storage is hard (to wit, --nocopy is still unstable), and long term the better architecture is for IPFS to manage the storage and just mount things for Nix, which is roughly the opposite of --nocopy
We don’t create extra any new temporary or permanent GC roots on the Nix side (LocalStore). We do implement Store::addTempRoot for the IPFS store, so Nix temp roots become IPFS permanent roots. We hope either Nix gain a hook for cleaning up these temporary pins, or IPFS gains a notion of temporary pins, so that we don’t leak pins on the IPFS side.
It’s just copied, same as with the other direction.
Your understanding about knowing the CID and not needing a build coinciding is totally correct.
The current bug is probably simpler/stupider than what you are thinking. Right now, clients to remote builders push arbitrary non-content-addressed data to the remote builder. This is fine for when the client is hydra, but not fine if e.g. a company wants it’s employees to use a build farm without giving any rouge employee the ability to push malware to commonly-queried paths and infect everyone else.
