To me, now feels like the best time ever to start thinking about decentralizing nix: with the whole NixOS foundation drama, I have no confidence that its infrastructure is going to last long-term and that the community will be able to successfully take over its administration in case it collapses. If NixOS was a decentralized zero-trust system to start with, then its administration would have no power and it would need next to no infrastructure to be maintained, because each willing user would host a part of it.
1 Like
Hypothetically, yes, a collapse would certainly motivate development of decentralized build system. However, this seems unlikely to happen in the foreseeable future. It would be more effective to appeal to something like faster build times.
1 Like
I doubt the current community & infra would fall apart overnight - and I say that as someone who is pretty supportive of the efforts of the various “splinter groups.” There’s a lot of people still ‘here’ and a fair number of companies with some technological investment. I think a reasonable worst case scenario would be a slow decline, which would allow plenty of time to investigate alternatives.
There are already a few people thinking about alternatives, whether centralized or decentralized. It’s also worth noting that even with a centralized approach, different decisions could be made around what to build and what to store that would reduce costs and still be useful (e.g. it’s nice to keep a lot of binary ‘history’ around but not actually necessary for day-to-day use.)
I agree at the moment the lack of immediate necessity probably means people are less motivated to put effort into decentralized solutions, but there’s also stuff like ca-derivations that if fully implemented/deployed would make them substantially easier, so people may be holding fire a bit.
TBH I feel the design of nix pretty goes some way to allowing this already, in the sense that you can just checkout nixpkgs and be able to build everything yourself - it’s just inconvenient, and until recently at least hasn’t seemed necessary.
1 Like
Especially since hydra is run on the most expensive option of them all 
Do you mean CA derivations, more efficient compression, or something else?
Also, IMO we’d benefit from the mirror model, like every other Linux distro, rather than basically half the world being unable to use NixOS effectively due to reliance on Fastly POPs. (I’m aware of the TUNA mirrors, but still…)
AWS? Surprised.
There was some work towards supporting IPFS which would maybe also solve this,
but indeed, even just manual mirrors would work great.
Correction: I’m not sure what hydra runs on, but the binary cache itself is on S3, if I read NixOS Foundation Financial Summary : A Transparent Look into 2023 correctly
2 Likes
There’s mention of Hetzner and Equinix Metal in there too. Hetzner is providing an AArch64 machine. To me, none of that is “overpriced cloud nonsense” (a ridiculous statement anyway).
Of course the NixOS foundation has the prerogative to use whichever infra works for them, but the point above was that running a buildfarm does not have to be tens of thousands of euros per month - using a server you own can be more cost-effective and performant in some cases (especially since most individuals would not be getting thousands in free hosting / bandwidth donated to them). After all, “cloud” is just someone else’s computer 
And we’d have more opportunity to decentralise the infra if people are accurately able to assess the costs/options.
running a buildfarm does not have to be tens of thousands of euros per month
Agreed, it would be interesting to see where greater efficiency can be found within the existing infrastructure.
especially since most individuals would not be getting thousands in free hosting / bandwidth donated to them
Not really sure what you mean by this?
“cloud” is just someone else’s computer
This is an incredibly reductionist description of what cloud computing is…
1 Like
I’m very confident that the infrastructure isn’t going anywhere and a big part of that is why I want to make sure our financials are transparent. The main goal is to make sure we consistently have 2 years of funds to secure any scenario. Regardless to direct funding, there are multiple teams collaborating to ensure we have solid fallback options if needed.
This includes a lot of ongoing work by the Infra team, currently split into a few areas. There’s the the long-term S3 cache effort were it would be interesting to explore decentralization to see if it can support making Nix more sustainable. We also have an experiment running outside of AWS (@edef / @fricklerhandwerk), and I’m working with the AWS open source team to extend our credits (hope to have an update on this in the coming weeks).
Just to reiterate, any work or help is greatly appreciated!
8 Likes
Just brainstorming here; you have been warned 
Would it be reasonable to create a decentralized build system based on a cryptocurrency?
This could give those who use their hardware to build and distribute Nix packages an incentive of sorts. Generally, relying on others’ good will is not sustainable long-term. It would also create a public ledger that provides some form a reputation and history, which may mitigate some security concerns.
Perhaps users can opt to only download binaries built by reputable builders. Perhaps they can opt to pay a fee that would give the package priority and be awarded to the first builder. Perhaps builds could be verified through other builders, for which they would receive a small award.
I realize that the world is getting tired of new coins, but I see a potentially valid use case. My only concern is that the coin must serve an actual purpose, and be safeguarded against financial speculation.
1 Like
What would be the most effective way for the community to participate in this regard, such that we’re not doing work that’s already in motion?
@waffle8946 you could study @flokli’s work on the deduplicating content-addressed store for Tvix. Eventually we’d want to plug that behind the CDN, but someone needs to build a prototype for that and experiment with performance.
1 Like
I appreciate you asking! and I just noticed I left this message on draft! (apologies)
On this topic there’s about 3 related potential threads:
- S3 Cache Long Term effort which @edef is leading with support from @fricklerhandwerk and a few others. (noticed frickler responded above)
- S3 Cache AWS Credits - I’m leading that to get 12 more months of additional runway for us
- Infra team involvement @hexa
4 Likes
None of this has anything to do with finance or cryptocurrency.
The moonshot is fixing the way we deal with shared libraries. We pay an immense disk-space and bandwidth cost every time we rebuild a library which is depended upon by other packages. It is an insane cost. No other distro (except Guix) pays this cost, and it’s getting increasingly impractical.
I’ve said a bit more about how to fix this elsewhere but it’s still in the oven and not done cooking yet.
So long as its symbol table doesn’t change, modifying glibc shouldn’t trigger rebuilds of things downstream of it. It also shouldn’t change the contents of packages downstream of it, except for a tiny intermediate “indirection table”. The only thing that should change in the downstream outpaths is a pointer to that indirection table.
This goes way beyond CA derivations (but works best in combination with them) and better compression. It’s not the same thing as pkgs.replaceDependency but can be used instead of it if the dependency’s symbol table hasn’t changed.
3 Likes
I’ve wanted to apply the same approach to shared libraries, but I’m wondering how you’d intend to accomplish it without CA derivations? You still need a way to turn the new glibc sources into a stubs output, and short‐circuit the rebuild of downstream packages based on the actual hash of that output, right? Dealing with header file changes is also tricky.
LLVM already has a pretty comprehrensive‐looking tool for creating stub shared library files, and of course macOS has been natively doing this kind of thing with .tbd files for years. Though I think you would still need to do a minimal amount of O(n) work across all packages by rewriting the paths in the resulting binaries from the stubs to whatever actual library version you want them to be linked to in the final system. (That’s a lot better than rebuilding the world for a glibc security update, though.)
3 Likes
I’ve thus far presumed that the store hash is based on inputs + content of the derivation (damaged by too much git?).
So the hash of a derivation will not change if I modify the build phases (w/o input changes)?
Build phases, etc. should be considered inputs here; Nix expressions eventually produce a .drv file with the builder, inputs, and variables, and the hash covers all the information in the .drv file. The key point is that it’s not a hash of the build output.
1 Like
Exactly. Levers of power attract unpleasant people and bad behavior. Eliminating those levers is not easy, but the benefits are worth it.
The infra is not the most worrying lever of power, or even a major one. The fact that nixpkgs’ design makes maintaining long-lived forks extremely painful is probably the biggest.[^1] Some of the obstacles to forking nixpkgs include:
- Constant treewide commits cause horrendous merge conflicts. These treewides are often for trivial reasons, like English grammar nitpicks. Nobody is weighing the benefit of this stuff against the cost of causing merge conflicts.
-
The traditional nixpkgs formatting style was designed to minimize merge conflicts, but the new autoformatter is amplifying them instead. This is not “aesthetics”, and it is a serious enough problem that “mumble mumble RFC process” is not a justification. Making merge conflicts (even) worse will make nixpkgs (even) more centralized, which will – in the long run – make the political battles even more fierce.
-
No static types in the Nix language means we have to rely on expensive build success/failure as the only CI, and we can’t check any meaningful assurances given to out-of-tree uses – which means that moving leaf packages out-of-tree means giving up any CI. [^2]
[^1]: I always thought it was madness to try to maintain a long-lived nixpkgs fork, but apparently there is now a group attempting to do that.
[^2]: With a real type system like Haskell’s, typechecking nixpkgs would provide a guarantee of things like “for all well-typed out-of-tree use of this expression, the resulting use will be well-typed”. Right now none of our CI has the ability to express this kind of “forall” quantification – we’re stuck at the bottom level of the arithmetical hierarchy.
PS, why don’t discourse footnotes work here anymore?
5 Likes