nix.dev isn’t a community resource - it was donated to the NixOS foundation.
Fair point. I’ve updated the wording.
For caching, this means avoid public caches like cache.nixos.org whenever possible and using trusted private caches like FlakeHub Cache instead. Public caches have lots and lots of good stuff in them, of course, but they’re massive and the security and review practices behind them are not always air tight. When you pull from them to build production systems, you’re working with a porous trust model. We built FlakeHub Cache as a strictly private cache with no public access whatsoever to fill this precise gap in traditional approaches to Nix caching.
Huh, that’s concerning, do you have more about the trustworthiness of the NixOS cache?
It would be of great concern if the cache cannot be trusted. Should I remove it from my secure systems?
(EDIT) There is no cause for concern. If you were concerned, like the people who asked me about this blog post, there is nothing but vague aspersion made at the NixOS project, all to prop-up a VC’s proprietary and private project.
The intended message, AFAICT, is as such:
If (as an organization, mainly) you have special needs that would benefit from a full rebuild of every package involved, you might want to remove the foundation’s cache, otherwise it will be used.
In doing so, you may want to use another cache. Otherwise it gets inconvenient to manage the rebuilds across deployments.
There are different products around managing private caches, and maybe doing more stuff too. There’s also the option of running one “from scratch” for private use. So as always, if you have special needs regarding builds, and don’t know how the NixOS ecosystem can manage those, don’t hesitate to consult, and maybe with more than one VC-backed business that has everything to lose if the adoption of their private platform is insufficient.
DISCLAIMER: I currently have no conflict of interest making these aspersions. I am not currently doing consulting work around the NixOS ecosystem, and am not currently looking for doing so. There are enough NixOS-ecosystem consultants that can help, including ones that will not have my bias, if you so prefer.
You certainly shouldn’t make such claims without proper explanation. It hurts (me) even more on a company blog featuring @edolstra and @grahamc.
The argument is against nixpkgs and not cache.nixos.org when you say “they’re massive and the security and review practices behind them are not always air tight”, right? It may sound like you imply that anyone can push to cache.nixos.org, but of course that’s not true.
Clearly not, as it’s contrasted to the flakehub cache, which also just caches nixpkgs if these are supposed to be comparable.
Fair enough re: “without proper explanation.” This post was intended as a summary of other content that speaks in a bit more depth. I am indeed fully aware that not just anyone can push to c.n.o. and I am fully aware of the existence of Hydra. But to my knowledge, Hydra does push everything to the cache that makes it through Nixpkgs PR review. How do you get a PR accepted in Nixpkgs? What’s the review process? What are the standards? Well, it’s hard to really answer that question in a straightforward way, as different subdomains of Nixpkgs have different practices. Some packages get updated because one maintainer submits a change and another maintainer (perhaps they’re in cahoots!) approves it. So what you encounter is a kind of loose patchwork of standards and practices, leading to a pretty clear provenance problem. To be fair, the likelihood that any given package in c.n.o. is a rat’s nest of Bitcoin miners is extremely small. But if I’m a security-minded organization running NixOS in production, running nix flake update && sudo nixos-rebuild switch amidst this backdrop would certainly keep me awake at night.
And so no, I’m not suggesting that ad-hoc push access to c.n.o. is a problem. But I am suggesting that for larger organizations with stringent provenance requirements, there may indeed be a problem with using Nixpkgs without some set of vigilant practices around that, such as intensely vetting the subset of Nixpkgs you use and pushing that subset to a trusted cache be it FlakeHub Cache or Cachix or whatever rather than simply pulling from c.n.o.
Nixpkgs is a technological marvel and surely not something I think organizations of any level of “seriousness” should simply try to live without. But c.n.o. is a community cache. And that’s 100% fine for a wide range of use cases. But I don’t consider it a best practice for larger organizations to pull directly from c.n.o.
The same arguments hold also for every community distro, don’t they? (Debian, etc.) And it’s about using nixpkgs, not cache.nixos.org in particular.
All of this is true in an organization that employs the flakehub cache to cache things they built which in turn use github:NixOS/nixpkgs in a flake input somewhere. Stating that the issue is with the community cache is a misattribution of where the problem lies.
I can see how the thread could get missing when trying to sell a new product, especially as company execs probably don’t understand the difference, and likely want to hear something that sounds like it’s solving their supply chain security problem, but it is meaningful, because the flakehub cache doesn’t actually do anything about this problem by itself; you also have to build your own custom distro, without depending on nixpkgs at all (which would in turn naturally mean you cannot use the community cache anyway).
You could state that using the community cache implies trusting the NixOS foundation members who have write access and the security of the hydra infrastructure (which is a much higher bar than random nixpkgs contributors), or you could state that using nixpkgs at all has the usual implications of trusting FOSS projects and their contributors, with all the usual implications on supply chain security.
Mixing the two concepts however is simply mis-stating how the ecosystem works, and your post as a result comes off as needlessly alarming (or dangerously naive, depending on who reads it). No, is needlessly alarming, since what it’s stating is false.
I don’t think so. What I’m saying is that there are ways of using Nixpkgs that don’t involve pulling from cache.nixos.org. I specifically laid one out in my response where I suggest, as an example, pushing a vetted subset of Nixpkgs to a private cache.
But the cache is not the problem here. It’s a vetted subset of nixpkgs that gives you what you’re saying. You can do such a review and still use cache.nixos.org to get the property.
Shouldn’t that essentially be the same result due to nix’s reproducibility properties? I know a small subset of packages isn’t reproducible. But imho instead of suggesting companies to use your paid products and rebuild the world again and again, investing money into build compute, it would be a lot better for the ecosystem to push them to help move efforts like trustix forward.
But that wouldn’t help your bottom line, so you rather try to sell a product with questionable marketing strategies.
These caches aren’t inherently trustworthy either, anybody could push to fh or cachix. There’s even less vetting in those cases, the code can literally be from anywhere.
It also reads to me like security-through-obscurity if the only reason fh cache is “more secure” is because the code is hidden. There’s allegedly also ACLs in place if a private cache is set up, but as the code isn’t available, verifying that the fh cache is secure becomes harder. (I’m not asking fh to go open-source, just my observation here.)
You’ve not addressed my question:
Huh, that’s concerning, do you have more about the trustworthiness of the NixOS cache?
Is there any evidence that the cache was tampered with?
Since, I suppose you otherwise know that since the inputs are the same, the same output (or if not bit-for-bit reproducible, the same “equivalent” output) would be produced given the same inputs in a Nix build.
Right? Or is there something you’ve found about Nix where those properties cannot be relied upon, and people would be able to work around the properties of Nix and produce bad store paths in the cache via side-effects outside of a given derivation?
Is there an actual threat model here, and if so, can you describe it? As far as I understand here, this is FUD. Unless there is something notable that everyone other than Determinate Systems are missing.
And as a side-note, you do know that when you’re using Nix, if you’re vetting the Nixpkgs input, and end-up modifying it to fix an “issue” that came from “the community” going rogue and sending bad changes upstream, from the intrinsic properties of Nix, removing that bad change means the inputs differ, and then the outputs differ, causing a rebuild.
So unless there is a severe issue you should have reported to the NixOS foundation regarding trustworthiness of the cache, even your thread model that is actually about Nixpkgs is not relevant to the trustworthiness of the upstream cache.
cc @grahamc, as the CEO, and someone easy to talk to, maybe your have more notes to share regarding this? I am asking with honesty and curiosity. It might be a misunderstanding after all.
For all the issues in the communications surrounding this topic, this is not one of them.
It has to be assumed that as far as pushing/pulling store paths, the use FlakeHub and Cachix as described here would be considered a private cache, where only your vetted builds are being pushed. AFAIK neither of those implicitly makes any store paths from any other users available in “your” cache.
I disagree, it’s store paths in a black box that we hope is secure. Which to my knowledge has not been audited by a third party. It directly addresses their claims that fh is more secure.
Right, and that’s why I said as far as pushing/pulling store paths.
The security and trust in their platform was left as an open question, and is a somewhat different topic to what was implied here.
I guess that the wording “anybody could push to fh or cachix” is the crux of it. As I understood it, I thought it meant that as designed [it’s not] anyone pushing on these services would make the paths available to other caches.
Is it that you meant that the the platform’s trust is unknown, and anyone could be bypassing the design of the systems to put untrusted store paths in place? I don’t know the design of FlakeHub, and might be missing something about Cachix, but AFAIK Nix paths are signed, and in turn, those backdoored store paths would need to be signed correctly for your system to use them, no?
Anyway, that is putting more FUD in the air, I’m not sure it’s helpful at this point to cast FUD around those services, even in retaliation.
If a company is US-based and maybe has ties to US military contractors, should you trust their private cache?
Given it’s so extremely easy to build from source with nix then when you’re in a highly regulated environment it makes sense to build from source instead of trusting cache.nixos.org. this is about risks modelling not about closing concrete security holes.
Why wouldn’t you rebuild everything from scratch if you can then reason exactly about the provenance of the things you depend on? It solves a lot of compliance headaches and moves a lot of the “risk” in house.
Even if we can confidently say that CNO isn’t compromised today , if it happens in the future it completely isolates you from that risk.
That’s a win in my book. And if you can spare the compute to do the builds yourself that’s awesome.
I actually wish companies did this more. We should actively encourage it. Especially if then those companies can contribute to reproducibility statistics and FOD tracking like Lila and Trustix. It would actually make our supply chain security story stronger – not weaker.
Artifact signing takes this risk away no? They can’t meddle with signed artifacts.
Same reason why I’m not worried about cache nixos org being stored by Amazon which has the same concerns in terms of being US based.