Perhaps worthwhile to ask if [Github has] plans to increase trust of tarballs.
All for asking, but I think we already have mostly what we really need. Instead of the release tarball, it is possible to use /org/repo/archive/refs/tag/$TAG.tar.gz and ensure that a hash is included so that tags cannot shift. Alternatively we can use /org/repo/archive/$COMMIT.tar.gz for hypothetically greater stability, but then it’s less obvious exactly what that commit is or should be.
Beyond this, I’m not sure the GH can do more. While I think that a soft mandate for archives over source release tarballs would be good, I agree with the detractors to some degree in that the root-of-trust problem is more-or-less unsolvable. When a single person cannot be trusted (in cases like this or considering stolen credentials), you start adding more factors, which in this case would mean more people. So, hypothetically, maybe more people could sign release tarballs (I’m hesitant to think this would help much in practice).
However, I think we already can leverage the “more people” angle by simply pulling archives, because that’s where more people are already looking.
I think the much more challenging problem is package repositories like npm, cargo, hackage, and etc. The bundles here need to be ready for consumption, and therefore can’t necessarily match the source repo (especially cases with code gen or etc). But when it comes to release tarballs of the source, we’re better off just going to the source-iest of source, because that’s nix’s super power: it can do that!
I think the way we obtain sources was of least importance in case of xz: we couldn’t do much on Nixpkgs side of things to prevent this, other than invest more in shorter and more observable bootstrap chains. It’s also a bit disappointing we chose to wait for the builds, I guess that means we’re not ready to use grafting the way Guix does. We wouldn’t have been ready if the vulnerability happened to target NixOS too.
Forgetting about xz, I like @Atemu’s comments, and I think a tighter integration with VCSs is desirable: VCS names put things in context better than fetchurls, allow linking external sources of metadata better than nix’s recursive hashes, etc. On a related note, I think it’s concerning that most our fetchFromGitHub (&c) expressions normally only specify the mutable refs/tags and nix hashes, but not commits.
It’s a catch-22 here. With commits, content can’t change, but it’s more work to really verify that the commit chosen is the commit intended. With tags, we can use hashes to ensure that things don’t move, but it creates an equivalent scenario to sources being dropped and one becomes dependent on binary caches unless the tag change can be reverted. The risk that sources are pulled always exists.
Either way, both have much better guarantees than source tarballs! I don’t think there’s any need to “forget about xz,” this change would have never loaded the stinger that injected the poison—that was only in the release tarball.
I’m game for that! Tag, commit, and then I’d still throw in content sha to protect against things like TLS or VCS subversion.
malicious code can be (and has been in case of xz ) injected in VCS as well
Apologize for belaboring the point, especially since you and I agree on the part that truly matters to me. However, I think it’s again worth noting the subtlety here. The code checked in to VCS was harmless (as far as we know) as is. Even the most careful audit of that code likely would have missed the risk, because the key connective tissue was not there. There was nothing that linked the build and the checked in files together.
When I read the detailed description of the attack,
my first thought was, “well, that’s good news, nix should be immune to that!” It wasn’t based on how sources were fetched, but hindsight is 20/20 and now we know and can leverage this defense.
While this isn’t the be-all-end-all, based on how this particular attack worked, I feel very strongly that these changes would help guard against future ones exactly like it.
We have to specify the recursive NAR hash anyways for Nix.
It was not. We know it was harmful. It was merely inactive and could have realistically been activated without any of us noticing because who of us here audits the code of lzma, zstd, bzip, gzip etc? I sure don’t.
It was mere luck that someone did eventually end up noticing it.
The only reason Nix was not vulnerable is that it wasn’t targeted. It would have been trivial for the attacker to do so if they had wanted to.
We do agree on the facts, and my intention is only to highly them. Malicious code was in VCS. If we pulled directly from VCS that malicious code would have been inactive.
If we were going to pull in malicious code regardless, wouldn’t we prefer it to be inactive?
It would have been trivial for the attacker to do so if they had wanted to.
If the policy was to avoid release tarballs, it would have been a little harder. So let’s make it a little harder for the next one.