Serious question, can we … consider policy about using release tarballs in nixpkgs instead of building from source.
EDIT: For context, this was split out of CVE-2024-3094: Malicious code in xz 5.6.0 and 5.6.1 tarballs - #64 by L-as. The goal is to discuss a potential RFC or nixpkgs policy of using source from authoritative git repos, instead of trusting release tarball artifacts.
What do you mean? The tarball in question contained source, which Nixpkgs built.
Except that it’s not the source. It contains pre-generated, in this case backdoored, assets. It makes changing the source to a non-release rev a pain. I really have a hard time understanding how we find it acceptable.
But the Git repository contains those same assets! Are you proposing just that any file anywhere in a repository or in a tarball that Nixpkgs uses as a source must be human-readable? That seems unusually strict.
That’s not my understanding of the situation at hand. Everything I’ve read indicates that the critical changes to the m4 file were ONLY present in the release tarball. Part of why it avoided earlier detection.
That’s my understanding too, but both the tarball and the repository contained binary assets. The tarball contained the final malicious change and the repository did not, but it could have been the other way around. Both sources are just as difficult to analyze.
The benefit of the autogenerated stuff from autotools is that people don’t have to have autotools installed to build the package.
This is not useful for nixpkgs.
It’s nice to not have to go through the autotools steps in a derivation, if only to speed it up and simplify it. It’s also kinda nice to point to a release tarball as “these people said this exact source code is a good one”. But I really don’t think any of this matters, and I would support a general preference for git revisions rather than release tarballs.
To be clear, I didn’t mean to speak of the binary test assets. I would agree that trying to ban such things would be highly impractical. And of course what I’m proposing doesn’t fundamentally change the notion of auditing what you’re actually building.
But, I still think there’s practical advantages of not using release tarballs. It still makes it much easier to override the source of an autotools-packaged derivation, and I guess I see this as a spectrum:
And to @ElvishJerricco’s comment, (edit: I mean this in agreement with @ElvishJerricco in case its unclear) I just don’t … get it? I guess? It just feels like an archaic holdover.
Oh, if all you meant by ‘build from source’ is ‘(re)run the autotools steps on every build’, I have no objection to that as policy. I just thought it’d be odd to single out distribution-via-tarball as somehow inferior (it was the standard in free software for a long time!) to distribution-via-Git-forge.
Forcing autoreconf may not be enough in this case. The malicious actor seems to have put measures in place to counter a possible autoreconf:
https://lwn.net/ml/debian-devel/ZggQpT5sJlEryTX1@thunder.hadrons.org/
The m4/build-to-host.m4 file in addition to hooking the payload into
the build system, also got its serial number bumped from 3 to 30.And the bigger issue is that «autoreconf -f -i» does not even refresh
the files (as you’d expect from the --force), if the .m4 serial is higher.
So in Debian currently, the gettext.m4 in the tarball does not get
refreshed (still pulling in the malicious build-to-host.m4, which
would not happen with the gettext version from Debian), and if we
updated to a newer gettext then it would not update build-to-host.m4
anyway due to its bumped serial.[…]
On the other side of the coin, is that the malicious actor added
precisely that .m4 file into its .gitignore file (as you’d usually
expect due to the new gettext version pulling that in), so if we were
ignoring changes based on trusting upstream, then that could slip
through if we only use this for checking, unless we repackage or always
clean these before builds:
First: they were, for all the effects and purposes, source. Binary files, certainly, but not less source than, say, the Brave web browser.
Second: text-readable source code can also contain backdoors.
Third: we still do not deal with the issue of hermetic bootstrap. This is the only method we can ensure a “completely” free-from-blobs experience.
Fourth: it would require a huge coordination outside Nixpkgs too. How can we convince projects with binary blobs that those blobs should be avoided, and that those blobs should be generated from sources always when possible?
Off-topic, but for a source-based, general-purpose distro it is a bit pointless.
I don’t agree with this. The tarball did not match the authoritative git sources. I am not talking about binary test assets. I’m talking about the backdoor’d M4 stuff.
It wouldn’t though. All I’m advocating is using authoritative git sources for autotools-based projects instead of “release tarballs”.
Until that backdoor was discovered, everyone supposed both upstreams were interchangeable.
And until this specific day no one truly cared about “authoritative VS mirrored” sources.
Let’s be serious, a truckload of people prefer to look at a GitHub mirror to the detriment of the authoritative git sources.
Further, the developer of the authoritative git sources of XZ was also a developer of the mirror.
Nice, I agree.
But I do not think this will be easy. I remember well the hurdles I had when converting scons from SourceForge tarballs to GitHub tarballs…
You can do the autoreconf in a separate derivation if you want to, in any case we should always fetch from source, and we should verify that the tarball matches the given commit, by calculating tree hash, fetching specified commit data, and checking that they match. The server can make up an invalid commit by using invalid metadata/parent commit, but to do this for a chosen commit (hash) would essentially be finding a valid preimage for it with a specific structure, not unlike what Bitcoin does but essentially with difficulty set to maximum since you control all the output bits of the hash, with the nonce being commit metadata and the tree hash being the block you want to include.
It isn’t perfect but it’s better, ensuring that git clones give you the same source code as the tarball.
FWIW whether this would have solved the xz issue entirely doesn’t need to be true for this to be useful, closing just one vector of attack at minimal cost is concretely useful.
Of course SHA1 (used by git for hashing) is broken but eventually it will use SHA256 and it’s still better than nothing.
If I have time I might do it but working on Nixpkgs is a pain.
Maybe, let’s work more on the top of actual code rather than too much discussion.
Here’s something I have been thinking to do, in a WIP format:
I believe the TODO list is to:
(1) provide a diffoscope version for developers so we can analyze the diffs
(2) provide “normalizers”, e.g. normalize for line endings, normalize for PO translation files, etc, etc. We are bound to have divergence between release tarballs and git sources. Which normalizers are acceptable is an interesting question, ideally, normalizing should not increase the chance of hiding executable code in the wild, but let’s say if someone hides a binary in the “processed” .po translation and load it, it’s annoying.
(3) provide “reproducers”, e.g. ways to reproduce certain generated files — I know that autoconf will leak the exact versions in the generated files, this is a challenge to overcome, not a big problem. For example, .gmo could be reproduced. Then, it also becomes a reproducible build situation too.
(4) sprinkle that slowly and surely to build confidence in our bootstrap chain.
I won’t be able to finish all of that by myself, and I am pretty sure this will much more tractable with focused community efforts and upstream collaboration as I have heard that upstreams such as autoconf would be open to work towards reproducible builds for their stuff or moving away from m4 macros.
For people wanting a diffoscope output: /nix/store/746sj2mdjma4cfz1gq9mh1wymsrqnapi-diffoscope-261/bin/diffoscope --new-file --html-dir /nix/store/2zwg8r96if9i20z677b1pdz3wkrqncr3-check-source-equivalence-xz-vs-xz-5.6.1.tar.bz2 /nix/store/6w6yn8xs3pglswlzvm2i2cn3wd4jvrr3-xz tested/xz-5.6.1 here’s a quickly hacked one (notice the Nix induced noise with the perm bits!).
A release tarball can contain arbitrary data; there is no guarantee that it corresponds to the code in the tagged revision for that release as was demonstrated in xz. That makes it almost a no-brainer for me.
In the normal case however, this amounts to some pre-generated configure script and other autotools baggage.
One major aspect of this topic that I think is commonly overlooked is that this affects reproducibilty: The release tarball contains artifacts (not code!) which are generated from the actual code in an entirely non-reproducible manner.
I think it is worth discarding release tarballs for this reason alone because I am of the opinion that all artifacts should be defined in the terms of a Nix derivation. Reversely, the only fixed-output inputs to a derivation should be code produced by humans that cannot be generated automatically and can therefore never be reproducible.
Related discussion: btrfs-progs: build from source, not release tarball by Atemu · Pull Request #124974 · NixOS/nixpkgs · GitHub
Some further discussion of using git sources over release tarballs:
Pro:
Con:
SOURCE_DATE_EPOCH. With tarballs, the newest file mtime would otherwise be used.I continue to be confused by this focus on release tarballs versus repositories, when the argument seems to center around generated files. Is it not a common pattern to include autotools-generated files in the repository? I believe many projects do it each way. Git repositories don’t automatically protect us from malicious developers; they may even give a false sense of security. Diffing tarballs against the repository won’t catch something that was maliciously inserted into both locations. It seems like we’re groping around for some technical solution we can be seen to have done here, when the real problems are social: Are we getting pre-embargo notices of vulnerabilities? Are we doing enough audits of critical packages? Are our users properly aware of the security consequences of using the unstable channel?