What I’m thinking is: Sure, it takes a lot of time to build everything, but probably 10% of these are a core that all servers use, then some architectures are more popular than others, etc.
It should be possible to opt-in into some very-early-unstable-head-channel, that guarantees only a core subset built (toolchains and core utilities, possibly only for x86_64), get 90% of the build from the binary cache, then every user and organization can build whatever they are using outside of that core themselves.
Then I’m curious - is throwing more money at the problem scaling well here? As in - in own self interest, if NixOS users (e.g. organizations) funded more build machines, would things go faster, or are there some architectural roadblocks that might make it not as simple?
TL;DR: The commit git.tukaani.org - xz.git/commitdiff hid a . before the definition of a sandbox function, disabling Linux landlock. The long-time maintainer of xz reverted it today.
To me, this means that the first vulnerability that’s become so public was not their sole goal, and even if the build-time issue doesn’t apply, others may. That commit would be in the 5.6 and 5.6.1 releases as well.
Hello NixOS Community !
A source based package and still has a backdoor due to building from a tarball instead of the git repo, so what about all the pre-built binaries in nixpkgs ?.
That actually does exist - see the nixos-unstable-small channel. You can see on https://status.nixos.org/ that nixos-unstable-small updated 11 hours ago to (at this time) 35fde99980eb. And you can see more details if you click through to the hydra job; it builds pretty often.
However, the commit still has to reach master before it any channel even starts trying to pull it in, and the revert #300028 was queued only into staging (because it triggers a mass-rebuild and they didn’t want to block CI for everything else behind it). And it unfortunately didn’t build in staging on the first try (because github took down the repo) so now it’s got to come (with a fixup) through staging-next (see xz-5.6.x is trojaned · Issue #300055 · NixOS/nixpkgs · GitHub).
Going through staging seems defensible given the analysis that nixpkg’s build didn’t actually trigger the vulnerability, but the revert hasn’t taken the fastest path into nixos-unstable-small (which would have been to merge direct into master, blocking the nixpkgs-unstable queue for quite a while). I don’t know of a way it could have been sent only nixos-unstable-small and staging, without triggering mass-rebuilds for nixpkgs-unstable.
We shouldn’t wait for the rebuild, xz, libarchive, all related packages should be removed/marked insecure and pushed to master.
Let people rebuild what they need themselves.
There is reason to believe that older versions are possibly compromised too.
It is not as simple as that. As already mentioned before, xz is part of the bootstrap binaries through that stdenv which means you cannot even write a simple string to $out without first compiling multiple gccs if you have no cache hits. On my system I would need to rebuild close to 14k packages and some of them won’t even compile on my laptop unless I reduce max-jobs to 1.
If the revert would be just pushed to master, no PR created/updated in the next couple of days would succeed its CI checks and if it was pushed to nixos-unstable, then we would prevent people updating their systems.
(Point of order: could the “what kinds of source distribution should nixpkgs accept” discussion please go to a new thread? It’s unrelated to the current remediation efforts.)
xz is part of stdenv (along with a few other tools). It’s needed to unpack .tar.xz tarballs. Virtually everything depends on stdenv. nix why-depends might help you figure they details:
I would estimate it to be at least a couple of days if not a week and I would need to baby sit it and restart a couple of times and that with multiple remote builders available that have 96 cores and 256 GB RAM.
Since the vulnerability is not exploitable at this point, it’s not for me.
This is just a not best case estimation. It is hard to estimate this but it should totally only take 4 days or 6 or 8. It is really hard to tell.
It seems that the main problem here is not that xz (the program) is included in stdenv, but that the xz binary in stdenv is provided by the same derivation that provides liblzma.
I wonder, if it would be possible to make stdenv only expose the binary build tools, without exposing the associated libraries. That way, in the future we could first quickly update liblzma (rebuilding only the packages that actually link to it via buildInputs etc) and then update the xz binary used by stdenv at a later date.
