NixOS Reproducible Builds: minimal installation ISO successfully independently rebuilt

Announcements
13

With 25.11 out, it’s time for an update!

TL;DR: we can still fully reproduce the minimal installation ISO, and are making steady progress in tracking and exposing attestations.

Reproducing the minimal ISO

The process I used for rebuilding the minimal ISO while avoiding reliance on the binary cache as much as possible has not materially changed since the previous update:

Starting the NixOS 20.03 VirtualBox appliance in qemu (needs more memory than before, notably building clang seems to need more than 20g):

$ tar -xvf ~/Downloads/nixos-20.03.1405.a17e021b948-x86_64-linux.ova
$ qemu-img convert -f vmdk -O qcow2 nixos-20.03.1405.a17e021b948-x86_64-linux-disk001.vmdk rb.qcow2
$ qemu-img resize rb.qcow2 150G
$ qemu-system-x86_64 -m 25600 -smp 16 -cpu host -enable-kvm -drive file=rb.qcow2

Then inside the vm (building as root to avoid a problem with using the old nix-daemon for new builds):

# Fetch the build definitions
$ nix-shell -p git
$ git clone https://github.com/nixos/nixpkgs
$ cd nixpkgs

# build the Nix version to build Nix with (with binary cache disabled)
$ git checkout 8e8f31731938
$ nix-shell -p nix -I nixpkgs=/home/demo/nixpkgs --option substitute false

# get the commit to reproduce from
$ git checkout 09eb77e94fa2

# build the Nix version to build the ISO with (with binary cache disabled)
$ sudo nix-build -p nix -I nixpkgs=/home/demo/nixpkgs --option substitute false
$ nix-shell -p nix -I nixpkgs=/home/demo/nixpkgs --option substitute false

# workaround for https://github.com/NixOS/nixpkgs/issues/467866
$ nix-build . -A elfutils

# perform the iso build (with binary cache disabled)
$ sudo nix-build nixos/release-combined.nix -A nixos.iso_minimal.x86_64-linux --option substitute false --max-jobs 6 --arg nixpkgs "{ revCount = 901827+1335; shortRev = \"09eb77e94fa2\"; rev = \"09eb77e94fa25202af8f3e81ddc7353d9970ac1b\"; }" --arg stableBranch true

I was able to take nixpkgs at 09eb77e94fa2 and produce an ISO with sha256sum 57bcfaef17a57fb544cea59b735da569ad7e174e7b4fde31661e3fd4f1815f0f, which matches what was built on Hydra (hit ‘details’ or download the iso and see for yourself).

Limitations

This result greatly improves our confidence that the binary cache has not been tampered with. However, before we can say the ISO has been “fully reproduced” from source, we should still verify if there are any packages in this set that rely on binary upstream releases instead of building from source.

Instead of starting with an old OVA image, it would be great if we could ‘bootstrap’ the whole reproduction environment. This is a whole other project, though, called “bootstrappable builds”. For now the risk that someone in the year 2020 had the foresight to inject malware into the image that’d remain undetected and is somehow effective to poison this build is extremely low.

Other news

Partly supported by the Sovereign Tech Fund, me and @JulienMalka have been working on improving the hash collection infrastructure mentioned in the previous update. This now helps us make better reports giving insight in the reproducibility of the objects in our binary cache, but also unlocks hash verification without having to rely on a single trusted party, as multiple builders can upload their (signed) hashes. It’ll also help doing more continuous rebuilds rather than the one-off checks we did before. We hope to build this out further in the future. You can view live reports at https://reproducibility.nixos.social/.

Where to from now?

Further plans:

  • Continue fixing known nondeterminisms, notably working towards making the Graphical ISO reproducible as well.
  • Continue on hash collection
  • Extend the infrastructure to regularly independently rebuild artifacts, instead of ‘one-off’ checks like the ones described in this thread.
  • Flag which derivations still depend on a binary upstream release instead of being built from source

Want to help?

19 Likes