Signed release ISOs

How difficult would it be to get the release ISOs signed and have that available?

Debian, Arch, and many other distros have that in addition to the checksum for the ISOs.

I know we have something for nars and whatnot, but the ISO would be helpful to have.

This isn’t to comment on your actual request, more an exploration of the current Nix infra and reporting back what I found on a bit of deep dive.

I got curious and found that basically the ISO is indirectly signed by the cache right now through the NAR mechanisms you mention. As long as you trust Hydra, it’s relatively trivial to check the web interface and ensure it reports a matching SHA256 hash for the ISO between hydra and the NixOS download page.

The explanation to do that is under 1 fold.

If you want a deeper explanation of how to verify that signature “locally”, there be dragons and you can unfold the nested fold. Based on Where are the cryptographic signatures? (SHA256SUMS, Release.gpg) - #7 by srd424, A Nix Binary Cache Specification | Farid Zakaria’s Blog and some independent searching/source grepping.

So with apologies for the wall (click to unfold):

1. The download page link tells you info about the ISO

From the NixOS download page, if we click on the SHA256 link (https://channels.nixos.org/nixos-26.05/latest-nixos-graphical-x86_64-linux.iso.sha256), it redirects to the current SHA256. Right now, that shows:

28e527f54815b2f8f93424ae43f78402fcb21aae42577e88c4acdfb1c2a2f984 nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso

So I know that the current graphical iso is for release 26.05 and is an evaluation with nixpkgs 6b316287bae2 and is for x86_64. The hash is reported as 28e527f54815b2f8f93424ae43f78402fcb21aae42577e88c4acdfb1c2a2f984.

2. That ISO is built by Hydra

From there, I go to hydra.nixos.org and click on the nixos:release-26.05 jobset. I find that evaluation 1825934 was the build for 26.05 with nixpkgs 6b31…e2 and then search the jobs for “iso”, which shows me

Succeeded 	330772285 	nixos.iso_graphical.aarch64-linux 	3d ago 	nixos-graphical-26.05.1183.6b316287bae2-aarch64-linux.iso 	aarch64-linux
Succeeded 	330772286 	nixos.iso_graphical.x86_64-linux 	3d ago 	nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso 	x86_64-linux
Succeeded 	330772277 	nixos.iso_minimal.aarch64-linux 	3d ago 	nixos-minimal-26.05.1183.6b316287bae2-aarch64-linux.iso 	aarch64-linux
Succeeded 	330772284 	nixos.iso_minimal.x86_64-linux 	3d ago 	nixos-minimal-26.05.1183.6b316287bae2-x86_64-linux.iso 	x86_64-linux

Checking build 330772286 for the graphical.x86_64 build, I see the ISO build product. If I click on the grey details button (NOT THE “details” tab), then I confirm the hash matches 28e527f54815b2f8f93424ae43f78402fcb21aae42577e88c4acdfb1c2a2f984.

**(if you trust hydra and value your sanity, stop here)**

3. The hydra build is cached on cache.nixos.org substituter, with a lot of information in the narinfo including a signature

The grey details button also shows a “full path” /nix/store/psbv0ym1g24lsi8sa36b03vl4cc72zd4-nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso. That means the truncated store path hash is psbv0ym1g24lsi8sa36b03vl4cc72zd4.

This gets cached on cache.nixos.org. You go to cache.nixos.org/psbv0ym1g24lsi8sa36b03vl4cc72zd4.narinfo:

StorePath: /nix/store/psbv0ym1g24lsi8sa36b03vl4cc72zd4-nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso
URL: nar/0m5hg5a9jrixl26abvk1gck5vnrqq7dssvi8sj1yhwjklibm3ljz.nar.xz
Compression: xz
FileHash: sha256:0m5hg5a9jrixl26abvk1gck5vnrqq7dssvi8sj1yhwjklibm3ljz
FileSize: 3752187652
NarHash: sha256:151x1r0kij4qjsakp90f8d2dggr7vvq3h4kxp78b7a0xk3k6757h
NarSize: 3800990944
References: 
Deriver: xj186bbh8gzw8zjyxshclhkv820qnad3-nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso.drv
Sig: cache.nixos.org-1:uAL1gHWlhWX3iFXtZPRQePzWiM17AJ02BBkwpytDTHoAPVmK+yQavq/RH9LiaKR3NCfCPELajLrAWBIJHyFnDQ==

We see a signature, but we don’t yet know what it signs exactly.

4. The ISO within the NAR archive does match the SHA256 on the NixOS download page

I can download the iso by nix-store --realise /nix/store/psbv0ym1g24lsi8sa36b03vl4cc72zd4-nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso.

sha256sum /nix/store/psbv0ym1g24lsi8sa36b03vl4cc72zd4-nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso/iso/nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso confirms the SHA256 matches if you didn’t want to just trust the reported hash from the Hydra web interface

5. The cache public key is in NixOS/nixpkgs

From nixos/modules/config/nix.nix, it is defined

nix.settings = {
      trusted-public-keys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" ];
      trusted-users = [ "root" ];
      substituters = mkAfter [ "https://cache.nixos.org/" ];
      system-features = defaultSystemFeatures;
    };

6. Verifying the signature trivially

If you used nix-store --realise to download from the cache, I’m pretty sure it both verifies the narinfo Signature verifies the narHash and that the narHash matches the resulting store path.

I believe nix-store --verify-path /nix/store/psbv0ym1g24lsi8sa36b03vl4cc72zd4-nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso also does these checks, but I’m not sure if it does both the narHash matches store check AND the narHash is signed check.

7. Nontrivial verification

7a. Verifying the narHash matches the nix store

To do this explicitly, first check that the store path hash matches the narHash:

  • nix-hash --type sha256 --base32 /nix/store/psbv0ym1g24lsi8sa36b03vl4cc72zd4-nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso/ returns 151x1r0kij4qjsakp90f8d2dggr7vvq3h4kxp78b7a0xk3k6757h, matching the narHash.
  • This can also be confirmed another way by dumping the nar file, taking the sha256 and converting it to base32 as nix-store --dump /nix/store/path > narFile.nar; nix-hash --to-base32 --type sha256 $(sha256sum narFile.nar | cut -f 1 -d' ').

7b. Verifying the narHash is signed

What is actually signed in the narinfo is the narinfo fingerprint. The fingerprint format is defined in [hydra]/subprojects/nix-perl/lib/Nix/Manifest.pm as return "1;" . $storePath . ";" . $narHash . ";" . $narSize . ";" . join(",", @{$references});

That means that the fingerprint here is: 1;/nix/store/psbv0ym1g24lsi8sa36b03vl4cc72zd4-nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso;sha256:151x1r0kij4qjsakp90f8d2dggr7vvq3h4kxp78b7a0xk3k6757h;3800990944;. (note the trailing semicolon since there are no references)

It’s actually super annoying to check if the signature matches. For me, I had to have Gemini get me halfway to a verification script and clean it up:

import base64
from cryptography.hazmat.primitives.asymmetric import ed25519

pubkey_string = "6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
sig_string = "uAL1gHWlhWX3iFXtZPRQePzWiM17AJ02BBkwpytDTHoAPVmK+yQavq/RH9LiaKR3NCfCPELajLrAWBIJHyFnDQ=="
fingerprint = "1;/nix/store/psbv0ym1g24lsi8sa36b03vl4cc72zd4-nixos-graphical-26.05.1183.6b316287bae2-x86_64-linux.iso;sha256:151x1r0kij4qjsakp90f8d2dggr7vvq3h4kxp78b7a0xk3k6757h;3800990944;"

pubkey = ed25519.Ed25519PublicKey.from_public_bytes(base64.b64decode(pubkey_string))
sig = base64.b64decode(sig_string)
data = fingerprint.encode('utf-8')

try:
    pubkey.verify(sig,data)  
    print("Signature matches")

except Exception:
    print("Signature DOES NOT match");

Finally you can run it from within a nix-shell -p 'python3.withPackages (ps: [ps.cryptography])' and get a “Signature matches”.

Recap

So to recap the chain of verification somewhat working backwards up this long explanation:

  • we have a signature for the NAR file, which we can check against the signature and verify that the narHash is signed by the cache
  • We then can check the narHash against the store path to ensure that the store path matches
  • Because the store path contains the ISO, this indirectly verifies that the ISO in the store path is signed by the cache
  • Then you can check that the ISO in the NAR matches the one on the download page by comparing the SHA256 hash. Because they match, and the ISO in the store has been indirectly signed, the ISO on the download page is also indirectly signed

In addition to the cache key, hydra is probably signing it’s builds and cache.nixos.org is probably checking the hydra key before caching? Just a guess.

2 Likes

Direct signature of the ISOs wouldn’t be hard to add, I think. Probably with the same key which is singing all of cache.nixos.org – but what’s the threat model here?

As noted, if you have Nix(OS) already, you can simply nix-build that ISO, i.e. substitute and verify that signature – all automatic, without further steps. Sure, it’s a bit more expensive, but if you want extra verification, I don’t expect this is a blocker.

If you do not have Nix yet, how are you supposed obtain a trust anchor? Download the public key from the nixos.org web? Is that safer than downloading the hash of the ISO from that same web? Some other safer way?

Maybe we could put the full store path of the ISOs alongside the ISOs with an example? I did a nix copy --from https://cache.nixos.org but had to dig through the hydra outputs for nixos-26.05 release nixos-26.05.1183.6b316287bae2 just now to figure out what the exact store path was.

1 Like

That should be relatively easy. The script doing channel update already touches the /nix/store path (my $srcFile = $product->{path}) and downloads it via nix from S3.

1 Like

Yeah, it’s also an easy way to show people that the release process is (mostly) not magic and you can use Nix to grab it from the cache like anything else if you want.

(A nix build command with nixpkgs on the particular rev of the release may help too).

1 Like