Nix store: copy vs sigs

[uep]

I’m seeing some confusing behaviour with regard to copying store items between machines and signatures.

Background

I updated one machine the other day to the current unstable channel, rev 4a01ca36d6b. For reasons I’m uncertain about, quite a few packages were built from source. It may have been very soon after the channel updated, but even then it was quite unexpected. Regardless, I wasn’t really paying attention at the time and only really noticed after it was done. I’m not (in this post; see footnote) particularly interested in why they were cache misses, and note it only for background context or in case it ends up sharing a root cause with what follows.

This morning, I went to update my (rather slower) laptop, assuming that the problem would have cleared - either the cache caught up, or a new build was present, or both. But, as it turned out, the channel was still at the same point, and there were still lots of cache misses.

“Odd, but I should at least be able to re-use what I already built.”

Copying closures

So after a little reading to refresh my memory, I tried to copy the system closure of from one machine’s store to the other. They run basically identical configurations in terms of packages.

# nix-copy-closure dan@192.168.21.197 /run/current-system
[...]
error: cannot add path '/nix/store/5b4laxahcbsqk0khzk4vrgj6nc1jjsjs-tracker-miners-3.3.1' because it lacks a valid signature

“Yeah, ok, fair; I haven’t set up local build keys and trusts.”

Remote build

On the laptop, I ran a nixos-rebuld with --build-host with the machine that already had the successful update. I wasn’t sure if this would try and use the already-built packages in the store, or build again, but at least that system has much more cpu.

It tried to use the store, basically doing the same copy as above and failing with the same error.

"Ugh, ok, let’s see if we can fix the sigs"

Verifying store and updating sigs

I ran a nix store verify --all and got loads of untrusted packages, presumably all those that had been built locally. Again, more than I expected. I added a -s https://cache.nixos.org/ to use sigs from upstream, and everything was fine, but of course those sigs weren’t persisted locally past that one run.

I did a bit more reading and discovered nix store copy-sigs --all -s https://cache.nixos.org/ to copy the sigs. After that, the verify is clean on subsequent runs.

“Hooray, i don’t need to figure out adding sigs to existing builds!”

"Odd, the cache has signatures for packages it isn’t serving. Why? Eh, at least I can use them now"

And yet, still no

Even with the sigs copied, such that nix store verify is clear, copying the store path fails. Picking a (failing) package at random:

[dan@oenone:~]$ nix store verify /nix/store/1bdsa926n8hy2gsgr794lj9yvhzcizk2-gnome-music-42.1

[dan@oenone:~]$ nix-copy-closure 192.168.21.197 /nix/store/1bdsa926n8hy2gsgr794lj9yvhzcizk2-gnome-music-42.1
copying 1 paths...
copying path '/nix/store/1bdsa926n8hy2gsgr794lj9yvhzcizk2-gnome-music-42.1' to 'ssh://192.168.21.197'...
error: cannot add path '/nix/store/1bdsa926n8hy2gsgr794lj9yvhzcizk2-gnome-music-42.1' because it lacks a valid signature
error: unexpected end-of-file

What am I missing?

I’m not sure why this isn’t working at this point.

I probably should go ahead and set up keys for local builds, and maybe even sign the existing packages with that key, or maybe just have them there for “next time” on new builds. But I’m not clear on why that would be necessary if all the packages are already carrying trusted sigs.

---

Footnote: What’s going on with the cache, anyway?

There’s something a bit odd going on with the cache, missing packages but existing sigs. I have a suspicion some of it might be related to other issues discussed here:

Slow cache.nixos.org

I’ve reported this problem at #infra:nixos.org Matrix room, but haven’t heard from them. Following is the video demonstrating the problem: https://asciinema.org/a/506447 Following is the output from the diagnostics script: http://ix.io/40Rt From my observation, it seems like if the file is not cached at local cache.nixos.org server(s), then the fetch from origin is slow. Once the file is cached on cache.nixos.org server, it downloads on the client just fine. And from my use it seems like I’…

So didn’t want it to be the focus of this post

[Atemu]

More evidence of cache funkyness: What's wrong with the cache?

[ElvishJerricco]

Sometimes the cache takes a good amount of time to catch up. What’s more, your system will cache which store paths are cache misses, so even if Hydra has done the build and cached it, you may still end up building it locally if you attempted to build it at least once before Hydra had cached it. You can clear this miss cache by deleting ~/.cache/nix/binary-cache-v6.sqlite and ~/.cache/nix/binary-cache-v6.sqlite-journal.

As for your trouble with nix-copy-closure, I’m unsure. I tested the issue out by building nixpkgs#hello with --substituters '' so that it wouldn’t come from the cache, and did not have the same experience.

1. Build without cache. nix build --substituters '' nixpkgs#hello
2. Attempt to nix-copy-closure to another machine. This expectedly fails due to lack of signature.
3. Run nix store copy-sigs -s https://cache.nixos.org /nix/store/z5wwy5nagzy15gag42vv61c2agdpz2f2-hello-2.12/ to get the signature on my local machine.
4. Attempt to nix-copy-closure again. This time it succeeds. This is where our experiences differ.

So there must be something I’m missing in my attempt to reproduce your problem.

[Atemu]

Cached misses aren’t it in my case, the cache really doesn’t have the output path despite hydra claiming the build succeeded.

[uep]

Mine either: different hosts.

But I’m really much more interested in this thread being about how to make copied sigs stick.