Why is copy to remote nix store so slow?

Help
1

I have 2 computers at home with nearly identical configurations. To save bandwidth, I had the idea to update one (which took 20 minutes) and then use
nixos-rebuild switch --use-remote-sudo --target-host $host --flake ".#$host"
to copy the derivations to the second computer.

copying 2861 paths...
copying path '/nix/store/013yilq34lcbwb393h4fba4s76gydrbp-zram-generator.conf' to 'ssh://remote'...
...

However, this is horribly slow. I just aborted the whole thing after 3.5 hours. Then did a normal nixos-rebuild switch on the second computer, which took 17 minutes.

Why is it faster to download all derivations from the internet than copying it over lan?

2

I suppose the nix store objects are compressed using xz for the transfer, which is heavily CPU bound.

3

You may also want to add --use-substitutes to the nixos-rebuild call, so that anything the remote can fetch matching derivations from cache.nixos.org.

4

Which is opposite to what OP wants. OP wants to save on external bandwidth usage.

Yes :frowning:

I have given up on using SSH or SSH-NG stores.

I have a list of things to experiment with on my personal computer, all of them require more steps and disallow nixos-rebuild doing the actual switch.

Some of them:

  1. sshfs mount and nix copy --to that mount; nix copy --from with local path on the remote
  2. reversing the mount directions
  3. copy to local, ZIP/TAR/whatever, copy to remote, unpack there, copy from unpacked into store
  4. implement a better way of transferring store paths in nix itself (I don’t know C++ very well)
2 Likes
5

A local nginx cache is way more effective, will prune least used objects and won’t need new keys.

6

I’m using a similar setup here. I first nix build the second system closure on the first system, the nix copy it to the second system, copy flake.{nix,lock}, the do nixos-rebuild switch --target-host ....

I also observed that if the second machine is reachable via WiFi and Ethernet ssh can get confused by the Hostname/IP mapping and copying is very slow.