Why is copy to remote nix store so slow?

Help
1

I have 2 computers at home with nearly identical configurations. To save bandwidth, I had the idea to update one (which took 20 minutes) and then use
nixos-rebuild switch --use-remote-sudo --target-host $host --flake ".#$host"
to copy the derivations to the second computer.

copying 2861 paths...
copying path '/nix/store/013yilq34lcbwb393h4fba4s76gydrbp-zram-generator.conf' to 'ssh://remote'...
...

However, this is horribly slow. I just aborted the whole thing after 3.5 hours. Then did a normal nixos-rebuild switch on the second computer, which took 17 minutes.

Why is it faster to download all derivations from the internet than copying it over lan?

1 Like
2

I suppose the nix store objects are compressed using xz for the transfer, which is heavily CPU bound.

3

You may also want to add --use-substitutes to the nixos-rebuild call, so that anything the remote can fetch matching derivations from cache.nixos.org.

4

Which is opposite to what OP wants. OP wants to save on external bandwidth usage.

Yes :frowning:

I have given up on using SSH or SSH-NG stores.

I have a list of things to experiment with on my personal computer, all of them require more steps and disallow nixos-rebuild doing the actual switch.

Some of them:

  1. sshfs mount and nix copy --to that mount; nix copy --from with local path on the remote
  2. reversing the mount directions
  3. copy to local, ZIP/TAR/whatever, copy to remote, unpack there, copy from unpacked into store
  4. implement a better way of transferring store paths in nix itself (I don’t know C++ very well)
5 Likes
5

A local nginx cache is way more effective, will prune least used objects and won’t need new keys.

3 Likes
6

I’m using a similar setup here. I first nix build the second system closure on the first system, the nix copy it to the second system, copy flake.{nix,lock}, the do nixos-rebuild switch --target-host ....

I also observed that if the second machine is reachable via WiFi and Ethernet ssh can get confused by the Hostname/IP mapping and copying is very slow.

1 Like
7

Revisiting this for the ~4th time. Things haven’t changed significantly here are two versions of copy you could be using.

# just following the manual slow:
## --no-check-sigs can be removed if signing is setup correctly
nix copy --no-check-sigs --from 'ssh-ng://<server>' <store-path>

# lix equivalent, still slow
nix-copy-closure --from '<server>' --log-format bar <store-path>

nix-copy-closure has an additional --gip flag that speeds things up. That flag is just tacking ssh builtin compression onto the ssh url. That’s good news as either Compression yes in .ssh/config or --from 'ssh-ng://<server>?compress=true' when manually calling the command both significantly improve the transfer speed.

The bulk transfer strategy referenced above may still be faster. This is minor change just isn’t in the manual (or is assumed).

8

maybe you could try

3 Likes