Nix store copy closure or import/export with binary diff

Help
1

I am trying to update a nix os system through a 600 baud link.

The issue is, it could take a very long time, even if it only copies the paths that don’t exist on the remote.

I wrote a simple python script which traverses the files in a store given the previous system hash, and produces binary diffs using the excellent bsdiff tool.

The result is then compressed using tar and xz -9

Another script uses bspatch to restore the files on the remote side and it perfectly replicates the original store using 1/6-1/15th the data transferred.

I am looking at files < 1MB instead of 10MB, so a few hours to transfer instead of several days.

My issue now is, nix store import/export will not let me import paths outside the store, and nix realize is not working either.

How can I restore the files? Doesn’t it seem like this ability could be built into nix store import/export to begin with? It could also work with nix copy closure, since in that case it knows the remote paths, so it should be able to automatically do it.

Any thoughts? Pointers to how I can easily solve this?

2 Likes
2

OK, I’m intrigued now - have you somehow managed to launch a deep space probe running NixOS? :)

I’d be lying if I said I fully understood, but is the issue basically registering the newly restored files with the database? The installer uses nix-store --load-db to do this, and the format seems to be described in this ancient mailing list post (it doesn’t seem to have changed though.) There’s also an undocumented nix-store --register-validity operation, which I assume does something similar one path at a time.

3

I managed to figure it out… I have to do export from different revisions and binary diff the results.

It works incredibly well, about 1/30th the file size.

And yes this is for a deep space probe running nixos.

4 Likes
4

This is actually awesome wow