Speeding up NixOS installation by having Nix store on an external disk

Help
1

Hi folks,

I wonder if this would work…

I have 2 PCs and flake with configuration for both of them - the configuration for both is very similar.

I install first PC by:

  1. format disk, mount partitions…
  2. run nixos-install --flake .#first
  3. Reboot

After that I follow: Move Nix Store to new partition - NixOS Wiki

and I compress the whole Nix store: tar -afp /external_disk/nix.tar.gz /nix

Then I walk to the second PC (different house, no way to SSH-in) and I install NixOS like this:

  1. format disk, mount partitions and mount the external disk…
  2. cd /mnt ; tar -xf /external_disk/nix.tar.gz
  3. run nixos-install --flake .#second
  4. Reboot

My understanding is that the flake locks all versions and therefore the second PC won’t need to download/build 95% and it will just download/build 5%. Later I can run nix store gc and remove items that were built/downloaded for the first PC and therefore have no use on the second.

Does it sound OK? Should I rather use nix nix copy instead of tar? I looked at nix copy man page and I have a hard time to understand how to copy everything…

Thank you.

2

I see no reason why this would fail. But indeed, nix copy might be a bit cleaner (but I don’t think it supports compression directly… one solution if you care about space on the external drive is to use a file system that does compression on the fly like ZFS). Notably, nix copy allows you to pick the exact closure of a system without picking everything else (for instance, you might have other flakes on the system, stuff installed impurely or via home-manager that will also be packed with your method).

I wrote a related answer here (to do a backup on an external disk… which is close enough to what you are trying to achieve). Note that in your case, you want to use --to /your/path since you have no SSH access. You can also look at the documentation of nix copy here.

3

I just copied/pasted the example. I’d most likely just tar it without gzip.

I’ll try nix copy. Thank you.

4

Testing on a VM:

su -
nix shell nixpkgs#nix
nix --version # nix (Nix) 2.18.1
mkdir /copy_here
nix copy --to file:///copy_here $(readlink -f /nix/var/nix/profiles/system-22-link)

and it is still working… will report back if it finishes :wink:

5

It copied a lot of *.narinfo files but not the actual binaries. Unfortunately that means that tar is the only way to go.

6

Very weird… what that readlink outputs? nix copy seems to work for me (as does nix-store --export)

7

I confirm this also seems to works on my system… maybe try to use the exact command below with the --no-check-sigs? (without I get an error)

$ mkdir /tmp/nix
$ nix copy --to /tmp/nix /run/current-system --no-check-sigs
$ ls /tmp/nix/nix/store/*bash-5*/bin
bash sh
1 Like
8

I thought I needed to convert it to /nix/store/abcde123…

Thank you. That worked.

with 23.05 nix

ls -l /mnt/
total 0
nix --version
nix (Nix) 2.13.6
nix copy --to /mnt /run/current-system
error: cannot add path '/nix/store/06glsrjrhasqpa7lvz4kz6yh2i3f14dv-libreelec-dvb-firmware-1.4.2-xz' because it lacks a signature by a trusted key
tree /mnt/
/mnt/
└── nix
    ├── store
    │   ├── 01yazlfa6r4fp3djsmp94frqm7ffpjdb-vscode-extension-streetsidesoftware-code-spell-checker-2.20.5.lock
    │   ├── 02jnr85964kaz0i4j8x12gakzw2bz09y-hm_powermanagementprofilesrc
    │   └── 02jnr85964kaz0i4j8x12gakzw2bz09y-hm_powermanagementprofilesrc.lock
    └── var
        └── nix
            ├── db
            │   ├── big-lock
            │   ├── db.sqlite
            │   ├── reserved
            │   └── schema
            ├── gc.lock
            ├── gcroots
            │   ├── per-user
            │   │   └── root
            │   └── profiles -> /mnt/nix/var/nix/profiles
            ├── profiles
            │   └── per-user
            │       └── root
            └── temproots

13 directories, 8 files
nix copy --to /mnt /run/current-system --no-check-sigs
du -sh /mnt/
20G     /mnt/

with unstable nix

ls -l /mnt/
total 0
nix shell nixpkgs#nix
nix --version
nix (Nix) 2.18.1
nix copy --to /mnt /run/current-system
error: cannot add path '/nix/store/06glsrjrhasqpa7lvz4kz6yh2i3f14dv-libreelec-dvb-firmware-1.4.2-xz' because it lacks a signature by a trusted key
tree /mnt/
/mnt/
└── nix
    ├── store
    │   ├── 01yazlfa6r4fp3djsmp94frqm7ffpjdb-vscode-extension-streetsidesoftware-code-spell-checker-2.20.5
    │   ├── 01yazlfa6r4fp3djsmp94frqm7ffpjdb-vscode-extension-streetsidesoftware-code-spell-checker-2.20.5.lock
    │   └── 031mfxkf1lpklwwx40crxsnvqmva5fis-kjobwidgets-5.106.0-bin.lock
    └── var
        └── nix
            ├── db
            │   ├── big-lock
            │   ├── db.sqlite
            │   ├── reserved
            │   └── schema
            ├── gc.lock
            ├── gcroots
            │   ├── per-user
            │   └── profiles -> /mnt/nix/var/nix/profiles
            ├── profiles
            │   └── per-user
            └── temproots

12 directories, 7 files
nix copy --to /mnt /run/current-system --no-check-sigs
du -sh /mnt/
20G     /mnt/
1 Like
9

when copying complex file systems, rather than simple directories, tar will never cut it. /nix on NixOS is a bit more complex… but probably not that complex.

cpio was the choice to preserve all the extra metadata on a file systems, and there are a lots… .it’s not just files!!!

However even using cpio , i don’t feel that doing a ‘file system clone’ of the nix store is not a good idea, as there is bound to be a little bit of state tying the store to the machine, but correct me if i’m wrong…

lthe mighty luc goes into nix copy here and has a nice write up about this boring command.

if your using channels, then both machines must be on the same revision of that channel for anything to be transportable.

however, with the same flake.lock , this changes the game slightly.

the wiki page you mentions is only for nix, not NixOS.

However I’d be interested to learn what ‘state’ is on /nix…and how a system can be ‘ghost imaged’ in it’s entirety with unix file system copy primitives…

rather than the nix-copy commands

10

As far as I know, the nix store are just files. Symlinks, directories and files are all files.

Which one?

I’m not cloning, I’m just speeding up download but adding stuff to nix store. And that will happen before nixos-install so the system should look the same like I didn’t copy anything.

The only difference will be stuff, that is not relevant to the second PC and that will be removed after installation by nix store gc.

1 Like
12

okay, your prewarming the cache on the other machine… i understand now.

copy with tar should be fine, i’m overly cautious about file system copies after years of backing up / on unix servers… i need to throw this caution away…old habits, they die hard i guess.

I’m trying to work out the relationship between the sqlite db (which has a bit of state in it)… … give it a whirl…see what happens.

i think it will be ok.

13

Or just cp -a. Obliviously nix copy is the best and sqlite will be created by that.

That’s the beauty of nix, it doesn’t need ACLs or SELinux for files or OCI for building and that means that one day we might have first class support for BSDs, embedded distros and maybe even Fuchsia OS :wink:

1 Like
14

The error about missing signatuse is why you want to add --no-check-sigs.

15

well, BSD would bit a dream come true…

the linux kernel seems to be turning into a kitchen sink kernel.