Evaluate nix on remote system

Help
1

I tried to use nix on RPi 3 and the initial evaluation is very slow as it takes a lot of RAM. Can I offload evaluation to PC?

1 Like
2

Yes! This is probably one of my favorite nix features:

nixos-rebuild switch --use-remote-sudo --target-host example.com --build-host localhost

I use flakes, so specifying the flake URL with --flake lets me choose the config to deploy, but with non-flakes workflows you’d want to use that environment variable that skips my mind.

You’ll also need to set nix.trustedUsers.

You may need to configure the build architecture, since you’re deploying to an RPI, but I have yet to experiment with that personally.

3 Likes
3

From my understanding, this does the evaluation of the configuration of the remote builder, not the configuration of the target host. Have I missed something ?

4

Yep:

I.e., that command + whatever means you have to specify the configuration that should be built will achieve “build the remote configuration on my fast localhost and deploy it to the remote”.

I guess you’re missing the step of having a copy of the remote configuration locally? For hosts that don’t build themselves I typically have an empty /etc/nixos, and instead have their whole config in a git repository hosted somewhere my build host has easy access to.

My git history for my main server has this snippet as a note to self in it, if it helps explain that:

nixos-rebuild switch --use-remote-sudo --target-host example.com --build-host localhost -I nixos-config=./configuration.nix

… though I’m unsure whether that -I bit is correct, I’m not sure how well I understood nix back then.

Nowadays I use:

nixos-rebuild switch --use-remote-sudo --target-host example.com --build-host localhost --flake .#examplecom
5

Oh, ok. Thanks for the explainations, I’m not very familiar with flakes.

6

After several trials and errors, I came up with those conclusions based on the hints given by @TLATER.

So you wan’t to do the evaluation from a remote builder. But all the point of doing this is I guess to do it on a more powerful machine. So I assume you’re doing it on a x86_64 builder.

Which means that you can’t use previous nixos-rebuild command above because it doesn’t have the --argstr option.

Which makes it impossible to propagate system = "aarch64-linux" to all function calls in nixpkgs.

But I’ve got an alternative for you.

First, same as above, copy all the nixos sys config of your RPi to the remote builder (maybe copying it on a tmpfs makes the evaluation even quicker).

Second, still in the builder, instantiate it with the system arg example:

NIXOS_CONFIG=/path/to/copied-RPi-nixos-config/configuration.nix nix-instantiate '<nixpkgs/nixos>' -A system --argstr system aarch64-linux

This will output a derivation that you can copy to your RPi’s store with a new feature in nixUnstable :

Use this if your RPI’s ssh server is configured with something else than port 22:

nix-shell -p nixUnstable
# in the shell, using new --derivation feature
NIX_SSHOPTS='-p 888' \
  nix copy --to 'ssh://user@hostname-of-your-rpi' \
    --derivation /nix/store/examplederivationbgx0nkshrh9cdrm-nixos-system-serverpi21.05.4658.dde1557825c.drv \
    --option 'extra-experimental-features' 'nix-command'

Otherwise, forget the NIX_SSHOPTS part :

nix copy --to 'ssh://user@hostname-of-your-rpi' \
  --derivation /nix/store/examplederivationbgx0nkshrh9cdrm-nixos-system-serverpi21.05.4658.dde1557825c.drv \
  --option 'extra-experimental-features' 'nix-command'

Once the derivation and all it’s dependencies are copied, you can login to your RPi and build the system config with

nix-store -r /nix/store/examplederivationbgx0nkshrh9cdrm-nixos-system-serverpi21.05.4658.dde1557825c.drv

Which will output the final store path of your system build.
The last thing you want to do is to activate this new system build.

Activation will make the bootloader choose this new system by default so that you don’t have to activate it after each boot.
It will also restart systemd services/units if needed.

If you prefer to just test it do that:

/nix/store/examplebuild5kiqjp7npg4924mx6gz9-nixos-system-
serverpi-21.05.4658.dde1557825c/bin/switch-to-configuration test

the other actions for switch-to-configuration are switch (to activate, like in nixos-rebuild switch), boot (same as nixos-rebuild boot) and dry-activate (same too).

And you should be good to go, not the easiest thing to do but very doable.

Hope this answers your question, at least it definitly works for me and I’ll use this in the future (probably will make a script to automate this).

1 Like
7

Thanks for the sleuthing @ScottHamilton ! I wonder if the system property on flakes is properly propagated, as that would simplify this quite a bit. Eventually I’ll get around to using my RPI too and test all of this :wink: