Deploy NixOS configurations on other Machines

Help
1

I want to deploy my NixOS configurations remotely.

I’m currently using a nix flake with a nixosConfigurations output attribute to build configurations for different machines. However, this requires cloning this repo to every machine and building the configuration locally. This makes has security issues (private keys stored on public facing machines), performance implications (requires setting up remote builders to get performant builds on resources constrained devices), as well as organizational annoyances (like having to clone this repo on every machine).

There seem to be a few projects to NixOS configurations including:

I’ve looked as NixOps as it’s the ‘official’ project. But the manual seems to suggest doing everything declaratively which is not ideal. (An aside, NixOps 1.7 fails to build due to a security issue and NixOps 2.0 is unstable, which is a bit daunting to a new user). There, however, does appear to be undocumented flake integration with nixopsConfigurations.

How are other people solving this problem? Is there a golden path?

Thanks

3 Likes
2

All those tools basically do this:

nix build -L .#nixosConfigurations.remotehost.config.system.build.toplevel
nix copy --to ssh-ng://root@remotehost ./result
ssh root@remotehost nix-env -p /nix/var/nix/profiles/system --set $(readlink ./result)
ssh root@remotehost /nix/var/nix/profiles/system/bin/switch-to-configuration switch
6 Likes
3

I am happy to help with Cachix Deploy — documentation

1 Like
4

I use two simple scripts which are basically just the following command:

nixos-rebuild switch --use-remote-sudo --build-host localhost --target-host remote-server.domain.local --flake ".#remote-server"

This way my notebooks builds the config for all systems and then pushes them to the remote servers.
I even compile the config for two Raspberry Pis.

This updates all my servers

And this takes an argument and updates just a single machine

One could probably make it nicer and get the list of hosts directly from the flake but for the moment it is good enough and allows me to focus on the servers themselves.
One thing which I still have to fix is limit password less sudo to the nix command but that isn’t something which is part of the script but part of the NixOS config.

2 Likes
5

FYI, nixos-rebuild supports remote builds via --target-host. You can run that from your repo and point nixos-config’s NIX_PATH at some machine’s config; i.e. -I nixos-config=configs/machine/default.nix.

6

Thanks for the replies, all!

@Atemu @Nebucatnetzer

It looks like nixos-rebuild switch --flake .#<host> --target-host <host> is exactly what I’m looking for.

@misuzu

nix build -L .#nixosConfigurations.remotehost.config.system.build.toplevel
nix copy --to ssh-ng://root@remotehost ./result
ssh root@remotehost nix-env -p /nix/var/nix/profiles/system --set $(readlink ./result)
ssh root@remotehost /nix/var/nix/profiles/system/bin/switch-to-configuration switch

This is very informative, and I suspect what nixos-rebuild is doing under the hood, but is a bit too low-level to be ergonomic.

I am happy to help with Cachix Deploy — documentation

I’m not familiar with Cachix, but I think this is a bit too complicated for my use case. I wasn’t looking for a CI system but rather a script to way to deploy on remote machines. Thank you for the offer though!

@Nebucatnetzer

One could probably make it nicer and get the list of hosts directly from the flake but for the moment it is good enough and allows me to focus on the servers themselves.

I think

hosts=($(echo `nix eval .#nixosConfigurations --apply 'pkgs: builtins.concatStringsSep " " (builtins.attrNames pkgs)'` | xargs ))

might help with this!

1 Like
7

No problem, glad I could help :slight_smile: and thanks for the input with the hosts.
Looking forward to test it.

1 Like
8

This is now the updated script:

#!/usr/bin/env bash

hosts=($(echo `nix eval .#nixosConfigurations --apply 'pkgs: builtins.concatStringsSep " " (builtins.attrNames pkgs)'` | xargs ))
skip=(
    "gwyn"
    "loki-test"
    "desktop-vm"
)

rsa_key="$HOME/.nixos/secrets/ssh_keys/ansible/ansible.key"
export NIX_SSHOPTS="-t -i $rsa_key"

for host in "${hosts[@]}"
do
    # Check if the host is in the skip list
    if [[ " ${skip[*]} " =~ " ${host} " ]];then
        continue
    fi
    fqdn="$host.2li.local"
    echo $fqdn
    nixos-rebuild switch -j auto --use-remote-sudo --build-host localhost --target-host $fqdn --flake ".#$host"
    echo "reboot $fqdn"
    ssh -i $rsa_key $fqdn 'sudo reboot'
    echo
    echo
done
4 Likes
9

NB this doesn’t work for different architectures nixos-rebuild fails cross-build a flake · Issue #166499 · NixOS/nixpkgs · GitHub.

Adapted from @misuzu’s answer, a workaround is:

$ sudo nix build /etc/nixos#nixosConfigurations.rasp-pi.config.system.build.toplevel --builders 'ssh-ng://rasp-pi aarch64-linux' --max-jobs 0 builders-use-substitutes --option builders-use-substitutes true
# relies on $(readlink ./result) already being in the rasp-pi's nix store from using it as a remote builder
# it would be nice if there was a way to avoid having to copy this back to the workstation machine
$ ssh root@rasp-pi nix-env -p /nix/var/nix/profiles/system --set $(readlink ./result)
$ ssh root@rasp-pi /nix/var/nix/profiles/system/bin/switch-to-configuration switch

https://github.com/NixOS/nixpkgs/issues/200398#issuecomment-1318899763

10

It does just fine for me. All mu servers are Raspberry Pi 4.

11

Ah I see you’re building on a remote system with different architecture while I build on localhost an x86_64-linux system configurations for aarch64-linux systems.

12

Not sure this is similar to what you have but I am running

nixos-rebuild switch --flake .#emerald --target-host user@<raspberry-pi-ip>

On a x86_64-linux laptop and trying to deploy to a raspberry pi, whose configuration looks like:

      emerald = nixpkgs.lib.nixosSystem {
        system = "aarch64-linux";
        modules = [
          ...
        ];
      };

This run into the problem as mentioned above:

error: build of '/nix/store/48x117xydlkij563h99xwk9npwl0ggc9-ensure-all-wrappers-paths-exist.drv' on 'ssh-ng://nixbuilder@radahn' failed: error: a 'aarch64-linux' with features {} is required to build '/nix/store/48x117xydlkij563h99xwk9npwl0ggc9-ensure-all-wrappers-paths-exist.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

kind of suspect that I messed up with the system = ... – this has been quite confusing for me.

13

You also need --build-host.

14

Thanks! I forgot to mention that the target system, though can be accessed via ssh, is not connected to internet. I assume --build-host would fail in this case?

15

You can add this line to your PC then you can build for aarch64 as well and push it to the Pi.

You then need to set —build-host localhost in order to build it on your PC.
Since we now have a binary cache for aarch64 the whole thing is quite fast.

16

In that case you could perhaps add it as a remote builder. IIRC remote builders always gather deps on the controlling machine and copy them over. This is annoying in most cases but probably what you want here.

17

Thanks. Actually had that line on my laptop, and running nixos-rebuild switch --target-host from the laptop does not work, error is still about not being able to build aarch64-linux.

Haven’t tried that from the pi side though, will give it a try.

18

Yep I think you are right. Worst case I will add another pi that connects to the internet as the remote builder.

19

The important part would be —build-host.
The full command would then be

nixos-rebuild switch -j auto --use-remote-sudo --build-host localhost --target-host user@pi.local --flake ".#target"

3 Likes
20

Worked like a charm. Thanks!

1 Like