No problem, glad I could help and thanks for the input with the hosts.
Looking forward to test it.
This is now the updated script:
#!/usr/bin/env bash
hosts=($(echo `nix eval .#nixosConfigurations --apply 'pkgs: builtins.concatStringsSep " " (builtins.attrNames pkgs)'` | xargs ))
skip=(
"gwyn"
"loki-test"
"desktop-vm"
)
rsa_key="$HOME/.nixos/secrets/ssh_keys/ansible/ansible.key"
export NIX_SSHOPTS="-t -i $rsa_key"
for host in "${hosts[@]}"
do
# Check if the host is in the skip list
if [[ " ${skip[*]} " =~ " ${host} " ]];then
continue
fi
fqdn="$host.2li.local"
echo $fqdn
nixos-rebuild switch -j auto --use-remote-sudo --build-host localhost --target-host $fqdn --flake ".#$host"
echo "reboot $fqdn"
ssh -i $rsa_key $fqdn 'sudo reboot'
echo
echo
done
NB this doesn’t work for different architectures nixos-rebuild fails cross-build a flake · Issue #166499 · NixOS/nixpkgs · GitHub.
Adapted from @misuzu’s answer, a workaround is:
$ sudo nix build /etc/nixos#nixosConfigurations.rasp-pi.config.system.build.toplevel --builders 'ssh-ng://rasp-pi aarch64-linux' --max-jobs 0 builders-use-substitutes --option builders-use-substitutes true
# relies on $(readlink ./result) already being in the rasp-pi's nix store from using it as a remote builder
# it would be nice if there was a way to avoid having to copy this back to the workstation machine
$ ssh root@rasp-pi nix-env -p /nix/var/nix/profiles/system --set $(readlink ./result)
$ ssh root@rasp-pi /nix/var/nix/profiles/system/bin/switch-to-configuration switch
https://github.com/NixOS/nixpkgs/issues/200398#issuecomment-1318899763
It does just fine for me. All mu servers are Raspberry Pi 4.
Ah I see you’re building on a remote system with different architecture while I build on localhost an x86_64-linux
system configurations for aarch64-linux
systems.
Not sure this is similar to what you have but I am running
nixos-rebuild switch --flake .#emerald --target-host user@<raspberry-pi-ip>
On a x86_64-linux
laptop and trying to deploy to a raspberry pi, whose configuration looks like:
emerald = nixpkgs.lib.nixosSystem {
system = "aarch64-linux";
modules = [
...
];
};
This run into the problem as mentioned above:
error: build of '/nix/store/48x117xydlkij563h99xwk9npwl0ggc9-ensure-all-wrappers-paths-exist.drv' on 'ssh-ng://nixbuilder@radahn' failed: error: a 'aarch64-linux' with features {} is required to build '/nix/store/48x117xydlkij563h99xwk9npwl0ggc9-ensure-all-wrappers-paths-exist.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}
kind of suspect that I messed up with the system = ...
– this has been quite confusing for me.
You also need --build-host
.
Thanks! I forgot to mention that the target system, though can be accessed via ssh
, is not connected to internet. I assume --build-host
would fail in this case?
You can add this line to your PC then you can build for aarch64 as well and push it to the Pi.
You then need to set —build-host localhost
in order to build it on your PC.
Since we now have a binary cache for aarch64 the whole thing is quite fast.
In that case you could perhaps add it as a remote builder. IIRC remote builders always gather deps on the controlling machine and copy them over. This is annoying in most cases but probably what you want here.
Thanks. Actually had that line on my laptop, and running nixos-rebuild switch --target-host
from the laptop does not work, error is still about not being able to build aarch64-linux
.
Haven’t tried that from the pi side though, will give it a try.
Yep I think you are right. Worst case I will add another pi that connects to the internet as the remote builder.
The important part would be —build-host
.
The full command would then be
nixos-rebuild switch -j auto --use-remote-sudo --build-host localhost --target-host user@pi.local --flake ".#target"
Worked like a charm. Thanks!
it doesn’t work for me when the target architecture is different than the host architecture that calls the command.
This one works as it targets the same arch - x86_64-linux
nixos-rebuild switch --flake .#deckard --target-host deckard --use-remote-sudo
Now, a similar command but I am targeting an aarch64 machine and using nixbuild builder (this one also works):
nixos-rebuild build --max-jobs 0 --build-host "eu.nixbuild.net" --target-host surfer --use-remote-sudo --flake .#surfer
However if I replace build
with switch
the command fails:
$ nixos-rebuild switch --max-jobs 0 --build-host "eu.nixbuild.net" --target-host surfer --use-remote-sudo --flake .#surfer
/nix/store/5yksn2xwy3aif5pxz353i64i0fwvj5gp-nixos-rebuild/bin/nixos-rebuild: line 382: /nix/store/zlh7zakv2fn97fb4q2y6abzsdp6jflfd-coreutils-9.3/bin/mktemp: cannot execute binary file: Exec format error
Any idea what could be wrong?
Did you declare the hostSystem
in surfer
’s config?
No, I only have system = "aarch64-linux
. I thought that hostSystem
property is only required when doing some cross-compilation, isn’t it?
I kind of solved my problem, i.e. I found the correct set of flags that work but I would like to understand what is going on.
The correct command looks as follow:
nixos-rebuild --max-jobs 0 --builders "ssh://eu.nixbuild.net aarch64-linux - 100 1" --flake .#surfer --target-host surfer --fast --use-remote-sudo switch
what is interesting is that the documentation says:
--fast
Equivalent to --no-build-nix. This option is useful if you call nixos-rebuild frequently (e.g. if you’re hacking on a NixOS module).
where:
--no-build-nix
Normally, nixos-rebuild first builds the nixUnstable attribute in Nixpkgs, and uses the resulting instance of the Nix package manager to build the new system configuration. This is necessary if the NixOS modules use features not provided by the currently installed version of Nix. This option disables building a new Nix.
However running it with --no-build-nix
instead of --fast
fails as before:
nixos-rebuild --no-build-nix --max-jobs 0 --builders "ssh://eu.nixbuild.net aarch64-linux - 100 1" --flake .#surfer --target-host surfer switch
/nix/store/5yksn2xwy3aif5pxz353i64i0fwvj5gp-nixos-rebuild/bin/nixos-rebuild: line 382: /nix/store/zlh7zakv2fn97fb4q2y6abzsdp6jflfd-coreutils-9.3/bin/mktemp: cannot execute binary file: Exec format error
What was this pointing at before? Right now the line is just a closing bracket.
Right, forget to make it a permalink. It would be this line here: nixos/default.nix at dac50a1ab18bbf5628a6bcac96902d70da84dcb7 - nixos - Gitea: Git with a cup of tea
Thanks! Of course, you need to be able to build on your own machine.