Using remote building with ssh

Help
1

Hello, i use

 sudo nixos-rebuild switch --flake .#VPS-Server-005 --target-host manager@someIP --ask-sudo-password

for make my build remotly, this work faily great until not.

[sudo] password for manager: 
warning: Git tree '/etc/nixos' is dirty
[sudo] password for manager@someIp: 
building the system configuration...
warning: Git tree '/etc/nixos' is dirty
(manager@someIp) Password: 
copying 0 paths...
stopping swap device: /dev/disk/by-uuid/d5604472-7e21-4894-b30b-d4c4a0cdd945
stopping the following units: acme-renew-wateir.fr.timer, acme-setup.service, acme-wateir.fr.service, forgejo-secrets.service, forgejo.service, newt.service, postgresql-setup.service, redis-searx.service, searx-init.service, searx.service, systemd-modules-load.service, systemd-sysctl.service, vaultwarden.service
NOT restarting the following changed units: -.mount
activating the configuration...
[agenix] creating new generation in /run/agenix.d/2
[agenix] decrypting secrets...
decrypting '/nix/store/frv7rbz1jvsds664774p84bv74wg6af5-LtnxWKwZdDIxAKzp' to '/run/agenix.d/2/LtnxWKwZdDIxAKzp'...
decrypting '/nix/store/xnd41h98170ppq9zj6g7w4xw0bk934j0-YfDrVBDJcVoYNZeJ' to '/run/agenix.d/2/YfDrVBDJcVoYNZeJ'...
decrypting '/nix/store/2bfdfw4vhr65q7lxw7v4cqz3bmw3r8s7-xHeDf80ikqG65h3u' to '/run/agenix.d/2/xHeDf80ikqG65h3u'...
[agenix] symlinking new secrets to /run/agenix (generation 2)...
[agenix] removing old secrets (generation 1)...
removing group ‘git’
removing group ‘acme’
removing group ‘searx’
removing group ‘vaultwarden’
removing user ‘acme’
removing user ‘git’
removing user ‘searx’
removing user ‘vaultwarden’
[agenix] chowning...
Activation script snippet 'agenixChown' failed (1)
chown: invalid user: ‘vaultwarden:0’
setting up /etc...
Failed to run activate script
reloading user units for manager...
restarting sysinit-reactivation.target
reloading the following units: dbus.service, firewall.service
restarting the following units: boot.mount, nginx.service, sshd.service
starting the following units: postgresql-setup.service, systemd-modules-load.service, systemd-sysctl.service
Failed to start local-fs.target
Failed to restart boot.mount
Command 'ssh -o ControlMaster=auto -o ControlPath=/tmp/nixos-rebuild.vktqx4zb/ssh-%n -o ControlPersist=60 manager@someIp -- sudo --prompt= --stdin env NIXOS_INSTALL_BOOTLOADER=0 systemd-run -E LOCALE_ARCHIVE -E NIXOS_INSTALL_BOOTLOADER --collect --no-ask-password --pipe --quiet --service-type=exec --unit=nixos-rebuild-switch-to-configuration /nix/store/8n79f9v2vyb62qjqan9pjlicj6rdqjb5-nixos-system-VPS-Server-005-25.11.20251226.f560cce/bin/switch-to-configuration switch' returned non-zero exit status 255.

After that i can’t no longer ssh into my builder server.

janv. 05 00:02:49 ThinkCentre-Server-004 sshd-session[2424]: Received disconnect from ::1 port 59978:11:  [preauth]
janv. 05 00:02:49 ThinkCentre-Server-004 sshd-session[2424]: Disconnected from authenticating user root ::1 port 59978 [preauth]
janv. 05 00:02:49 ThinkCentre-Server-004 sshd-session[2424]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1  user=root
janv. 05 00:02:49 ThinkCentre-Server-004 unix_chkpwd[2435]: password check failed for user (root)
janv. 05 00:02:49 ThinkCentre-Server-004 sshd-session[2433]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1  user=root
janv. 05 00:02:51 ThinkCentre-Server-004 sshd-session[2433]: Failed password for root from ::1 port 58272 ssh2
janv. 05 00:02:51 ThinkCentre-Server-004 unix_chkpwd[2437]: password check failed for user (root)
janv. 05 00:02:53 ThinkCentre-Server-004 sshd-session[2433]: Failed password for root from ::1 port 58272 ssh2
janv. 05 00:02:53 ThinkCentre-Server-004 unix_chkpwd[2440]: password check failed for user (root)
janv. 05 00:02:55 ThinkCentre-Server-004 sshd-session[2433]: Failed password for root from ::1 port 58272 ssh2
janv. 05 00:02:55 ThinkCentre-Server-004 unix_chkpwd[2442]: password check failed for user (root)
janv. 05 00:02:55 ThinkCentre-Server-004 sshd-session[2438]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1  user=root
janv. 05 00:02:55 ThinkCentre-Server-004 sshd-session[2433]: Received disconnect from ::1 port 58272:11:  [preauth]
janv. 05 00:02:55 ThinkCentre-Server-004 sshd-session[2433]: Disconnected from authenticating user root ::1 port 58272 [preauth]
janv. 05 00:02:55 ThinkCentre-Server-004 sshd-session[2433]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1  user=root
janv. 05 00:02:56 ThinkCentre-Server-004 unix_chkpwd[2451]: password check failed for user (root)
janv. 05 00:02:56 ThinkCentre-Server-004 sshd-session[2449]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1  user=root
janv. 05 00:02:56 ThinkCentre-Server-004 sshd-session[2438]: Failed password for root from ::1 port 58278 ssh2
janv. 05 00:02:57 ThinkCentre-Server-004 sshd-session[2438]: Connection closed by authenticating user root ::1 port 58278 [preauth]
janv. 05 00:02:58 ThinkCentre-Server-004 sshd-session[2449]: Failed password for root from ::1 port 60232 ssh2
janv. 05 00:03:00 ThinkCentre-Server-004 unix_chkpwd[2452]: password check failed for user (root)
janv. 05 00:03:00 ThinkCentre-Server-004 unix_chkpwd[2455]: password check failed for user (root)
janv. 05 00:03:00 ThinkCentre-Server-004 sshd-session[2453]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1  user=root
janv. 05 00:03:01 ThinkCentre-Server-004 sshd-session[2449]: Failed password for root from ::1 port 60232 ssh2
janv. 05 00:03:02 ThinkCentre-Server-004 unix_chkpwd[2456]: password check failed for user (root)
janv. 05 00:03:03 ThinkCentre-Server-004 sshd-session[2453]: Failed password for root from ::1 port 60246 ssh2
janv. 05 00:03:03 ThinkCentre-Server-004 sshd-session[2449]: Failed password for root from ::1 port 60232 ssh2
janv. 05 00:03:03 ThinkCentre-Server-004 sshd-session[2449]: Received disconnect from ::1 port 60232:11:  [preauth]
janv. 05 00:03:03 ThinkCentre-Server-004 sshd-session[2449]: Disconnected from authenticating user root ::1 port 60232 [preauth]
janv. 05 00:03:03 ThinkCentre-Server-004 sshd-session[2449]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1  user=root
janv. 05 00:03:04 ThinkCentre-Server-004 sshd-session[2453]: Connection closed by authenticating user root ::1 port 60246 [preauth]
janv. 05 00:03:11 ThinkCentre-Server-004 unix_chkpwd[2497]: password check failed for user (root)
janv. 05 00:03:11 ThinkCentre-Server-004 sshd-session[2489]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1  user=root
janv. 05 00:03:13 ThinkCentre-Server-004 sshd-session[2489]: Failed password for root from ::1 port 35938 ssh2
janv. 05 00:03:14 ThinkCentre-Server-004 sshd-session[2489]: Connection closed by authenticating user root ::1 port 35938 [preauth]
janv. 05 00:03:34 ThinkCentre-Server-004 unix_chkpwd[2626]: password check failed for user (root)
janv. 05 00:03:34 ThinkCentre-Server-004 sshd-session[2624]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1  user=root
janv. 05 00:03:36 ThinkCentre-Server-004 sshd-session[2624]: Failed password for root from ::1 port 38548 ssh2
janv. 05 00:03:38 ThinkCentre-Server-004 sshd-session[2624]: Connection closed by authenticating user root ::1 port 38548 [preauth]
janv. 05 00:03:43 ThinkCentre-Server-004 unix_chkpwd[2630]: password check failed for user (root)
janv. 05 00:03:43 ThinkCentre-Server-004 sshd-session[2628]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1  user=root
janv. 05 00:03:44 ThinkCentre-Server-004 sshd-session[2631]: Connection closed by ::1 port 56432
janv. 05 00:03:44 ThinkCentre-Server-004 sshd-session[2632]: Invalid user admin from ::1 port 56434
janv. 05 00:03:44 ThinkCentre-Server-004 sshd-session[2632]: pam_unix(sshd:auth): check pass; user unknown
janv. 05 00:03:44 ThinkCentre-Server-004 sshd-session[2632]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1
janv. 05 00:03:45 ThinkCentre-Server-004 sshd-session[2628]: Failed password for root from ::1 port 56418 ssh2
janv. 05 00:03:45 ThinkCentre-Server-004 sshd-session[2628]: Connection closed by authenticating user root ::1 port 56418 [preauth]
janv. 05 00:03:46 ThinkCentre-Server-004 sshd-session[2632]: Failed password for invalid user admin from ::1 port 56434 ssh2
janv. 05 00:03:47 ThinkCentre-Server-004 sshd-session[2632]: Connection closed by invalid user admin ::1 port 56434 [preauth]

This log come from the builder machine,

full log of this day http://0x0.st/Po2K.txt

It’s seems like this make my builder server ddos it’s self

2

Use of

  --no-reexec

Don’t change the error message and still make my builder server finish into emergency mode

3

Did you check this error?

4

Hello yeah i have see it, but it’s minor and don’t important for now.

I’m much more worry about the fact that one of my machine go into emergency mode that some env file are not proprely unencrypt

Btw after read the error i belive it’s just because agenix try to chmod a file to a user that just are delete before

5

For me it looks like the activation script does not finish due to the error and that could be the reason why your system does not activate properly.

6 
sudo nixos-rebuild switch --flake .#VPS-Server-005 --target-host manager@someIp --ask-sudo-password
[sudo] password for manager: 
warning: Git tree '/etc/nixos' is dirty
[sudo] password for manager@someIp: 
building the system configuration...
warning: Git tree '/etc/nixos' is dirty
(manager@someIp) Password: 
copying 0 paths...
stopping swap device: /dev/disk/by-uuid/d5604472-7e21-4894-b30b-d4c4a0cdd945
stopping the following units: acme-renew-wateir.fr.timer, acme-setup.service, acme-wateir.fr.service, forgejo-secrets.service, forgejo.service, newt.service, postgresql-setup.service, redis-searx.service, searx-init.service, searx.service, systemd-modules-load.service, systemd-sysctl.service, vaultwarden.service
NOT restarting the following changed units: -.mount
activating the configuration...
[agenix] creating new generation in /run/agenix.d/2
[agenix] decrypting secrets...
decrypting '/nix/store/frv7rbz1jvsds664774p84bv74wg6af5-LtnxWKwZdDIxAKzp' to '/run/agenix.d/2/LtnxWKwZdDIxAKzp'...
decrypting '/nix/store/xnd41h98170ppq9zj6g7w4xw0bk934j0-YfDrVBDJcVoYNZeJ' to '/run/agenix.d/2/YfDrVBDJcVoYNZeJ'...
[agenix] symlinking new secrets to /run/agenix (generation 2)...
[agenix] removing old secrets (generation 1)...
removing group ‘searx’
removing group ‘git’
removing group ‘vaultwarden’
removing group ‘acme’
removing user ‘acme’
removing user ‘vaultwarden’
removing user ‘git’
removing user ‘searx’
[agenix] chowning...
setting up /etc...
reloading user units for manager...
restarting sysinit-reactivation.target
reloading the following units: dbus.service, firewall.service
restarting the following units: boot.mount, nginx.service, sshd.service
starting the following units: postgresql-setup.service, systemd-modules-load.service, systemd-sysctl.service
^C

So fix this issue, sorry for the time, make me find other issue, but all fix now.

Still same issue nothing change

7

I have find the problem, when i use

 sudo nixos-rebuild switch --flake .#VPS-Server-005 --target-host manager@someIp --ask-sudo-password

It’s switch my builder configuration not the one from my VPS, Which is not what i want at all.

8

The issue was

sudo nixos-rebuild switch --flake .#VPS-Server-005 --target-host manager@someIP --ask-sudo-password

I was using it on the wrong way.

This command is to use on the builder machine, not where you want to have the configuration endup.

So just check that you use it on the good way.

9

You could use --build-host if you want to evaluate on you local computer and build on another machine.

1 Like