Failed to restart sysinit-reactivation.target

Help
1

Since upgrading to NixOS 24.11 and deploying with deploy-rs I am randomly seeing this error from different hosts:

$ deploy  -s '.#servers2'
🚀 ℹ️ [deploy] [INFO] Evaluating flake in .
🚀 ℹ️ [deploy] [INFO] The following profiles are going to be deployed:
[servers2.system]
user = "root"
ssh_user = "root"
path = "/nix/store/qbnn9c9d1wsfmhkabfm20x3kna1vkjrr-activatable-nixos-system-servers2-24.11.20250106.3f0a8ac"
hostname = "servers2.pri"
ssh_opts = []

🚀 ℹ️ [deploy] [INFO] Building profile `system` for node `servers2`
🚀 ℹ️ [deploy] [INFO] Copying profile `system` to node `servers2`
🚀 ℹ️ [deploy] [INFO] Activating profile `system` for node `servers2`
🚀 ℹ️ [deploy] [INFO] Creating activation waiter
👀 ℹ️ [wait] [INFO] Waiting for confirmation event...
⭐ ℹ️ [activate] [INFO] Activating profile
stopping the following units: penguin-nurse.service
activating the configuration...
[agenix] creating new generation in /run/agenix.d/5
[agenix] decrypting secrets...
decrypting '/nix/store/65pjyhv87mcj2ijq6hh6byqw0wwr0nnv-backup_de1292.age' to '/run/agenix.d/5/backup_de1292'...
decrypting '/nix/store/fjhhd725phyiga091l617j6xis0zk4dw-backup_minion.age' to '/run/agenix.d/5/backup_minion'...
decrypting '/nix/store/i3r9q1dcmhs4jb2wyav5ibxqvqrkn28y-grafana.age' to '/run/agenix.d/5/dex_grafana'...
decrypting '/nix/store/i3r9q1dcmhs4jb2wyav5ibxqvqrkn28y-grafana.age' to '/run/agenix.d/5/dex_grafana_oauth'...
decrypting '/nix/store/5865w0x4gi5j2zf066mpwapilwi2blzv-nurse.age' to '/run/agenix.d/5/dex_nurse'...
decrypting '/nix/store/08xw1fhpbnh09542hdxbkah2razn48mg-phonedb.age' to '/run/agenix.d/5/dex_phonedb'...
decrypting '/nix/store/ink7l856w5s6ivg54sy938rk5s7n8c7z-photos.age' to '/run/agenix.d/5/dex_photos'...
decrypting '/nix/store/5vzb5crpjb324m8kg04x6fi0s04wkp9n-robotica.age' to '/run/agenix.d/5/dex_robotica'...
decrypting '/nix/store/75g65hijg1nh0d9kd3cf5lia5w6kn63c-scrooge.age' to '/run/agenix.d/5/dex_scrooge'...
decrypting '/nix/store/bmy9zw3gfp961hcfs5718msbqf8v4106-linuxpenguins.age' to '/run/agenix.d/5/linuxpenguins'...
decrypting '/nix/store/qs2kpf8wnzf55j9m2i7m135rb4nwvpb7-openobserve.age' to '/run/agenix.d/5/openobserve'...
decrypting '/nix/store/cywvpbqh0kp2m44da3lxwn1k3fxvch34-penguin_memories.age' to '/run/agenix.d/5/penguin_memories'...
decrypting '/nix/store/ayr1cq0hqnmdycvzxm1nsdq4hbinn4bb-penguin_nurse.age' to '/run/agenix.d/5/penguin_nurse'...
decrypting '/nix/store/x8cp4dz45irc9vm93s3ah2rpvd64cjn3-phone_db.age' to '/run/agenix.d/5/phone_db'...
decrypting '/nix/store/flm5v2sfcaz26j5gq57n9snvzyb2flxj-pri.age' to '/run/agenix.d/5/pri'...
decrypting '/nix/store/4m80lgzjmf52h3f8x8p1gx3v59pyrrp3-root_password.age' to '/run/agenix.d/5/root_password'...
decrypting '/nix/store/l65kpa070bbdqasdggyima3132nfv04c-sasl_passwd.age' to '/run/agenix.d/5/sasl_passwd'...
[agenix] symlinking new secrets to /run/agenix (generation 5)...
[agenix] removing old secrets (generation 4)...
[agenix] chowning...
setting up /etc...
reloading user units for root...
restarting sysinit-reactivation.target
Failed to restart sysinit-reactivation.target: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
starting the following units: penguin-nurse.service
⭐ ⚠️ [activate] [WARN] De-activating due to error
switching profile from version 199 to 198
⭐ ⚠️ [activate] [WARN] Removing generation by ID 199
removing profile version 199
⭐ ℹ️ [activate] [INFO] Attempting to re-activate the last generation
stopping the following units: penguin-nurse.service
activating the configuration...
[agenix] creating new generation in /run/agenix.d/6
[agenix] decrypting secrets...
decrypting '/nix/store/65pjyhv87mcj2ijq6hh6byqw0wwr0nnv-backup_de1292.age' to '/run/agenix.d/6/backup_de1292'...
decrypting '/nix/store/fjhhd725phyiga091l617j6xis0zk4dw-backup_minion.age' to '/run/agenix.d/6/backup_minion'...
decrypting '/nix/store/i3r9q1dcmhs4jb2wyav5ibxqvqrkn28y-grafana.age' to '/run/agenix.d/6/dex_grafana'...
decrypting '/nix/store/i3r9q1dcmhs4jb2wyav5ibxqvqrkn28y-grafana.age' to '/run/agenix.d/6/dex_grafana_oauth'...
decrypting '/nix/store/5865w0x4gi5j2zf066mpwapilwi2blzv-nurse.age' to '/run/agenix.d/6/dex_nurse'...
decrypting '/nix/store/08xw1fhpbnh09542hdxbkah2razn48mg-phonedb.age' to '/run/agenix.d/6/dex_phonedb'...
decrypting '/nix/store/ink7l856w5s6ivg54sy938rk5s7n8c7z-photos.age' to '/run/agenix.d/6/dex_photos'...
decrypting '/nix/store/5vzb5crpjb324m8kg04x6fi0s04wkp9n-robotica.age' to '/run/agenix.d/6/dex_robotica'...
decrypting '/nix/store/75g65hijg1nh0d9kd3cf5lia5w6kn63c-scrooge.age' to '/run/agenix.d/6/dex_scrooge'...
decrypting '/nix/store/bmy9zw3gfp961hcfs5718msbqf8v4106-linuxpenguins.age' to '/run/agenix.d/6/linuxpenguins'...
decrypting '/nix/store/qs2kpf8wnzf55j9m2i7m135rb4nwvpb7-openobserve.age' to '/run/agenix.d/6/openobserve'...
decrypting '/nix/store/cywvpbqh0kp2m44da3lxwn1k3fxvch34-penguin_memories.age' to '/run/agenix.d/6/penguin_memories'...
decrypting '/nix/store/ayr1cq0hqnmdycvzxm1nsdq4hbinn4bb-penguin_nurse.age' to '/run/agenix.d/6/penguin_nurse'...
decrypting '/nix/store/x8cp4dz45irc9vm93s3ah2rpvd64cjn3-phone_db.age' to '/run/agenix.d/6/phone_db'...
decrypting '/nix/store/flm5v2sfcaz26j5gq57n9snvzyb2flxj-pri.age' to '/run/agenix.d/6/pri'...
decrypting '/nix/store/4m80lgzjmf52h3f8x8p1gx3v59pyrrp3-root_password.age' to '/run/agenix.d/6/root_password'...
decrypting '/nix/store/l65kpa070bbdqasdggyima3132nfv04c-sasl_passwd.age' to '/run/agenix.d/6/sasl_passwd'...
[agenix] symlinking new secrets to /run/agenix (generation 6)...
[agenix] removing old secrets (generation 5)...
[agenix] chowning...
setting up /etc...
reloading user units for root...
restarting sysinit-reactivation.target
starting the following units: penguin-nurse.service
⭐ ❌ [activate] [ERROR] The activation script resulted in a bad exit code: Some(4)
🚀 ❌ [deploy] [ERROR] Activating over SSH resulted in a bad exit code: Some(1)
🚀 ℹ️ [deploy] [INFO] Revoking previous deploys
🚀 ❌ [deploy] [ERROR] Deployment to node servers2 failed, rolled back to previous generation

If I do the same command again it works fine the 2nd time.

There is Nixos-switch rebuild quite long to run restarting sysinit-reactivation.target - #3 by tobiasBora but this didn’t help. As far as I am aware there are no errors in my configuration. And I do not have virtualisation.vmware.guest.enable set on these boxes.

I have been looking through the journal logs with journalctl, but can’t find anything.

My suspicion is there is a timeout somewhere that is too short.

Any ideas?

1 Like
2

I’m seeing this as well for a few weeks. Restarting the computer seems to help for some time.

3

Do you use deploy-rs?

4

No, just nixos-rebuild switch but with --target-host (remote machine over ssh), but so far it’s always the same machine where this happens.

5

OK, good to here. Was worried it might be deploy-rs specific, as unlikely as that might seem.

Not 100% sure, but yes, I think it always is the same machine for me too.

Every time after the failure it works perfectly. Rebooting doesn’t seem to solve the issue for me.

6

For me the problem persists until I restart the target machine

7

I would have a far better chance of debugging if it occurred every time. :frowning:

I tried just rebooting and then did a deploy. Which would normally have failed. But it worked. But unclear if this is coincidence. Or because I rebooted.

I just noticed deploy-rs has an --activation-timeout parameter. Which I am guessing gets passed to the nixos activation script somehow. Wonder what that defaults to. Maybe it is too small?