System inoperable after automatic upgrades

Help
1

Hi!

I have a NixOS machine that’s running 24/7. It’s currently running NixOS 24.05 (with a small number of packages pulled from unstable - none of them touch the core installation and are likely to be relevant here). Also, the system is using lanzaboote and agenix.

I enabled automatic upgrades in my configuration.nix:

system.autoUpgrade = {
  enable = true;
  flake = "/etc/nixos";
  flags = [ "--update-input" "nixpkgs" ];
  randomizedDelaySec = "1h";
};

This is usually working fine. However, I now had the situation twice that the system became inoperable after performing that update during the night, meaning: The network connection went down and it no longer responded to pings.

After resetting the system and looking at the journal from the previous boot (journalctl -b -1), this is what I get:

https://faui2k11.de/random/journal-trimmed.txt

The following lines look particularly relevant:

Aug 06 05:11:00 pandora systemd[1]: nixos-rebuild-switch-to-configuration.service: Failed to open /run/systemd/transient/nixos-rebuild-switch-to-configuration.service: No such file or directory
[...]
Aug 06 05:11:00 pandora systemd[1]: nixos-rebuild-switch-to-configuration.service: Failed to open /run/systemd/transient/nixos-rebuild-switch-to-configuration.service: No such file or directory
[...]
Aug 06 05:11:05 pandora systemd[1]: Reexecuting requested from client PID 625338 ('systemctl') (unit nixos-rebuild-switch-to-configuration.service)...
Aug 06 05:11:05 pandora systemd[1]: Reexecuting.
Aug 06 05:11:05 pandora systemd[1]: systemd 255.9 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified)
Aug 06 05:11:05 pandora systemd[1]: Detected architecture x86-64.
Aug 06 05:11:05 pandora systemd[1]: bpf-lsm: LSM BPF program attached
Aug 06 05:12:35 pandora systemd[1]: Failed to fork off sandboxing environment for executing generators: Protocol error
Aug 06 05:12:35 pandora systemd[1]: Freezing execution.
Aug 06 05:13:00 pandora dbus-daemon[679]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Aug 06 05:13:00 pandora nixos-upgrade-start[625362]: Failed to execute operation: Connection timed out

Any advice on how I could debug this further?

Thanks!

1 Like