Boot problem: `rescue.target` started due to broken `systemd-system-update-generator`?

After a scheduled firmware update NixOS (22.05) always enters into the rescue.target on boot. This is true for current and previously working generations. The reported issue seems to be that the unit default.target is not found.

A failing boot:

$ journalctl -x --boot=-4 -p3
Nov 23 17:34:10 nixos systemd[1]: Unit default.target not found.
Nov 23 17:34:10 nixos kernel: dell_smm_hwmon: unable to get SMM Dell signature
Nov 23 17:34:12 nixos bluetoothd[872]: profiles/sap/server.c:sap_server_register() Sap driver initialization failed.
Nov 23 17:34:12 nixos bluetoothd[872]: sap-server: Operation not permitted (1)
Nov 23 18:04:21 nixos kernel: watchdog: watchdog0: watchdog did not stop!

By overriding systemd.unit in kernel params the following is hinted:

Nov 23 20:33:44 nixos unknown: booting system configuration /nix/store/6s32z0sf56lnbi0c5lz487krc20hmr0z-nixos-system-nixos-22.05.20221122.56ca700
Nov 23 20:33:44 nixos systemd-system-update-generator[534]: Offline system update overridden by kernel command line systemd.unit= setting

So this seems to be the systemd-system-update-generator is broken?

In the rescue.target shell systemd reports that the target default.target is a broken link. The unit is not enabled but shows up with systemctl list-unit-files as “bad”. The rescue.target cannot be skipped as suggested with Control-D because of the broken link.

When booted with kernel param systemd.unit=graphical.target systemd has the following to say about default target once booted, but I guess that may be because of the systemd.unit param?

$ systemctl get-default 
Failed to get default target: Link has been severed

$ readlink -f /etc/systemd/system/default.target  
/nix/store/jimz0md5q3l7w1nrgd2j3bi1w9l6da7l-systemd-250.4/example/systemd/system/graphical.target

I’ve tried to remove the scheduled firmware update with fwupdmgr but the NixOS version no longer has the clear-offline command.

Hosted by Flying Circus.