> Note; this was moved from Issue [449939](https://github.com/NixOS/nixpkgs/issu…es/449939) because is sufficiently unique to likely be a different bug.
I do not know which packages to `git blame` so this Issue is without aid of templates, however, due to what ever bug causing a near bricked device I hope y'all can forgive me x-)
## TLDR -- symptoms of broken bootloader and/or kernel causing bootloop
- Press power button
- Backlight turns on
- Fans start spinning
- Flash of hits of maybe text
- Backlight turns off
- Fans stop spinning
- Backlight turns on... and repeats from there
This is very, **very**, bad because some manufactures have made choices to not enable BIOS menu on boot when in such a state!
No amount of _F_ buttoning, or mashing other keys, results in popping a BIOS menu.
Meaning if users do not have an alternate drive **already** configured as a boot option in the BIOS menu, then device is a **brick**
## TLDR -- steps I took to un-scruch-this-pooch
- Get screw-driver, use choice words, and unplug internal drive
- Plugin USB bootable drive previously configured in BIOS when things did work
- Power-up device and boot into USB
- Reboot into BIOS
- Reorder boot priority to prefer USB drive
- Turn off device
- Plugin internal drive and re-screw screws
- Power-up device an boot into USB
- Use VM with hardware pass-through to boot internal drive
- Question why, really, why will it boot in VM but not on bare-mettle... but don't question it too deeply
- SSH into VM booted from internal drive
- Find where things when wrong -> `pushd /etc/nixos && git log`
- Checkout previous commit -> `git checkout <knownGoodHash>`
- Rebuild OS -> `nixos-rebuild --impure --flake . switch`
- Reinstall bootloader -> `nixos-rebuild --impure --flake . --install-bootloader boot`
- Shutdown NixOS VM then Arch host
- Unplug USB with Arch installed on it
- Cross fingers, and tows, while powering-up into NixOS on bare mettle
---
## Specs and logs and things
<details><summary>`uname -a`</summary>
```
[nixos nixos]# uname -a
Linux nixos 6.17.4 #1-NixOS SMP PREEMPT_DYNAMIC Sun Oct 19 14:37:45 UTC 2025 x86_64 GNU/Linux
```
</details>
<details><summary>`journalctl` snippets after noticing things going wrong</summary>
```
warning: the following units failed: hdapsd@sda.service
× hdapsd@sda.service - hdapsd hard drive active protection system daemon - sda
Loaded: loaded (/etc/systemd/system/hdapsd@.service; static)
Active: failed (Result: exit-code) since Fri 2025-10-24 09:58:12 PDT; 9s ago
Duration: 178ms
Invocation: 9c8d6cc4992f486e8fd48ac18ae4b336
Docs: man:hdapsd(8)
Process: 4666 ExecStart=/nix/store/0zpv7y3kdp50an4dx6128hk64xa69l4h-hdapsd-20141203/bin/hdapsd -d sda (code=exited, status=255/EXCEPTION)
Main PID: 4666 (code=exited, status=255/EXCEPTION)
IP: 0B in, 0B out
IO: 104K read, 0B written
Mem peak: 1.8M
CPU: 79ms
Oct 24 09:58:12 nixos systemd[1]: Started hdapsd hard drive active protection system daemon - sda.
Oct 24 09:58:12 nixos hdapsd(sda)[4666]: Fri Oct 24 09:58:12 2025: Starting hdapsd
Oct 24 09:58:12 nixos hdapsd(sda)[4666]: Fri Oct 24 09:58:12 2025: Could not find a suitable interface
Oct 24 09:58:12 nixos systemd[1]: hdapsd@sda.service: Main process exited, code=exited, status=255/EXCEPTION
Oct 24 09:58:12 nixos systemd[1]: hdapsd@sda.service: Failed with result 'exit-code'.
```
```
Oct 24 09:56:42 nixos systemd-modules-load[3944]: Module 'ctr' is built in
Oct 24 09:56:42 nixos systemd-modules-load[3944]: Failed to find module 'hdapsd'
Oct 24 09:56:42 nixos acpid[3921]: 3 rules loaded
Oct 24 09:56:42 nixos acpid[3921]: waiting for events: event logging is off
Oct 24 09:56:42 nixos systemd[1]: Finished Load Kernel Modules.
Oct 24 09:56:42 nixos systemd[1]: Starting Apply Kernel Variables...
Oct 24 09:56:42 nixos systemd[1]: Started RealtimeKit Scheduling Policy Service.
Oct 24 09:56:42 nixos systemd[1]: Finished Coldplug All udev Devices.
Oct 24 09:56:42 nixos systemd-oomd[3947]: No swap; memory pressure usage will be degraded
...
Oct 24 09:56:43 nixos hm-activate-s0ands0[3926]: Activating checkLinkTargets
Oct 24 09:56:43 nixos kwin_wayland[2503]: Received unexpected add udev event for: "/dev/dri/card1"
Oct 24 09:56:43 nixos kwin_wayland[2503]: failed to open drm device at ""
Oct 24 09:56:43 nixos kwin_wayland[2503]: failed to open drm device at ""
Oct 24 09:56:43 nixos kwin_wayland[2503]: failed to open drm device at ""
Oct 24 09:56:43 nixos kwin_wayland[2503]: failed to open drm device at ""
...
Oct 24 09:56:44 nixos hm-activate-s0ands0[3926]: Activating reloadSystemd
Oct 24 09:56:44 nixos hm-activate-s0ands0[3926]: The user systemd session is degraded:
Oct 24 09:56:44 nixos hm-activate-s0ands0[4607]: UNIT LOAD ACTIVE SUB DESCRIPTION
Oct 24 09:56:44 nixos hm-activate-s0ands0[4607]: ● plasma-ksplash.service loaded failed failed Splash screen shown during boot
Oct 24 09:56:44 nixos hm-activate-s0ands0[4607]: ● nix-gc.timer not-found failed failed nix-gc.timer
Oct 24 09:56:44 nixos hm-activate-s0ands0[4607]: Legend: LOAD → Reflects whether the unit definition was properly loaded.
Oct 24 09:56:44 nixos hm-activate-s0ands0[4607]: ACTIVE → The high-level unit activation state, i.e. generalization of SUB.
Oct 24 09:56:44 nixos hm-activate-s0ands0[4607]: SUB → The low-level unit activation state, values depend on unit type.
Oct 24 09:56:44 nixos hm-activate-s0ands0[4607]: 2 loaded units listed.
Oct 24 09:56:44 nixos hm-activate-s0ands0[3926]: Attempting to reload services anyway...
Oct 24 09:56:44 nixos udisksd[3917]: Error probing device: Error sending ATA command IDENTIFY PACKET DEVICE to '/dev/sr0': Unexpected sense data returned:
0000: f0 00 0b 00 58 00 01 0a 00 00 00 00 00 00 00 00 ....X...........
0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
(g-io-error-quark, 0)
Oct 24 09:56:44 nixos thermald[3949]: [1761325004][WARN]Thermal DTS: No coretemp sysfs found
Oct 24 09:56:44 nixos thermald[3949]: [1761325004][MSG]Using config file /nix/store/11f5pz63dfax9y9495zayvb88vvp5jsp-thermald-2.5.10/etc/thermald/thermal-conf.xml
Oct 24 09:56:45 nixos thermald[3949]: [1761325005][ERR]Thermal DTS or hwmon: No Zones present Need to configure manually
Oct 24 09:56:45 nixos thermald[3949]: [1761325005][ERR]No thermal sensors found
Oct 24 09:56:45 nixos thermald[3949]: [1761325005][ERR]THD engine init failed
Oct 24 09:56:45 nixos thermald[3949]: [1761325005][MSG]Polling mode is enabled: 4
Oct 24 09:56:45 nixos tor[4199]: Oct 24 09:56:45.000 [notice] Set list of supported TLS groups to: ?*X25519MLKEM768 / ?SecP256r1MLKEM768 / *P-256:?X25519:P-224
Oct 24 09:56:45 nixos tor[4199]: Oct 24 09:56:45.000 [notice] Bootstrapped 0% (starting): Starting
Oct 24 09:56:45 nixos hm-activate-s0ands0[4608]: The service manager is degraded.
Oct 24 09:56:45 nixos hm-activate-s0ands0[4608]: Failed services: plasma-ksplash.service, nix-gc.timer
Oct 24 09:56:45 nixos hm-activate-s0ands0[4608]: Attempting to continue anyway...
Oct 24 09:56:45 nixos systemd[2148]: Reload requested from client PID 4608 ('sd-switch')...
...
Oct 24 09:56:48 nixos ModemManager[3916]: <msg> [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:02.1/0000:03:00.0': not supported by any plugin
Oct 24 09:56:53 nixos systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Oct 24 09:56:53 nixos systemd[1]: NetworkManager-dispatcher.service: Consumed 27ms CPU time, 6.9M memory peak, 10.1M read from disk.
Oct 24 09:57:08 nixos NetworkManager[3920]: <warn> [1761325028.5418] hostname: only one level of symbolic link indirection is allowed when monitoring /etc/hostname
Oct 24 09:57:08 nixos dbus-daemon[1173]: [system] Failed to activate service 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms)
Oct 24 09:57:08 nixos dbus-daemon[1173]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.74' (uid=0 pid=3920 comm="/nix/store/rilcccdw42a5wj4h>
Oct 24 09:57:08 nixos systemd[1]: Started Network Manager.
...
Oct 24 09:58:12 nixos systemd[1]: dev-disk-by\x2duuid-<UUID>.device: Job dev-disk-by\x2duuid-<UUID>.device/start timed out.
Oct 24 09:58:12 nixos systemd[1]: Timed out waiting for device /dev/disk/by-uuid/<UUID>.
Oct 24 09:58:12 nixos systemd[1]: Dependency failed for /dev/disk/by-uuid/<UUID>.
Oct 24 09:58:12 nixos systemd[1]: Dependency failed for Swaps.
Oct 24 09:58:12 nixos systemd[1]: swap.target: Job swap.target/start failed with result 'dependency'.
Oct 24 09:58:12 nixos systemd[1]: dev-disk-by\x2duuid-<UUID>.swap: Job dev-disk-by\x2duuid-<UUID>.swap/start failed with result 'dependency'.
Oct 24 09:58:12 nixos systemd[1]: dev-disk-by\x2duuid-<UUID>.device: Job dev-disk-by\x2duuid-<UUID>.device/start failed with result 'timeout'.
Oct 24 09:58:12 nixos systemd[1]: Started hdapsd hard drive active protection system daemon - sda.
Oct 24 09:58:12 nixos systemd[1]: Starting Network Manager Script Dispatcher Service...
Oct 24 09:58:12 nixos systemd[1]: Starting Ensure that NetworkManager declarative profiles are created...
Oct 24 09:58:12 nixos systemd[1]: Starting Network Manager Wait Online...
Oct 24 09:58:12 nixos systemd[1]: Starting libvirt legacy monolithic daemon...
Oct 24 09:58:12 nixos systemd[1]: Starting Name Service Cache Daemon (nsncd)...
Oct 24 09:58:12 nixos systemd[1]: Starting Hostname Service...
Oct 24 09:58:12 nixos systemd[1]: Started Network Manager Script Dispatcher Service.
Oct 24 09:58:12 nixos nsncd[4671]: Oct 24 16:58:12.445 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket"
Oct 24 09:58:12 nixos systemd[1]: Started Name Service Cache Daemon (nsncd).
Oct 24 09:58:12 nixos systemd[1]: Reached target Host and Network Name Lookups.
Oct 24 09:58:12 nixos systemd[1]: Reached target User and Group Name Lookups.
Oct 24 09:58:12 nixos systemd[1]: Starting Accounts Service...
Oct 24 09:58:12 nixos systemd[1]: Finished Network Manager Wait Online.
Oct 24 09:58:12 nixos systemd-timesyncd[3959]: Contacted time server 149.28.61.105:123 (0.nixos.pool.ntp.org).
Oct 24 09:58:12 nixos hdapsd(sda)[4666]: Fri Oct 24 09:58:12 2025: Starting hdapsd
Oct 24 09:58:12 nixos hdapsd(sda)[4666]: Fri Oct 24 09:58:12 2025: Could not find a suitable interface
Oct 24 09:58:12 nixos systemd[1]: hdapsd@sda.service: Main process exited, code=exited, status=255/EXCEPTION
Oct 24 09:58:12 nixos systemd[1]: hdapsd@sda.service: Failed with result 'exit-code'.
...
Oct 24 09:58:21 nixos nixos[3492]: switching to system configuration /nix/store/hzbi01hlk8aj108590xw8mq30hxlj43r-nixos-system-nixos-25.11.20251015.544961d failed (status 4)
Oct 24 09:58:21 nixos systemd[1]: nixos-rebuild-switch-to-configuration.service: Main process exited, code=exited, status=4/NOPERMISSION
Oct 24 09:58:21 nixos systemd[1]: nixos-rebuild-switch-to-configuration.service: Failed with result 'exit-code'.
Oct 24 09:58:21 nixos systemd[1]: nixos-rebuild-switch-to-configuration.service: Consumed 1.333s CPU time, 101M memory peak, 181.7M read from disk, 458.5K written to disk.
```
</details>
<details><summary>`flake.lock` **good**</summary>
```nix
{
"nodes": {
"flake-parts": {
"inputs": {
"nixpkgs-lib": [
"nur",
"nixpkgs"
]
},
"locked": {
"lastModified": 1733312601,
"narHash": "sha256-4pDvzqnegAfRkPwO3wmwBhVi/Sye1mzps0zHWYnP88c=",
"owner": "hercules-ci",
"repo": "flake-parts",
"rev": "205b12d8b7cd4802fbcb8e8ef6a0f1408781a4f9",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "flake-parts",
"type": "github"
}
},
"home-manager": {
"inputs": {
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1760500983,
"narHash": "sha256-zfY4F4CpeUjTGgecIJZ+M7vFpwLc0Gm9epM/iMQd4w8=",
"owner": "nix-community",
"repo": "home-manager",
"rev": "c53e65ec92f38d30e3c14f8d628ab55d462947aa",
"type": "github"
},
"original": {
"owner": "nix-community",
"repo": "home-manager",
"type": "github"
}
},
"nixos-hardware": {
"locked": {
"lastModified": 1760106635,
"narHash": "sha256-2GoxVaKWTHBxRoeUYSjv0AfSOx4qw5CWSFz2b+VolKU=",
"owner": "NixOS",
"repo": "nixos-hardware",
"rev": "9ed85f8afebf2b7478f25db0a98d0e782c0ed903",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "master",
"repo": "nixos-hardware",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1760524057,
"narHash": "sha256-EVAqOteLBFmd7pKkb0+FIUyzTF61VKi7YmvP1tw4nEw=",
"owner": "nixos",
"repo": "nixpkgs",
"rev": "544961dfcce86422ba200ed9a0b00dd4b1486ec5",
"type": "github"
},
"original": {
"owner": "nixos",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"nur": {
"inputs": {
"flake-parts": "flake-parts",
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1760561137,
"narHash": "sha256-G2x6BFCv9CD7V8oiANSSddYoqJZ2NXyFvuCYnk9WI44=",
"owner": "nix-community",
"repo": "NUR",
"rev": "cd6b2f26b608c71b7ad3c4809a3ea4f72d32f985",
"type": "github"
},
"original": {
"owner": "nix-community",
"repo": "NUR",
"type": "github"
}
},
"root": {
"inputs": {
"home-manager": "home-manager",
"nixos-hardware": "nixos-hardware",
"nixpkgs": "nixpkgs",
"nur": "nur",
"sops-nix": "sops-nix"
}
},
"sops-nix": {
"inputs": {
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1760393368,
"narHash": "sha256-8mN3kqyqa2PKY0wwZ2UmMEYMcxvNTwLaOrrDsw6Qi4E=",
"owner": "Mic92",
"repo": "sops-nix",
"rev": "ab8d56e85b8be14cff9d93735951e30c3e86a437",
"type": "github"
},
"original": {
"owner": "Mic92",
"repo": "sops-nix",
"type": "github"
}
}
},
"root": "root",
"version": 7
}
```
</details>
<details><summary>`flake.lock` **bad**</summary>
```nix
{
"nodes": {
"flake-parts": {
"inputs": {
"nixpkgs-lib": [
"nur",
"nixpkgs"
]
},
"locked": {
"lastModified": 1733312601,
"narHash": "sha256-4pDvzqnegAfRkPwO3wmwBhVi/Sye1mzps0zHWYnP88c=",
"owner": "hercules-ci",
"repo": "flake-parts",
"rev": "205b12d8b7cd4802fbcb8e8ef6a0f1408781a4f9",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "flake-parts",
"type": "github"
}
},
"home-manager": {
"inputs": {
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1761235135,
"narHash": "sha256-cux9xeceLIER1lBxUa1gMafkz7gg5ntcUmJBynWdBWI=",
"owner": "nix-community",
"repo": "home-manager",
"rev": "0adf9ba3f567da2d53af581a857aacf671aaa547",
"type": "github"
},
"original": {
"owner": "nix-community",
"repo": "home-manager",
"type": "github"
}
},
"nixos-hardware": {
"locked": {
"lastModified": 1760958188,
"narHash": "sha256-2m1S4jl+GEDtlt2QqeHil8Ny456dcGSKJAM7q3j/BFU=",
"owner": "NixOS",
"repo": "nixos-hardware",
"rev": "d6645c340ef7d821602fd2cd199e8d1eed10afbc",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "master",
"repo": "nixos-hardware",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1761114652,
"narHash": "sha256-f/QCJM/YhrV/lavyCVz8iU3rlZun6d+dAiC3H+CDle4=",
"owner": "nixos",
"repo": "nixpkgs",
"rev": "01f116e4df6a15f4ccdffb1bcd41096869fb385c",
"type": "github"
},
"original": {
"owner": "nixos",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"nur": {
"inputs": {
"flake-parts": "flake-parts",
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1761255338,
"narHash": "sha256-26s/zsHCXlzqR0rswpOPHoP91WZCgHOHxlM+LSx85ZQ=",
"owner": "nix-community",
"repo": "NUR",
"rev": "6bfa7d6468a76c71a2294a12e9e163cbb8b824af",
"type": "github"
},
"original": {
"owner": "nix-community",
"repo": "NUR",
"type": "github"
}
},
"root": {
"inputs": {
"home-manager": "home-manager",
"nixos-hardware": "nixos-hardware",
"nixpkgs": "nixpkgs",
"nur": "nur",
"sops-nix": "sops-nix"
}
},
"sops-nix": {
"inputs": {
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1760998189,
"narHash": "sha256-ee2e1/AeGL5X8oy/HXsZQvZnae6XfEVdstGopKucYLY=",
"owner": "Mic92",
"repo": "sops-nix",
"rev": "5a7d18b5c55642df5c432aadb757140edfeb70b3",
"type": "github"
},
"original": {
"owner": "Mic92",
"repo": "sops-nix",
"type": "github"
}
}
},
"root": "root",
"version": 7
}
```
</details>
<details><summary>Diff good vs bad `flake.lock` files</summary>
```diff
diff --git a/flake.lock b/tmp/flake.lock_bad
index 632942a..5c8eabb 100644
--- a/flake.lock
+++ b/tmp/flake.lock_bad
@@ -28,11 +28,11 @@
]
},
"locked": {
- "lastModified": 1760500983,
- "narHash": "sha256-zfY4F4CpeUjTGgecIJZ+M7vFpwLc0Gm9epM/iMQd4w8=",
+ "lastModified": 1761235135,
+ "narHash": "sha256-cux9xeceLIER1lBxUa1gMafkz7gg5ntcUmJBynWdBWI=",
"owner": "nix-community",
"repo": "home-manager",
- "rev": "c53e65ec92f38d30e3c14f8d628ab55d462947aa",
+ "rev": "0adf9ba3f567da2d53af581a857aacf671aaa547",
"type": "github"
},
"original": {
@@ -43,11 +43,11 @@
},
"nixos-hardware": {
"locked": {
- "lastModified": 1760106635,
- "narHash": "sha256-2GoxVaKWTHBxRoeUYSjv0AfSOx4qw5CWSFz2b+VolKU=",
+ "lastModified": 1760958188,
+ "narHash": "sha256-2m1S4jl+GEDtlt2QqeHil8Ny456dcGSKJAM7q3j/BFU=",
"owner": "NixOS",
"repo": "nixos-hardware",
- "rev": "9ed85f8afebf2b7478f25db0a98d0e782c0ed903",
+ "rev": "d6645c340ef7d821602fd2cd199e8d1eed10afbc",
"type": "github"
},
"original": {
@@ -59,11 +59,11 @@
},
"nixpkgs": {
"locked": {
- "lastModified": 1760524057,
- "narHash": "sha256-EVAqOteLBFmd7pKkb0+FIUyzTF61VKi7YmvP1tw4nEw=",
+ "lastModified": 1761114652,
+ "narHash": "sha256-f/QCJM/YhrV/lavyCVz8iU3rlZun6d+dAiC3H+CDle4=",
"owner": "nixos",
"repo": "nixpkgs",
- "rev": "544961dfcce86422ba200ed9a0b00dd4b1486ec5",
+ "rev": "01f116e4df6a15f4ccdffb1bcd41096869fb385c",
"type": "github"
},
"original": {
@@ -81,11 +81,11 @@
]
},
"locked": {
- "lastModified": 1760561137,
- "narHash": "sha256-G2x6BFCv9CD7V8oiANSSddYoqJZ2NXyFvuCYnk9WI44=",
+ "lastModified": 1761255338,
+ "narHash": "sha256-26s/zsHCXlzqR0rswpOPHoP91WZCgHOHxlM+LSx85ZQ=",
"owner": "nix-community",
"repo": "NUR",
- "rev": "cd6b2f26b608c71b7ad3c4809a3ea4f72d32f985",
+ "rev": "6bfa7d6468a76c71a2294a12e9e163cbb8b824af",
"type": "github"
},
"original": {
@@ -110,11 +110,11 @@
]
},
"locked": {
- "lastModified": 1760393368,
- "narHash": "sha256-8mN3kqyqa2PKY0wwZ2UmMEYMcxvNTwLaOrrDsw6Qi4E=",
+ "lastModified": 1760998189,
+ "narHash": "sha256-ee2e1/AeGL5X8oy/HXsZQvZnae6XfEVdstGopKucYLY=",
"owner": "Mic92",
"repo": "sops-nix",
- "rev": "ab8d56e85b8be14cff9d93735951e30c3e86a437",
+ "rev": "5a7d18b5c55642df5c432aadb757140edfeb70b3",
"type": "github"
},
"original": {
```
</details>