Systemd-boot failing to boot a NixOS Warbler system due to an EFI assertion failure

Help
1

Hi folks!
I’ve recently upgraded my NixOS config’ nixpkgs (nixos-unstable branch) input from 99dc8785f6a0adac95f5e2ab05cc2e1bf666d172 (Vicuna) to 36fd87baa9083f34f7f5027900b62ee6d09b1f2f (Warbler). systemd-boot is able to list the generations just fine. Picking the Warbler generation makes this message immediately appear:

../src/boot/log.c:30@efi_assert: systemd-boot: Assertion 'BS->AllocatePages(type, memory_type, n_pages, &addr) == EFI_SUCCESS' failed at ../src/boot/util.h:95@xmalloc_pages, halting.

Picking the previous Vicuna generation boots the system as expected. I’ve tried:

  • Re-installing systemd-boot (by setting the NIXOS_REINSTALL_BOOTLOADER=1 environment variable & calling bin/switch-to-configuration on my Warbler build)
  • Enabling CSM
  • Disabling TPM

All of these still lead to the aforementioned EFI assertion message with the Warbler generation. I’m pretty inclined to say this cannot be a hardware or firmware issue, because my previous generation is booting just fine.

I’m pretty stumped on how I should proceed right now. If anyone’s got any pointers on what I should try, it’d be greatly appreciated! Thanks!

Output of bootctl:

System:
      Firmware: UEFI 2.80 (American Megatrends 5.27)
 Firmware Arch: x64
   Secure Boot: disabled
  TPM2 Support: yes
  Measured UKI: no
  Boot into FW: supported

Current Boot Loader:
      Product: systemd-boot 257.2
     Features: ✓ Boot counting
               ✓ Menu timeout control
               ✓ One-shot menu timeout control
               ✓ Default entry control
               ✓ One-shot entry control
               ✓ Support for XBOOTLDR partition
               ✓ Support for passing random seed to OS
               ✓ Load drop-in drivers
               ✓ Support Type #1 sort-key field
               ✓ Support @saved pseudo-entry
               ✓ Support Type #1 devicetree field
               ✓ Enroll SecureBoot keys
               ✓ Retain SHIM protocols
               ✓ Menu can be disabled
               ✓ Boot loader sets ESP information
          ESP: /dev/disk/by-partuuid/e4ac4157-776b-477d-a011-a338f849f673
         File: └─/EFI/systemd/systemd-bootx64.efi

Random Seed:
 System Token: set
       Exists: yes

Available Boot Loaders on ESP:
          ESP: /boot (/dev/disk/by-partuuid/e4ac4157-776b-477d-a011-a338f849f673)
         File: ├─/EFI/systemd/systemd-bootx64.efi (systemd-boot 257.2)
               └─/EFI/BOOT/BOOTX64.EFI (systemd-boot 257.2)

Boot Loaders Listed in EFI Variables:
        Title: Linux Boot Manager
           ID: 0x0000
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/e4ac4157-776b-477d-a011-a338f849f673
         File: └─/EFI/systemd/systemd-bootx64.efi

        Title: Windows Boot Manager
           ID: 0x0003
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/daacebb2-f8bd-4e9b-aebd-0f0943c9ca1c
         File: └─/EFI/Microsoft/Boot/bootmgfw.efi

        Title: UEFI OS
           ID: 0x0002
       Status: active, boot-order
    Partition: /dev/disk/by-partuuid/e4ac4157-776b-477d-a011-a338f849f673
         File: └─/EFI/BOOT/BOOTX64.EFI

Boot Loader Entries:
        $BOOT: /boot (/dev/disk/by-partuuid/e4ac4157-776b-477d-a011-a338f849f673)
        token: nixos

Default Boot Loader Entry:
         type: Boot Loader Specification Type #1 (.conf)
        title: NixOS (Generation 94 NixOS Warbler 25.05pre-git (Linux 6.12.17), built on 2025-03-10)
           id: nixos-generation-94.conf
       source: /boot//loader/entries/nixos-generation-94.conf
     sort-key: nixos
      version: Generation 94 NixOS Warbler 25.05pre-git (Linux 6.12.17), built on 2025-03-10
   machine-id: 530f8732f2a44b4d9d2e99c4eee9f9d2
        linux: /boot//EFI/nixos/i6692qs0b61zmc83a34nj9jd361jndrv-linux-6.12.17-bzImage.efi
       initrd: /boot//EFI/nixos/nb07lcgjbfj90zi8vcklq0wlkxk1fg80-initrd-linux-6.12.17-initrd.efi
      options: init=/nix/store/g67838l3016xk6mgdpslsh61p87lfqbi-nixos-system-michaili-fortress-25.05pre-git/init intel_iommu=on vfio.pci.ids=10de:2504,10de:228e,144d:a80a split_lock_detect=warn splash loglevel=4

neither generations seem to differ from systemd-boot’s point of view, other than kernel & init:

# cat /boot/loader/entries/nixos-generation-93.conf 
title NixOS
sort-key nixos
version Generation 93 NixOS Vicuna 24.11pre-git (Linux 6.6.52), built on 2024-12-06
linux /EFI/nixos/xpx5f3c4n29pgn6a4rc8r4xhghsk3iyf-linux-6.6.52-bzImage.efi
initrd /EFI/nixos/fsmqhml8ipg6x30v0292nhc6arql3kaw-initrd-linux-6.6.52-initrd.efi
options init=/nix/store/55cnawk128ll6bqn7gwgmlafpn2n7544-nixos-system-michaili-fortress-24.11pre-git/init intel_iommu=on vfio.pci.ids=10de:2504,10de:228e,144d:a80a split_lock_detect=warn splash loglevel=4
machine-id 530f8732f2a44b4d9d2e99c4eee9f9d2

# cat /boot/loader/entries/nixos-generation-94.conf 
title NixOS
sort-key nixos
version Generation 94 NixOS Warbler 25.05pre-git (Linux 6.12.17), built on 2025-03-10
linux /EFI/nixos/i6692qs0b61zmc83a34nj9jd361jndrv-linux-6.12.17-bzImage.efi
initrd /EFI/nixos/nb07lcgjbfj90zi8vcklq0wlkxk1fg80-initrd-linux-6.12.17-initrd.efi
options init=/nix/store/g67838l3016xk6mgdpslsh61p87lfqbi-nixos-system-michaili-fortress-25.05pre-git/init intel_iommu=on vfio.pci.ids=10de:2504,10de:228e,144d:a80a split_lock_detect=warn splash loglevel=4
machine-id 530f8732f2a44b4d9d2e99c4eee9f9d2
2

Just made a discovery: Running sudo bootctl install with my old Vicuna generation installs systemd-boot 256.4, which lets me boot into both the new Warbler & Vicuna generation!

However, bootctl install’ing with my new generation installs systemd-boot 257.2, which breaks it again. So this must likely be a regression from systemd-boot then?

1 Like
3

What hardware is this on? It seems similar to this issue: systemd-boot 256.7 fails to boot on machines with no physical memory below 4GiB boundary · Issue #35026 · systemd/systemd · GitHub. That originally affected Apple Silicon, but an x86 user recently said they still have it.

4

This is on a x86_64 system. I’m currently trying to bisect nixpkgs & will report back once I’ve got a commit I can point to.

5

Right but we need to know the actual specific hardware for a situation like this. It’s almost certainly the same systemd update that caused the issue I linked before. To figure out a solution, we need to know what that hardware is doing weirdly.

6

@MichailiK Just try before and after this commit: systemd: 256.6 -> 256.7 · NixOS/nixpkgs@66f78a9 · GitHub

That’s the one that introduces the systemd change that probably is causing your issues.

7

Here’s lshw’s output, should hopefully be sufficient: lshw output (https://discourse.nixos.org/t/systemd-boot-failing-to-boot-a-nixos-warbler-system-due-to-an-efi-assertion-failure/61450/6) · GitHub (had to put it in a gist due to the 32k character limits on here)

Here’s a more brief output from fastfetch:

michaili@michaili-fortress
--------------------------
OS: NixOS 24.11 (Vicuna) x86_64
Host: Z690 AORUS ULTRA (-CF)
Kernel: Linux 6.6.52
Uptime: 25 mins
Packages: 1197 (nix-system), 230 (nix-user)
Shell: bash 5.2.32
Display (S24F350): 1920x1080 @ 60 Hz in 24" [External]
Display (32G1WG4): 1920x1080 @ 144 Hz in 31" [External] *
DE: KDE Plasma
WM: KWin (Wayland)
WM Theme: Breeze
Theme: Breeze (Dark) [Qt]
Icons: breeze-dark [Qt], breeze-dark [GTK2/3/4]
Font: Noto Sans (10pt) [Qt], Noto Sans (10pt) [GTK2/3/4]
Cursor: breeze (24px)
Terminal: konsole 24.8.1
CPU: 13th Gen Intel(R) Core(TM) i7-13700K (24) @ 5.40 GHz
GPU 1: NVIDIA GeForce RTX 3060 Lite Hash Rate [Discrete]
GPU 2: AMD Radeon RX 7600/7600 XT/7600M XT/7600S/7700S / PRO W7600 [Discrete]
GPU 3: Intel UHD Graphics 770 @ 1.60 GHz [Integrated]
Memory: 6.49 GiB / 31.10 GiB (21%)
Swap: 721.00 MiB / 32.00 GiB (2%)
Disk (/): 600.59 GiB / 1.78 TiB (33%) - btrfs
Local IP (enp3s0): 192.168.178.66/24
Locale: en_US.UTF-8

I’m trying to build my system with nixpkgs commit 66f78a9 at the moment moment (& the parent commit later as well). It’s gonna take a good while though because a bunch (500+) of packages dont seem to be cached & need to be built locally.

I’m not quite convinced this is a hardware issue, as I can always boot into my old NixOS Vicuna generations (regardless if I use the newer or older systemd-boot version). It’s only that newer systemd-boot versions fail to boot newer NixOS generations.

8

Yea that commit is in the middle of a staging branch. If you can find the master commits where it was branched off from and back into, those would be better.

It is and it isn’t. It’s a software bug in systemd-boot caused by the newer systemd-boot’s intolerance for certain hardware that the older version didn’t have. It was fixed for aarch64, but left the way it was for x86_64 under the assumption no x86_64 platform would have these conditions.

That said, I wouldn’t expect a 13th gen Intel CPU to have the original systemd issue in the first place, but I guess I can’t rule out the motherboard causing it? Not sure. I need to ask some more knowledgeable folks about that.

9

@MichailiK This is the staging round where that systemd bump was merged to master. Testing commits before and after this merge should be cached: staging-next 2024-10-15 by vcunat · Pull Request #348827 · NixOS/nixpkgs · GitHub

I think these were the commits that are most cached before and after that merge:

  • Older (systemd 256.6): 807e9154dcb1
  • Newer (systemd 256.7): 7ffd9ae656ae
10

Thank you for pointing me to these commits! I’ve tried booting my system using nixpkgs 807e9154dcb1 & 7ffd9ae656ae. Surprisingly, both boot (???) but I’ve observed the following behaviors as I was swapping between systemd-boot 256.6 and 256.7 & tried booting my NixOS generations:

:white_check_mark: systemd-boot 256.6 + NixOS Vicuna (807e9154dcb1) boots
:white_check_mark: systemd-boot 256.7 + NixOS Vicuna (7ffd9ae656ae) boots

:white_check_mark: systemd-boot 256.6 + NixOS Warbler (36fd87baa908) boots
:x: systemd-boot 256.7 + NixOS Warbler (36fd87baa908) shows the EFI assertion failure

Quite baffled by what’s going on here. The only next step I can think of is to “gradually” upgrade nixpkgs in my NixOS config with each systemd release, until I encounter this issue & perhaps bisect from there?

1 Like
11

Wait so you’re saying that the same 256.7 boot loader that worked to boot the Vicuna generation failed to boot the Warbler generation?

Ok, it’s probably a related but very different bug then. Somehow, something must have changed in our kernel or initrd that makes the new thing unhappy.

Can you list the sizes of the kernel and initrd files that were booted in each of the four scenarios you outlined above? We’ve had more than one bug caused by these sizes getting out of hand, but we thought those bugs were done for.

1 Like
12

@MichailiK can you try this patch please? It removes limitation of allocating below 4GB at any point in the code.

Please make a backup of your bootloader’s .efi and have a USB with a shell.efi on-hand so you can swap it back in case it fails to boot.

You can apply it as just a patch file by setting systemd.package

13
  • NixOS Vicuna (807e9154dcb1)
    • Kernel 11,203,072 (10.7 MiB)
    • initrd 29,745,860 (28.4 MiB)
  • NixOS Vicuna (7ffd9ae656ae)
    • Kernel 11,203,072 (10.7 MiB)
    • initrd 29,819,597 (28.4 MiB)
  • NixOS Warbler (36fd87baa908)
    • Kernel 1,2132,864 (11.6 MiB)
    • initrd 375,882,839 (358.5 MiB)

I’m unsure what’s changed between Vicuna & Warbler to produce such a significantly bigger initrd.

I’ve applied that patch to systemd-boot 256.7 just now, and it does let me successfully boot into the NixOS Warbler (36fd87baa908) generation now!
Anything else I can do to further track this issue down further?

14

So then it is an issue similar to systemd/systemd#35026.

Basically the gist of what’s going on (at least from my understanding) is that systemd really wants to support old versions of the Linux x86 boot protocol, which needed the kernel and related stuff (initrd) to be under 4GB). It’s long been deprecated but systemd-boot (or at least the current version of it) still tries to allocate everything under 4GB, for compatibility. There are a few places where there are checks if it has to allocate below 4GB, and if it doesn’t have to, it doesn’t, but evidently there are still some places where it still always tries to allocate below 4GB. With other allocations in that area and a 300MB initrd, and also with it reallocating and moving things a few times (causing memory fragmentation; the usable amount of contiguous memory could be much less), it’s possible for it to run out of RAM below the 4GB threshold.

The issue linked was about aarch64 machines, which always failed to allocate, since (some of them) literally have no physical memory under that 4GB boundary. So it was fixed by removing the < 4GB check for machines other than x86_64 and i386. For those machines, it was always assumed that there would be memory in that region. (and that it would be enough to fit the kernel and initrd and whatnot)

All of that to say, since this is a slightly separate issue (that only seems to happen with super large initrd size?), and the one about aarch64 machines is now closed, you should file a new one with systemd with all your system specifications and details and things. If I were to guess, the problematic commit is the same as for the aarch64 issue.

Whilst the issue is fixed you can just keep using the patch for now. It shouldn’t matter unless you plan on booting really ancient kernels (older than 3.8 – like 12+ years old at this point). You can see the patch itself is pretty simple – it just removes those < 4GB allocation restrictions. But for a more “upstreamable” and less quick-and-dirty patch, it would also probably need to remove all of the other code surrounding that legacy protocol compatibility… or maintain the compatibility and add the 4GB checks where they were missing.

15

Alright, I’ve filed systemd-boot fails to boot with a large initrd? · Issue #36706 · systemd/systemd · GitHub. Thank you very much @ElvishJerricco and @andre4ik3 for the help!

1 Like
16

Something awfully strange is going on here. A 350MB initrd is an order of magnitude beyond what I expect from NixOS (hi, I’m NixOS’s initrd maintainer). Furthermore, the idea that it would expand to exhaust 4G of RAM as @andre4ik3 says is… plausible but pretty unlikely. I think there’s two things going on here. 1) I need to know why your initrd is so large. 2) I need to know how much it expands to uncompressed; you can either expand it yourself or just share the initrd file so I can do it.

17

actually it won’t; the initrd is not expanded by systemd-boot. So the issue would have to be failing to find contiguous memory large enough for 358M below 4G, which seems unlikely but plausible.

Anyway I don’t need to know what it expands to in size because systemd-boot doesn’t use that. But regardless of the fact that systemd-boot apparently now has a bug for very large initrds, I do still need to figure out why your initrd is so large

18

Large initrd from NixOS Warbler (nixpkgs 36fd87baa908) build: Proton Drive

If it helps, here’s my smaller initrd from NixOS Vicuna (nixpkgs 99dc8785f6a) for reference: Proton Drive

My NixOS config is at GitHub - MichailiK/nix-config: A (very early) iteration of my NixOS configuration & might also be useful (it uses colmena & should be buildable using colmena build)

19

Oh wow apparently the nvidia firmware is super massive. You shouldn’t need that in initrd. Glancing at your config, I don’t see where you have it or where you have it added to initrd.

20

Oh god- yeah its here. The background of me adding that was to ensure under all circumstances that the vfio_pci module loads & binds to my Nvidia GPU first, before nouveau could, so I can use the GPU inside VMs. I’ll remove it. Thank you very much!
Mind telling me how you figured that its the nvidia firmware? When I tried using cpio to peek into my initrd files I only saw the Intel microcode:

$ cpio -vt < /boot/EFI/nixos/...-initrd-linux-6.12.17-initrd.efi
-rw-r--r--   0 root     root      9647104 Jan  1  1980 kernel/x86/microcode/GenuineIntel.bin
18843 blocks

$ cpio -vt < /boot/EFI/nixos/...-initrd-linux-6.6.52-initrd.efi
-rw-r--r--   0 root     root      8089600 Jan  1  1980 kernel/x86/microcode/GenuineIntel.bin
15801 blocks