AMD microcode updates not applying?

TLATER · August 19, 2023, 1:12pm

This is somewhat of a necro of AMD Microcode updates not working?, but that thread seems to have died.

In light of the recent AMD speculative execution bugs (yes, more of them), my dmesg has started saying:

tlater ~ $ dmesg | grep microcode
[    0.407693] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    0.407694] Speculative Return Stack Overflow: Mitigation: safe RET, no microcode

Since I have hardware.cpu.amd.updateMicrocode set, I figured this would long have been fixed, so I guessed I needed a BIOS update or such, but after applying it that hasn’t fixed the problem.

I checked my cpuinfo:

tlater ~ $ grep 'model\|microcode' /proc/cpuinfo
model           : 33
model name      : AMD Ryzen 5 5600X 6-Core Processor
microcode       : 0xa201025

The latest appears to be 0xaa00212.

This is a bit concerning, lots of NixOS systems will be vulnerable to that speculative execution bug, despite users believing they are mitigating it. Does anyone have any idea what’s going wrong?

lelgenio · August 19, 2023, 4:31pm

Not sure how much help it is, but here are my outputs:

Cpu: ryzen 5 5600
NixOs: release 23.05
Linux: 6.1.45
hardware.cpu.amd.updateMicrocode: true

dmesg | grep microcode

[    0.235067] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    0.235068] Speculative Return Stack Overflow: Mitigation: safe RET, no microcode
[    0.584402] microcode: CPU0: patch_level=0x0a20120a
[    0.584408] microcode: CPU1: patch_level=0x0a20120a
[    0.584413] microcode: CPU2: patch_level=0x0a20120a
[    0.584419] microcode: CPU3: patch_level=0x0a20120a
[    0.584425] microcode: CPU4: patch_level=0x0a20120a
[    0.584431] microcode: CPU5: patch_level=0x0a20120a
[    0.584435] microcode: CPU6: patch_level=0x0a20120a
[    0.584440] microcode: CPU7: patch_level=0x0a20120a
[    0.584444] microcode: CPU8: patch_level=0x0a20120a
[    0.584449] microcode: CPU9: patch_level=0x0a20120a
[    0.584455] microcode: CPU10: patch_level=0x0a20120a
[    0.584460] microcode: CPU11: patch_level=0x0a20120a
[    0.584463] microcode: Microcode Update Driver: v2.2.
[21719.450738] microcode: CPU1: patch_level=0x0a20120a
[21719.454234] microcode: CPU2: patch_level=0x0a20120a
[21719.457541] microcode: CPU3: patch_level=0x0a20120a
[21719.460664] microcode: CPU4: patch_level=0x0a20120a
[21719.463836] microcode: CPU5: patch_level=0x0a20120a
[21719.467439] microcode: CPU6: patch_level=0x0a20120a
[21719.470866] microcode: CPU7: patch_level=0x0a20120a
[21719.474639] microcode: CPU8: patch_level=0x0a20120a
[21719.478174] microcode: CPU9: patch_level=0x0a20120a
[21719.482105] microcode: CPU10: patch_level=0x0a20120a
[21719.485973] microcode: CPU11: patch_level=0x0a20120a

grep 'model\|microcode' /proc/cpuinfo

model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a

bwolf · August 20, 2023, 5:20pm

I’m also puzzling about this. I’m on 23.05 and updated it now without any changes since 1 or 2 weeks.

$ nix repl
:lf .#
builtins.parseDrvName inputs.nixpkgs.legacyPackages.x86_64-linux.microcodeAmd.name
{ name = "amd-ucode"; version = "20230809"; }

CPU info:

$ grep 'model\|microcode' /proc/cpuinfo | head -3
model		: 97
model name	: AMD Ryzen 9 7900 12-Core Processor
microcode	: 0xa601203

dmesg output:

$ dmesg | grep microcode
[    0.067011] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    0.067012] Speculative Return Stack Overflow: Mitigation: safe RET, no microcode
[    0.342131] microcode: CPU0: patch_level=0x0a601203
...
[    0.342234] microcode: CPU23: patch_level=0x0a601203
[    0.342237] microcode: Microcode Update Driver: v2.2.

So it seems the firmware is too old on my system (20230809). Maybe nixpkgs for 23.05 it not recent enough?

TLATER · August 20, 2023, 8:24pm

That’s not it, I went trawling through nixpkgs, latest linux-firmware is backported since ~March this year. We’re currently on August 9 as the version number states, but at least my microcode version is 2 years (!) old.

I’ve since learned that linux-firmware contains different microcodes for a variety of amd processor families. The nixpkgs package appears to bundle them all, but maybe one is missing? My next attempt was going to be grabbing a debian/arch/gentoo package and forcing it into initrd manually to see if it’s a packaging error.

mabl · August 21, 2023, 5:31pm

I found this statement on the Phoronix forum that seems to indicate that consumer AMD CPUs only get updated by the AGESA.

Lyndeno · August 21, 2023, 6:07pm

dmesg | grep microcode

[    1.136950] microcode: CPU1: patch_level=0x0a20120a
[    1.136950] microcode: CPU2: patch_level=0x0a20120a
[    1.136950] microcode: CPU3: patch_level=0x0a20120a
[    1.136950] microcode: CPU4: patch_level=0x0a20120a
[    1.136951] microcode: CPU5: patch_level=0x0a20120a
[    1.136951] microcode: CPU6: patch_level=0x0a20120a
[    1.136952] microcode: CPU7: patch_level=0x0a20120a
[    1.136953] microcode: CPU9: patch_level=0x0a20120a
[    1.136953] microcode: CPU8: patch_level=0x0a20120a
[    1.136954] microcode: CPU10: patch_level=0x0a20120a
[    1.136954] microcode: CPU11: patch_level=0x0a20120a
[    1.136955] microcode: CPU12: patch_level=0x0a20120a
[    1.136957] microcode: CPU13: patch_level=0x0a20120a
[    1.136957] microcode: CPU14: patch_level=0x0a20120a
[    1.136958] microcode: CPU15: patch_level=0x0a20120a
[    1.136959] microcode: CPU16: patch_level=0x0a20120a
[    1.136959] microcode: CPU17: patch_level=0x0a20120a
[    1.136960] microcode: CPU18: patch_level=0x0a20120a
[    1.136960] microcode: CPU20: patch_level=0x0a20120a
[    1.136960] microcode: CPU19: patch_level=0x0a20120a
[    1.136962] microcode: CPU22: patch_level=0x0a20120a
[    1.136962] microcode: CPU21: patch_level=0x0a20120a
[    1.136963] microcode: CPU23: patch_level=0x0a20120a
[    1.136964] microcode: CPU24: patch_level=0x0a20120a
[    1.136964] microcode: CPU25: patch_level=0x0a20120a
[    1.136965] microcode: CPU26: patch_level=0x0a20120a
[    1.136966] microcode: CPU27: patch_level=0x0a20120a
[    1.136966] microcode: CPU0: patch_level=0x0a20120a
[    1.136966] microcode: CPU28: patch_level=0x0a20120a
[    1.136967] microcode: CPU29: patch_level=0x0a20120a
[    1.136967] microcode: CPU31: patch_level=0x0a20120a
[    1.136968] microcode: CPU30: patch_level=0x0a20120a
[    1.136998] microcode: Microcode Update Driver: v2.2.

grep 'model\|microcode' /proc/cpuinfo

model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a

BIOS is version 4802 (2023-07-14) for Asus Prime X570-PRO with change notes:

Update AGESA version to ComboV2PI 1.2.0.A

Mitigate the AMD potential security vulnerabilities for AMD Athlon™ processors and Ryzen™ processors

Improve system stability

NixOS version is NixOS 23.11.20230728.2a9d660 (tapir). I believe this is before the microcode updates were added? My interpretation of the command output is that perhaps the BIOS update applied the mitigations to my system?

TLATER · August 22, 2023, 1:02am

That’s two years old, your microcode is just as outdated as mine.

Don’t know why the warning doesn’t show. Your kernel might be different from linuxPackages_latest? I switched to xanmod recently which doesn’t apparently.

Hmm, perhaps? This webpage seems to suggest so indeed, and apparently my motherboard firmware was released in July so falls outside of the timeline: https://www.amd.com/en/resources/product-security/bulletin/amd-sb-7005.html

However, my microcode has not been updated since 2021, and I’ve kept up with BIOS updatea. Has there simply been no consumer-facing microcode update since? There clearly are other microcode numbers floating around, so I doubt it…

dawidd6 · August 22, 2023, 8:54am

Related: https://www.reddit.com/r/linux/comments/15xvpfg/updating_your_amd_microcode_in_linux/

Lyndeno · August 22, 2023, 4:10pm

That would be why I don’t get the warning then. I am using the ZFS latestcompatiblelinux kernel, which is on an older version of Linux. And my NixOS revision would be before that warning would have been backported I believe.

TLATER · August 22, 2023, 5:47pm

If that’s true, well damn. Sucks to have an AMD processor. According to the documentation of the third party collection repo the motherboard vendors have some discretion as to whether to push the updates, in theory to ensure stability, but I’d not be surprised if Gigabyte just don’t bother.

I’ll wait until September and then see if I can contact support of either company to confirm this, and find out how AMD actually intends for users to upgrade their microcode. If it’s by BIOS update, and Gigabyte refuses to actually do so, guess I’ll add that unofficial workaround to nixos-hardware or something?

bwolf · August 22, 2023, 8:55pm

I checked the BIOS of my mainboard and found that the latest version from 2023-08-04 states “Update AGESA to ComboAM5 1.0.0.7b” which is of course older than the version AMD plans to push to OEMs (ComboAM5 1.0.8.0 (Target August 2023)). I wonder if AMD already published such a version. Personally I guess not

dawidd6 · August 26, 2023, 9:58pm

I think you’ve mistaken threads.

Lyndeno · October 19, 2023, 11:43pm

Any chance this improves our situation?

Lyndeno · October 20, 2023, 12:03am

Checked the patch, does not look like it. Not for 5950X anyway

nyanbinary · March 29, 2024, 6:33pm

Same issue with my Framework 13 AMD.

TLATER · March 29, 2024, 8:59pm

I’ve confirmed since then that the updates are tied to BIOS updates. Check their support to see if they’ve published an updated BIOS since.

It’s a bit unfortunate since we’re effectively stuck with 0-days until hardware manufacturers manage to push updates months down the line, but that’s AMD’s update model.

fwupd also works on a bunch of vendors’ laptops these days, I’d imagine framework publish updates with it - who knows if that includes BIOS though.