AMD microcode updates not applying?

[TLATER]

This is somewhat of a necro of AMD Microcode updates not working?, but that thread seems to have died.

In light of the recent AMD speculative execution bugs (yes, more of them), my dmesg has started saying:

tlater ~ $ dmesg | grep microcode
[    0.407693] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    0.407694] Speculative Return Stack Overflow: Mitigation: safe RET, no microcode

Since I have hardware.cpu.amd.updateMicrocode set, I figured this would long have been fixed, so I guessed I needed a BIOS update or such, but after applying it that hasn’t fixed the problem.

I checked my cpuinfo:

tlater ~ $ grep 'model\|microcode' /proc/cpuinfo
model           : 33
model name      : AMD Ryzen 5 5600X 6-Core Processor
microcode       : 0xa201025

The latest appears to be 0xaa00212.

This is a bit concerning, lots of NixOS systems will be vulnerable to that speculative execution bug, despite users believing they are mitigating it. Does anyone have any idea what’s going wrong?

[lelgenio]

Not sure how much help it is, but here are my outputs:

Cpu: ryzen 5 5600
NixOs: release 23.05
Linux: 6.1.45
hardware.cpu.amd.updateMicrocode: true

dmesg | grep microcode

[    0.235067] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    0.235068] Speculative Return Stack Overflow: Mitigation: safe RET, no microcode
[    0.584402] microcode: CPU0: patch_level=0x0a20120a
[    0.584408] microcode: CPU1: patch_level=0x0a20120a
[    0.584413] microcode: CPU2: patch_level=0x0a20120a
[    0.584419] microcode: CPU3: patch_level=0x0a20120a
[    0.584425] microcode: CPU4: patch_level=0x0a20120a
[    0.584431] microcode: CPU5: patch_level=0x0a20120a
[    0.584435] microcode: CPU6: patch_level=0x0a20120a
[    0.584440] microcode: CPU7: patch_level=0x0a20120a
[    0.584444] microcode: CPU8: patch_level=0x0a20120a
[    0.584449] microcode: CPU9: patch_level=0x0a20120a
[    0.584455] microcode: CPU10: patch_level=0x0a20120a
[    0.584460] microcode: CPU11: patch_level=0x0a20120a
[    0.584463] microcode: Microcode Update Driver: v2.2.
[21719.450738] microcode: CPU1: patch_level=0x0a20120a
[21719.454234] microcode: CPU2: patch_level=0x0a20120a
[21719.457541] microcode: CPU3: patch_level=0x0a20120a
[21719.460664] microcode: CPU4: patch_level=0x0a20120a
[21719.463836] microcode: CPU5: patch_level=0x0a20120a
[21719.467439] microcode: CPU6: patch_level=0x0a20120a
[21719.470866] microcode: CPU7: patch_level=0x0a20120a
[21719.474639] microcode: CPU8: patch_level=0x0a20120a
[21719.478174] microcode: CPU9: patch_level=0x0a20120a
[21719.482105] microcode: CPU10: patch_level=0x0a20120a
[21719.485973] microcode: CPU11: patch_level=0x0a20120a

grep 'model\|microcode' /proc/cpuinfo

model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a

[bwolf]

I’m also puzzling about this. I’m on 23.05 and updated it now without any changes since 1 or 2 weeks.

$ nix repl
:lf .#
builtins.parseDrvName inputs.nixpkgs.legacyPackages.x86_64-linux.microcodeAmd.name
{ name = "amd-ucode"; version = "20230809"; }

CPU info:

$ grep 'model\|microcode' /proc/cpuinfo | head -3
model		: 97
model name	: AMD Ryzen 9 7900 12-Core Processor
microcode	: 0xa601203

dmesg output:

$ dmesg | grep microcode
[    0.067011] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    0.067012] Speculative Return Stack Overflow: Mitigation: safe RET, no microcode
[    0.342131] microcode: CPU0: patch_level=0x0a601203
...
[    0.342234] microcode: CPU23: patch_level=0x0a601203
[    0.342237] microcode: Microcode Update Driver: v2.2.

So it seems the firmware is too old on my system (20230809). Maybe nixpkgs for 23.05 it not recent enough?

[TLATER]

@bwolf

it seems the firmware is too old on my system (20230809). Maybe nixpkgs for 23.05 it not recent enough?

That’s not it, I went trawling through nixpkgs, latest linux-firmware is backported since ~March this year. We’re currently on August 9 as the version number states, but at least my microcode version is 2 years (!) old.

I’ve since learned that linux-firmware contains different microcodes for a variety of amd processor families. The nixpkgs package appears to bundle them all, but maybe one is missing? My next attempt was going to be grabbing a debian/arch/gentoo package and forcing it into initrd manually to see if it’s a packaging error.

[mabl]

I found this statement on the Phoronix forum that seems to indicate that consumer AMD CPUs only get updated by the AGESA.

[Lyndeno]

dmesg | grep microcode

[    1.136950] microcode: CPU1: patch_level=0x0a20120a
[    1.136950] microcode: CPU2: patch_level=0x0a20120a
[    1.136950] microcode: CPU3: patch_level=0x0a20120a
[    1.136950] microcode: CPU4: patch_level=0x0a20120a
[    1.136951] microcode: CPU5: patch_level=0x0a20120a
[    1.136951] microcode: CPU6: patch_level=0x0a20120a
[    1.136952] microcode: CPU7: patch_level=0x0a20120a
[    1.136953] microcode: CPU9: patch_level=0x0a20120a
[    1.136953] microcode: CPU8: patch_level=0x0a20120a
[    1.136954] microcode: CPU10: patch_level=0x0a20120a
[    1.136954] microcode: CPU11: patch_level=0x0a20120a
[    1.136955] microcode: CPU12: patch_level=0x0a20120a
[    1.136957] microcode: CPU13: patch_level=0x0a20120a
[    1.136957] microcode: CPU14: patch_level=0x0a20120a
[    1.136958] microcode: CPU15: patch_level=0x0a20120a
[    1.136959] microcode: CPU16: patch_level=0x0a20120a
[    1.136959] microcode: CPU17: patch_level=0x0a20120a
[    1.136960] microcode: CPU18: patch_level=0x0a20120a
[    1.136960] microcode: CPU20: patch_level=0x0a20120a
[    1.136960] microcode: CPU19: patch_level=0x0a20120a
[    1.136962] microcode: CPU22: patch_level=0x0a20120a
[    1.136962] microcode: CPU21: patch_level=0x0a20120a
[    1.136963] microcode: CPU23: patch_level=0x0a20120a
[    1.136964] microcode: CPU24: patch_level=0x0a20120a
[    1.136964] microcode: CPU25: patch_level=0x0a20120a
[    1.136965] microcode: CPU26: patch_level=0x0a20120a
[    1.136966] microcode: CPU27: patch_level=0x0a20120a
[    1.136966] microcode: CPU0: patch_level=0x0a20120a
[    1.136966] microcode: CPU28: patch_level=0x0a20120a
[    1.136967] microcode: CPU29: patch_level=0x0a20120a
[    1.136967] microcode: CPU31: patch_level=0x0a20120a
[    1.136968] microcode: CPU30: patch_level=0x0a20120a
[    1.136998] microcode: Microcode Update Driver: v2.2.

grep 'model\|microcode' /proc/cpuinfo

model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a

BIOS is version 4802 (2023-07-14) for Asus Prime X570-PRO with change notes:

1. Update AGESA version to ComboV2PI 1.2.0.A
2. Mitigate the AMD potential security vulnerabilities for AMD Athlon™ processors and Ryzen™ processors
3. Improve system stability

NixOS version is NixOS 23.11.20230728.2a9d660 (tapir). I believe this is before the microcode updates were added? My interpretation of the command output is that perhaps the BIOS update applied the mitigations to my system?

[TLATER]

@Lyndeno

0xa20120a

That’s two years old, your microcode is just as outdated as mine.

Don’t know why the warning doesn’t show. Your kernel might be different from linuxPackages_latest? I switched to xanmod recently which doesn’t apparently.

@mabl

I found this statement on the Phoronix forum that seems to indicate that consumer AMD CPUs only get updated by the AGESA.

Hmm, perhaps? This webpage seems to suggest so indeed, and apparently my motherboard firmware was released in July so falls outside of the timeline: https://www.amd.com/en/resources/product-security/bulletin/amd-sb-7005.html

However, my microcode has not been updated since 2021, and I’ve kept up with BIOS updatea. Has there simply been no consumer-facing microcode update since? There clearly are other microcode numbers floating around, so I doubt it…

[dawidd6]

Related: Reddit - Dive into anything

[Lyndeno]

@TLATER

That’s two years old, your microcode is just as outdated as mine.

Don’t know why the warning doesn’t show. Your kernel might be different from linuxPackages_latest? I switched to xanmod recently which doesn’t apparently.

That would be why I don’t get the warning then. I am using the ZFS latestcompatiblelinux kernel, which is on an older version of Linux. And my NixOS revision would be before that warning would have been backported I believe.

[TLATER]

If that’s true, well damn. Sucks to have an AMD processor. According to the documentation of the third party collection repo the motherboard vendors have some discretion as to whether to push the updates, in theory to ensure stability, but I’d not be surprised if Gigabyte just don’t bother.

I’ll wait until September and then see if I can contact support of either company to confirm this, and find out how AMD actually intends for users to upgrade their microcode. If it’s by BIOS update, and Gigabyte refuses to actually do so, guess I’ll add that unofficial workaround to nixos-hardware or something?

[bwolf]

I checked the BIOS of my mainboard and found that the latest version from 2023-08-04 states “Update AGESA to ComboAM5 1.0.0.7b” which is of course older than the version AMD plans to push to OEMs (ComboAM5 1.0.8.0 (Target August 2023)). I wonder if AMD already published such a version. Personally I guess not

[dawidd6]

I think you’ve mistaken threads.

[Lyndeno]

AMD Family 19h CPU Microcode Updated (2023-10-19 Rev)

Any chance this improves our situation?

[Lyndeno]

Checked the patch, does not look like it. Not for 5950X anyway