AMD microcode updates not applying?

TLATER · August 19, 2023, 1:12pm

This is somewhat of a necro of AMD Microcode updates not working?, but that thread seems to have died.

In light of the recent AMD speculative execution bugs (yes, more of them), my dmesg has started saying:

tlater ~ $ dmesg | grep microcode
[    0.407693] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    0.407694] Speculative Return Stack Overflow: Mitigation: safe RET, no microcode

Since I have hardware.cpu.amd.updateMicrocode set, I figured this would long have been fixed, so I guessed I needed a BIOS update or such, but after applying it that hasn’t fixed the problem.

I checked my cpuinfo:

tlater ~ $ grep 'model\|microcode' /proc/cpuinfo
model           : 33
model name      : AMD Ryzen 5 5600X 6-Core Processor
microcode       : 0xa201025

The latest appears to be 0xaa00212.

This is a bit concerning, lots of NixOS systems will be vulnerable to that speculative execution bug, despite users believing they are mitigating it. Does anyone have any idea what’s going wrong?

lelgenio · August 19, 2023, 4:31pm

Not sure how much help it is, but here are my outputs:

Cpu: ryzen 5 5600
NixOs: release 23.05
Linux: 6.1.45
hardware.cpu.amd.updateMicrocode: true

dmesg | grep microcode

[    0.235067] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    0.235068] Speculative Return Stack Overflow: Mitigation: safe RET, no microcode
[    0.584402] microcode: CPU0: patch_level=0x0a20120a
[    0.584408] microcode: CPU1: patch_level=0x0a20120a
[    0.584413] microcode: CPU2: patch_level=0x0a20120a
[    0.584419] microcode: CPU3: patch_level=0x0a20120a
[    0.584425] microcode: CPU4: patch_level=0x0a20120a
[    0.584431] microcode: CPU5: patch_level=0x0a20120a
[    0.584435] microcode: CPU6: patch_level=0x0a20120a
[    0.584440] microcode: CPU7: patch_level=0x0a20120a
[    0.584444] microcode: CPU8: patch_level=0x0a20120a
[    0.584449] microcode: CPU9: patch_level=0x0a20120a
[    0.584455] microcode: CPU10: patch_level=0x0a20120a
[    0.584460] microcode: CPU11: patch_level=0x0a20120a
[    0.584463] microcode: Microcode Update Driver: v2.2.
[21719.450738] microcode: CPU1: patch_level=0x0a20120a
[21719.454234] microcode: CPU2: patch_level=0x0a20120a
[21719.457541] microcode: CPU3: patch_level=0x0a20120a
[21719.460664] microcode: CPU4: patch_level=0x0a20120a
[21719.463836] microcode: CPU5: patch_level=0x0a20120a
[21719.467439] microcode: CPU6: patch_level=0x0a20120a
[21719.470866] microcode: CPU7: patch_level=0x0a20120a
[21719.474639] microcode: CPU8: patch_level=0x0a20120a
[21719.478174] microcode: CPU9: patch_level=0x0a20120a
[21719.482105] microcode: CPU10: patch_level=0x0a20120a
[21719.485973] microcode: CPU11: patch_level=0x0a20120a

grep 'model\|microcode' /proc/cpuinfo

model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 5 5600 6-Core Processor
microcode	: 0xa20120a

bwolf · August 20, 2023, 5:20pm

I’m also puzzling about this. I’m on 23.05 and updated it now without any changes since 1 or 2 weeks.

$ nix repl
:lf .#
builtins.parseDrvName inputs.nixpkgs.legacyPackages.x86_64-linux.microcodeAmd.name
{ name = "amd-ucode"; version = "20230809"; }

CPU info:

$ grep 'model\|microcode' /proc/cpuinfo | head -3
model		: 97
model name	: AMD Ryzen 9 7900 12-Core Processor
microcode	: 0xa601203

dmesg output:

$ dmesg | grep microcode
[    0.067011] Speculative Return Stack Overflow: IBPB-extending microcode not applied!
[    0.067012] Speculative Return Stack Overflow: Mitigation: safe RET, no microcode
[    0.342131] microcode: CPU0: patch_level=0x0a601203
...
[    0.342234] microcode: CPU23: patch_level=0x0a601203
[    0.342237] microcode: Microcode Update Driver: v2.2.

So it seems the firmware is too old on my system (20230809). Maybe nixpkgs for 23.05 it not recent enough?

TLATER · August 20, 2023, 8:24pm

That’s not it, I went trawling through nixpkgs, latest linux-firmware is backported since ~March this year. We’re currently on August 9 as the version number states, but at least my microcode version is 2 years (!) old.

I’ve since learned that linux-firmware contains different microcodes for a variety of amd processor families. The nixpkgs package appears to bundle them all, but maybe one is missing? My next attempt was going to be grabbing a debian/arch/gentoo package and forcing it into initrd manually to see if it’s a packaging error.

mabl · August 21, 2023, 5:31pm

I found this statement on the Phoronix forum that seems to indicate that consumer AMD CPUs only get updated by the AGESA.

Lyndeno · August 21, 2023, 6:07pm

dmesg | grep microcode

[    1.136950] microcode: CPU1: patch_level=0x0a20120a
[    1.136950] microcode: CPU2: patch_level=0x0a20120a
[    1.136950] microcode: CPU3: patch_level=0x0a20120a
[    1.136950] microcode: CPU4: patch_level=0x0a20120a
[    1.136951] microcode: CPU5: patch_level=0x0a20120a
[    1.136951] microcode: CPU6: patch_level=0x0a20120a
[    1.136952] microcode: CPU7: patch_level=0x0a20120a
[    1.136953] microcode: CPU9: patch_level=0x0a20120a
[    1.136953] microcode: CPU8: patch_level=0x0a20120a
[    1.136954] microcode: CPU10: patch_level=0x0a20120a
[    1.136954] microcode: CPU11: patch_level=0x0a20120a
[    1.136955] microcode: CPU12: patch_level=0x0a20120a
[    1.136957] microcode: CPU13: patch_level=0x0a20120a
[    1.136957] microcode: CPU14: patch_level=0x0a20120a
[    1.136958] microcode: CPU15: patch_level=0x0a20120a
[    1.136959] microcode: CPU16: patch_level=0x0a20120a
[    1.136959] microcode: CPU17: patch_level=0x0a20120a
[    1.136960] microcode: CPU18: patch_level=0x0a20120a
[    1.136960] microcode: CPU20: patch_level=0x0a20120a
[    1.136960] microcode: CPU19: patch_level=0x0a20120a
[    1.136962] microcode: CPU22: patch_level=0x0a20120a
[    1.136962] microcode: CPU21: patch_level=0x0a20120a
[    1.136963] microcode: CPU23: patch_level=0x0a20120a
[    1.136964] microcode: CPU24: patch_level=0x0a20120a
[    1.136964] microcode: CPU25: patch_level=0x0a20120a
[    1.136965] microcode: CPU26: patch_level=0x0a20120a
[    1.136966] microcode: CPU27: patch_level=0x0a20120a
[    1.136966] microcode: CPU0: patch_level=0x0a20120a
[    1.136966] microcode: CPU28: patch_level=0x0a20120a
[    1.136967] microcode: CPU29: patch_level=0x0a20120a
[    1.136967] microcode: CPU31: patch_level=0x0a20120a
[    1.136968] microcode: CPU30: patch_level=0x0a20120a
[    1.136998] microcode: Microcode Update Driver: v2.2.

grep 'model\|microcode' /proc/cpuinfo

model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a
model		: 33
model name	: AMD Ryzen 9 5950X 16-Core Processor
microcode	: 0xa20120a

BIOS is version 4802 (2023-07-14) for Asus Prime X570-PRO with change notes:

Update AGESA version to ComboV2PI 1.2.0.A

Mitigate the AMD potential security vulnerabilities for AMD Athlon™ processors and Ryzen™ processors

Improve system stability

NixOS version is NixOS 23.11.20230728.2a9d660 (tapir). I believe this is before the microcode updates were added? My interpretation of the command output is that perhaps the BIOS update applied the mitigations to my system?

TLATER · August 22, 2023, 1:02am

That’s two years old, your microcode is just as outdated as mine.

Don’t know why the warning doesn’t show. Your kernel might be different from linuxPackages_latest? I switched to xanmod recently which doesn’t apparently.

Hmm, perhaps? This webpage seems to suggest so indeed, and apparently my motherboard firmware was released in July so falls outside of the timeline: https://www.amd.com/en/resources/product-security/bulletin/amd-sb-7005.html

However, my microcode has not been updated since 2021, and I’ve kept up with BIOS updatea. Has there simply been no consumer-facing microcode update since? There clearly are other microcode numbers floating around, so I doubt it…

dawidd6 · August 22, 2023, 8:54am

Related: https://www.reddit.com/r/linux/comments/15xvpfg/updating_your_amd_microcode_in_linux/

Lyndeno · August 22, 2023, 4:10pm

That would be why I don’t get the warning then. I am using the ZFS latestcompatiblelinux kernel, which is on an older version of Linux. And my NixOS revision would be before that warning would have been backported I believe.

TLATER · August 22, 2023, 5:47pm

If that’s true, well damn. Sucks to have an AMD processor. According to the documentation of the third party collection repo the motherboard vendors have some discretion as to whether to push the updates, in theory to ensure stability, but I’d not be surprised if Gigabyte just don’t bother.

I’ll wait until September and then see if I can contact support of either company to confirm this, and find out how AMD actually intends for users to upgrade their microcode. If it’s by BIOS update, and Gigabyte refuses to actually do so, guess I’ll add that unofficial workaround to nixos-hardware or something?

bwolf · August 22, 2023, 8:55pm

I checked the BIOS of my mainboard and found that the latest version from 2023-08-04 states “Update AGESA to ComboAM5 1.0.0.7b” which is of course older than the version AMD plans to push to OEMs (ComboAM5 1.0.8.0 (Target August 2023)). I wonder if AMD already published such a version. Personally I guess not

dawidd6 · August 26, 2023, 9:58pm

I think you’ve mistaken threads.

Lyndeno · October 19, 2023, 11:43pm

Any chance this improves our situation?

Lyndeno · October 20, 2023, 12:03am

Checked the patch, does not look like it. Not for 5950X anyway

nyanbinary · March 29, 2024, 6:33pm

Same issue with my Framework 13 AMD.

TLATER · March 29, 2024, 8:59pm

I’ve confirmed since then that the updates are tied to BIOS updates. Check their support to see if they’ve published an updated BIOS since.

It’s a bit unfortunate since we’re effectively stuck with 0-days until hardware manufacturers manage to push updates months down the line, but that’s AMD’s update model.

fwupd also works on a bunch of vendors’ laptops these days, I’d imagine framework publish updates with it - who knows if that includes BIOS though.

Etho · November 16, 2024, 1:42pm

I’m late to the party, but for those who might come across this while searching, I’ve created a flake to fetch and apply AMD microcode updates for unsupported CPUs: ucodenix.

Atemu · November 17, 2024, 12:07am

Would you like to upstream that to Nixpkgs/NixOS?

ElvishJerricco · November 17, 2024, 12:24am

Two problems with bringing it upstream.

It has an option to automatically identify the CPU at build time. It’s not on by default, but it’s an impurity that I don’t think we can realistically accept upstream at all, even as an option. So that would have to be removed.
It’s based on this repo, which appears to be a crowdsourced collection of microcodes. This isn’t exactly something I would be encouraging typical users to use, especially given this warning in the repo:

It is generally advised to request and/or wait for your OEM/OS to release newer fixes. Latest is not always better or tested. Manufacturers and OS mainteners usually have some insider/confidential info from microcode vendors on what got changed/fixed at newer microcode releases so if they ship older microcodes, it could be that newer versions have not been thoroughly tested, have been retracted/downgraded by the microcode vendor or not contain anything important enough to warrant an update. The microcodes here are gathered and provided with the sole purpose of helping people who are out of other viable solutions. Thus, they can be extremely helpful to those who have major problems with their systems for which their manufacturer refuses to assist due to indifference and/or system age.

Great to have the option, but doesn’t exactly fit with upstream NixOS IMO.

Etho · November 17, 2024, 2:19am

I agree with the first point, but the current method serves as a temporary workaround until solutions enabling automatic identification of the CPU model ID without compromising build reproducibility are implemented.
Regarding the second point, the disclaimer in the repository is understandable. However, while accounting for the risk of regressions, I believe it’s more dangerous to rely on a microcode that is several years old with unpatched vulnerabilities or to update your BIOS just to obtain a recent microcode than to retrieve an update from that repository. These updates come from official sources and are tested for integrity.
That said, I’m perfectly fine with ucodenix being a flake for advanced users.