PSA: keep NVMe storage devices from getting too hot

Hey,

maybe a more generic forum would be better for this but I don’t really know of
one that I’d like to join, so I am posting this here. Hope that’s alright with
you all. :sweat_smile:

Today I learned how to keep the temperature of my NVMe SSD in my laptop in check
without having to do hardware modifications for extra cooling. I didn’t find any
good post that would practically explain this for Linux, so I figured it might
be useful for someone. :slightly_smiling_face:

There is a good post about this here that explains the temperature values
conceptually: Technology Power Features - NVM Express

The tl;dr is that NVMe devices have configuration options that can be queried
using nvme get-feature --human-readable /dev/nvmeX. One of these standard
options is one to set maximum temperatures at which point you really would
rather have the SSD throttle down than keep getting hotter. Not every SSD has a
reasonable default for this. If your SSD is not cooled very well, like e.g. in a
laptop, you may want to set your own values for this.

Of course using this throttling means that performance will degrade if the
limits are hit. Personally I prefer this versus potential premature hardware
failure and a 90+ degree hot SSD very few centimeters away from a lithium ion
battery. Because you totally can hit temperatures like that when you really tax
the SSD, for example when doing a BTRFS scrub.

Here is a short example script that persistently sets the throttle
temperatures to 50 and 65 degrees, feel free to adapt those values to your
preference of course:

#!/bin/sh
set -eux

nvme_dev="$1"

# the lower and upper thermal throttle management temperatures in degrees
# celsius. for explanation see:
# https://nvmexpress.org/resource/technology-power-features/
temp1=50
temp2=65

celsius_to_kelvin() {
    celsius="$1"
    echo $((celsius + 273))
}

# the value for the "Host Controlled Thermal Management" feature. mash together
# the two temperatures into one integer in hexadecimal representation.
hctm=$(printf '0x%04x%04x' "$(celsius_to_kelvin "$temp1")" "$(celsius_to_kelvin "$temp2")")

nvme set-feature "$nvme_dev" --feature-id=0x10 --value="$hctm" --save

With these settings, I now still have the SSD hit up to 70 degrees during a
BTRFS scrub but I think that’s more tolerable than 90 degrees and it does take a
couple minutes more because of the throttling but it’s not critical anyway.
Maybe I’ll tune those values a bit more down the line.

Alternatively, if you want this to be managed at run time and not persistently
somewhere in the specific storage controller’s firmware, you could write a udev
rule for NixOS somewhat like this:

ACTION=="add|change", SUBSYSTEM=="nvme", \
  ATTRS{vendor}=="XXX", ATTRS{device}=="YYY", \
  RUN+="${pkgs.nvme-cli}/bin/nvme set-feature $devnode --feature-id=0x10 --value=0x01430152"

Replace XXX and YYY with the device and vendor from lspci -nn for your
particular device that you want to keep cool. Also the hex --value= is what
you would get from the script example, again feel free to adapt to your needs.
The only real difference is that we’re not using the --save argument.

I personally use a Framework Laptop 11th gen with an WD SN850x. That SSD gets
very hot, and the reason is that it just straight up doesn’t set a default
throttle temperature and the highest power state consumes 9 watts. Not even
something like 105 degrees as a limit. It just sets this value to 0. :roll_eyes:

7 Likes

it sounds like framework need work on their nvme cooling subsystem…

make sure you tell them, because this sounds like a design issue… or if you have intergrated the thing yourself, then it’s going to need some sort of case fan and airflow.

Which might be a bit more difficult in a laptop.

If the nvme is stock, then framework really need to address airflow and cooling, even under load it should be able to dissipate enough heat to keep it toastie, but not melting.

Your not in one of the hot climates are you? where the sun comes out that makes it exceed it thermal cooling capabilities?

Being in the northern hemisphere we have the advantage of ‘being cold most of the year’ which works wonders all sorts of tech cooling ;-)/.

2 Likes

Checking up on the two SSDs in my desktop, one has a thermal limit, one does not. Go figure.

Samsung_SSD_970_EVO_500GB
# nvme get-feature /dev/nvme0n1 --feature-id=0x10
get-feature:0x10 (Host Controlled Thermal Management), Current value:00000000

Samsung_980_PRO_with_Heatsink_2TB
# nvme get-feature /dev/nvme1n1 --feature-id=0x10
get-feature:0x10 (Host Controlled Thermal Management), Current value:0x01630164
1 Like

Ah, good points that need some more clarification on my part!

Actually, it’s mostly fine on Framework’s end. I bought the SSD as an upgrade through retail. It’s 4 TB and was relatively cheap when I bought it. They do offer the exact same model as a part in this capacity, however.
One thing that I actually can imagine is that Framework may be allowed to set this feature at their production or have a custom firmware even that has values set as the default for these temperature controls. That would make a lot of sense to me but that’s just me speculating.

I think that the cooling solution is actually pretty standard for this form factor. The M.2 seem pretty separated from the CPU/iGPU that have active cooling through one fan. The SSD temperate does not seem affected during compile jobs, for example. Separate active cooling for the M.2 in a 13 inch design is unlikely to be feasible. Maybe a small heatsink could be possible that could be attached to the top or bottom case but that makes repair and access a bit more difficult, so idk.

In general, I would like to add that this is not a problem that’s specific to either Framework or WD. Ever since PCIe has opened up the IO bottleneck for storage, temperatures have been going up. You can find plenty of cases when doing a web search, a recent and quite severe example being this here: Failing A PCIe 5.0 NVMe SSD In Less Than 3 Minutes Without Extra Cooling - Phoronix

I mostly just wanted to share a neat and totally valid solution to this issue that’s already built-in to any NVMe device because it’s part of the standard but not totally obvious and somewhat obtuse to actually make use of. :smile:

Also yes, I am in northern hemisphere, the country that the actor in your pfp is from and it’s cold. :snowflake::grin:

1 Like

Interesting, I guess Samsung didn’t feel like it was necessary to have a limit on the 970. I think that one doesn’t get as hot yet since it’s only PCIe 3.0?

I see…

thanks for the clarification…

it maybe worth if framework do set certain parameters…and replicate that in your setup…

these fast large SSD get toastie don’t they!!!

I wonder if the linux kernel can also be tuned to reduce io operations in a thermal runaway scenario ?

That would be a cool dynamic feature… not sure that’s possible, i’ve never need to limit a drive before!

1 Like