Security advisory: OpenSSH CVE-2024-6387 “regreSSHion” – update your servers ASAP

Announcements Security
14

Thanks for your hard work. :heartpulse:

2 Likes
15

Then just set your openssh package manually to the new one?

services.openssh.package = (builtins.getFlake "github:nixos/nixpkgs/7f993cdf26ccef564eabf31fdb40d140821e12bc?narHash=sha256-pY0wosAgcr9W4vmGML0T3BVhQiGuKoozCbs2t+Je1zc=").legacyPackages.x86_64-linux.openssh

You could even override the derivation yourself like pkgs.openssh.override { patches = [ thePatch.patch ]; }

4 Likes
16

The fix is in stable (24.05) now \o/

3 Likes
17

now-ish it reached nixos-unstable. nixos-24.05 was ~11 hours ago. All three -small channels were covered long ago, and the three “nixpkgs*” channels have also updated.

6 Likes
18

Thank you! https://nixpk.gs/pr-tracker.html?pr=323753 seems to be dying.

1 Like
19

And we’re done. Phew! The fixed packages are now in all of the unstable, 24.05, and 23.11 channels. If you haven’t already updated, you should do so now (in fact, you might as well do it again anyway, just in case you’re not on the channel or revision you thought you were). You can and should remove any workarounds and mitigations when you update. If you’re on 23.11, please take the opportunity to upgrade to 24.05 as soon as possible; it was out of official support by the time all this happened, and the next security vulnerability probably won’t get a fix backported.

Thank you for all the kind words, but I didn’t make this happen alone – shout‐outs are also due to the others who helped get this done, including ari (no Discourse account I think), @qyliss, @vcunat, @hexa, @R-VdP, @ElvishJerricco, @leona, @tgerbet, @j-k, and of course the OpenSSH developers. (I’m probably forgetting people, too.)

Here’s a timeline for the curious:

  • T+0: OpenSSH 9.8p1 is announced with the first public disclosure of this bug.
  • T+25m: Upstream posts minimal patches for previous versions. The 9.8p1 release notes are posted in the security triage channel.
  • T+35m: #323753 is opened to bump unstable to 9.8p1.
  • T+50m: #323753 is merged.
  • T+56m: #323761 is opened to apply the minimal patches to 24.05.
  • T+1h2m: #323761 is merged.
  • T+1h10m: #323765 is opened to apply the minimal patches to 23.11.
  • T+1h19m: #323768 is opened to patch openssh_{hpn,gssapi} on unstable.
  • T+1h29m: #323765 is merged.
  • T+2h56m: #323768 is merged.
  • T+3h18m: The openssh bump reaches nixos-unstable-small and the first users start getting a fix through updates.
  • T+4h: The openssh patch reaches nixos-24.05-small. (It reaches nixos-23.11-small around the same time, but I don’t have the exact time.)
  • T+4h2m: This advisory is posted.
  • T+4h30m: Backports of #323768 to 24.05 and 23.11 are merged.
  • T+4h34m: #323796 is opened to fix SSH in initrd on unstable, which would have blocked the large channel bumps and potentially broken people’s setups. Stable versions are not affected due to the use of minimal patches rather than a full OpenSSH upgrade.
  • T+5h29m: #323796 is merged.
  • T+8h9m: The openssh bump reaches nixpkgs-unstable.
  • T+16h43m: The openssh and openssh_{hpn,gssapi} patches reach nixos-24.05. The first users on the large channels start to get fixes.
  • T+1d2h56m: The openssh bump, openssh_{hpn,gssapi} patches, and initrd SSH fix reach nixos-unstable.
  • T+1d6h23m: The openssh_{hpn,gssapi} patches reach nixos-23.11-small.
  • T+1d8h4m: The openssh and openssh_{hpn,gssapi} patches reach nixos-23.11.
  • T+1d11h25m: The openssh_{hpn,gssapi} patches and the initrd SSH fix reach nixpkgs-unstable. (Hopefully you’re not using nixpkgs-unstable (as opposed to nixos-unstable) on a NixOS machine anyway, though.)

(I’m not sure when the openssh_{hpn,gssapi} fixes reached nixos-{unstable,24.05}-small, or when the initrd SSH fix reached nixos-unstable-small, but this is probably already more exhaustive than anyone cares about.)

This is my first time doing this kind of thing; lessons learned:

  • Our ability to get security updates that don’t cause mass rebuilds into the small channels is pretty good! A four‐hour timespan from the issue being publicly announced to shipping binaries to users isn’t the fastest release a distro managed, but I believe we beat some distros that were in on the embargo. If you’re running a server and are worried about getting security fixes quickly, then using the small channel (and potentially making your system builds include the NixOS tests that aren’t part of the small channel but seem relevant to your server) is a good idea. Unfortunately this doesn’t do as much for fixes that have to go via staging.

  • Large channels are slow to update and it’s hard to fix this. It’s probably good to include information about how users on large channels can pull in security updates on the small ones. On the other hand, the large channel tests are useful; they uncovered 9.8p1 breaking SSH in initrd on unstable. Someone who was relying on that to be able to boot their remote server might have had a nasty surprise if they tried to get ahead of the channel bump.

  • It’s important for people to have an easy way to check whether their system is vulnerable. I don’t know whether this just means having appropriate Nix‐fu commands ready or whether there’s more that should be done as part of the fix PRs themselves to surface it better.

  • Monitoring the oss-security list and posting important stuff to the triage channel is probably a reasonably high‐impact practice. (Though it’s possible or even likely that people are already doing this; 25 minutes is a very quick turnaround from a mailing list message to it being raised on Matrix anyway.)

  • Hydra requires more active management than you’d think and the people who do it are thankless heroes.

  • Unix signal delivery semantics are a mistake that we continue to pay the price for decades later.

  • It’s good to eat breakfast when you can.

50 Likes
20

I fear that the 24.05 branch is still at 9.7p1

https://search.nixos.org/packages?channel=24.05&from=0&size=50&sort=relevance&type=packages&query=openssh

(unstable is indeed 9.8p1)

21

The patches were backported, so the version number didn’t change. See slightly further up: Security advisory: OpenSSH CVE-2024-6387 “regreSSHion” – update your servers ASAP - #13 by emily

Hence the:

2 Likes
22

RIght, sorry, I only read the last status message.

24

Here’s some nix-fu for checking whether you have the backported patch:

nix derivation show $(which ssh) | jq -r '.[].inputSrcs[]' | grep CVE-2024
5 Likes
25

I believe you need $(readlink -f $(which ssh)), otherwise it’ll be looking at the derivation that produced your /run/current-system/sw/bin, rather than the OpenSSH one. (So don’t panic if this seems to indicate you’re still vulnerable.)

8 Likes
26

Can we determine if a nixos machine is patched with a tool like vulnix?

27

Yes, vulnix -R $(readlink -f $(which ssh)) will show you if there is an active advisory on it or not. Under the hood, it does something similar to what the nix derivation show/jq/grep proposed a few posts ago does.

3 Likes
28

It’s worth noting that all these snippets rely on the contents of $PATH, which may or may not match up with whatever is running in the systemd service, especially if user channels or other heavier customization is involved. It’ll work most of the time, of course, but it might not, which is a bit scary. I can’t think of a simple, foolproof and generic check either, though.

Maybe we could make this easier upstream? What do other distros do? “Sadly” this CVE takes too long to exploit to write a simple distro-independent vulnerability checker for it.

3 Likes
29

Would vulnix -S help with that? Not sure if we can filter to only get openssh result? Otherwise, I guess it would take a while to scan everything.

30

Not 100% foolproof and generic, but I was doing this to see which sshd are running at the moment:

for p in $(pidof sshd); do readlink /proc/$p/exe; done | sort | uniq -c

Note in particular that switching the systemd service won’t immediately replace all of them. Thankfully, as for servers you’re probably doing the update over ssh.

1 Like
31

This would have been appreciated, as none of the snippets in this thread works for me (error: '/nix/store/ik1hrivpiw5lkmarlzmpk8armfgpxwcf-openssh-9.7p1' does not have a known deriver).

A quick systemctl status sshd check would be ideal, but it doesn’t scale with either CVEs or time: the name tweak must be removed some time (when the patch doesn’t have to be applied in nixpgks?)

32

I take it vulnix -R $(readlink -f $(which ssh)) doesn’t work for you either? I’m not sure what circumstances would cause this.

33

vulnix fails with the same error, yes.

34

Thanks. I’m curious if you know any details of your setup that might be unusual – Nix settings, Nixpkgs pins, substituters, overlays, …?