Future design of hardening flags

Development Nixpkgs Architecture
1

Up until now, hardening flags (as understood by hardeningEnable, hardeningDisable et al.) have been fairly architecture/processor neutral.

On the horizon are a number of new compiler hardening options that are backed by new hardware features being added by vendors, and they vary quite significantly from architecture to architecture. We’re going to need to decide how fine-grained we want to make our hardening flags for these.

Let’s take as an example a new Control Flow Integrity (CFI) related feature present in recent Intel (and now AMD) processors - “Shadow Stack” (Shadow stack - Wikipedia, https://lwn.net/Articles/885220/). This is a feature that helps secure the “backward edge” of a control-flow-graph (i.e. function returns) from being hijacked. Once kernel support is present (very recent kernels only) it’s just a matter of enabling one compiler flag to start using this on supporting processors.

Aarch64 doesn’t (yet?) support shadow stack, but recent processors do have another means of implementing “backward edge” CFI using a similarly new-ish “pointer authentication” (PAC) feature. This is supported by both clang and gcc through the -mbranch-protection flag.

(Note this is not a discussion about support for older processors lacking support for these features - these features are generally designed in a backwards-compatible way using e.g. instructions that are interpreted as NOPs on unsupporting processors)

I can see three ways we could expose the x86_64 “shadow stack” feature:

  • A “hardware CFI backward edge” hardening flag (hwcfibwedge? …whether we should camel-case or snake-case our flag names is a separate debate…). This is the most abstracted option and denotes the intent to use hardware features to protect return addresses. On x86_64 this would mean use of shadow stack, on aarch64 this may mean use of return-pointer-authentication.
    • Pros:
      • Gathering information on packages where this broadly causes problems (and should be disabled) is aggregated across architectures. Minority architectures benefit from problem discovery on major architectures.
      • It may give us some leeway on exactly what flags are used to implement this, and we might be able to make different decisions as new features become available & widespread.
      • Apparent simplicity to users.
    • Cons:
      • The accuracy of this “problem discovery” may not be great if different mechanisms are being used behind the scenes.
  • A shadowstack hardening flag that corresponds to an actual shadow stack feature. Initially only having an effect on x86_64 but as other architectures gain support for it they can have support added.
    • Pros:
      • Problems are more likely to be similar across platforms
      • Packages that have problems with shadow stack implementations could potentially choose a different mechanism to provide this protection (e.g. return-pointer-authentication - exposed as retptrauth perhaps?)
    • Cons:
      • Opens the gate to confusing & complex logic around features used to achieve the same thing conflicting with each other.
  • A mshstk hardening flag. This is the literal name of the architecture-specific flag used by gcc & clang. This would directly control use of -mshstk on x86_64 and have no effect on other architectures. If another architecture gains shadow stack support, it gets its own flag.
    • Pros:
      • Accurate, fine grained control of flag usage. Enabling or disabling this flag won’t have any effect on any other architecture where the implementation details might be slightly different.
    • Cons:
      • No shared “problem discovery” - the work finding packages where the flag needs to be disabled must be repeated for each architecture. Minority architectures suffer.
      • Similar conflicting-options issues. I guess use of this flag implies a more manual/explicit approach to setting flags and users would take more of that burden on themselves?
      • Flag proliferation.
      • Once the user’s wanting this level of control, is the “hardening flags” system really serving its purpose? User could just as well use NIX_CFLAGS_COMPILE etc.

Keen to hear peoples thoughts on this. As I’ve written this I’ve changed my preferred option a couple of times, but am currently leant most heavily towards the first option.

7 Likes
2

Agreed

The accuracy of this “problem discovery” may not be great if different mechanisms are being used behind the scenes.

How likely do you consider this?
Like, the command lines of the compiler and linker are usually printed, right? So if an error can be attributed to a faulty CFI support, this shouldn’t be too hard to spot, correct?
If that’s only observable at runtime, it might be a little more complicated, but if that only affects a small subset of all our packages, then don’t think it’s such a big deal (at least if we roll this out right after branchoff, so we have ~half a year to stabilize unstable).

Opens the gate to confusing & complex logic around features used to achieve the same thing conflicting with each other.

To me this sounds like something that may easily happen given that ARM may also get an actual shadow stack (if I understand you correctly).

All in all, if I’m not underestimating the con for Option 1, I’d be in favor of that as well.

3

I think it’s likely to happen at least sometimes. I can imagine there being some program doing something unusual that gets tripped up (almost certainly at runtime, not compile-time) by shadow stacks, but doesn’t have a problem with pointer authentication. Or vice versa. The problem then becomes that we don’t have a flag for explicitly disabling one implementation but not the other. But perhaps it’s worth paying the price of over-eagerly disabling protections on a very small number of packages.

4

If we spot an issue with CFI in a package, then we can do hardeningDisable = [ "hwcfibwedge" ]; I guess? And let the maintainer pick exactly what’s needed with NIX_CFLAGS_COMPILE/NIX_LDFLAGS?

over-eagerly

Does this also affect dependencies of a program where this needs to be disabled?
The glibc INSTALL section only mentions the opposite case when using --enable-cet=permissive, i.e. that a CET-enabled program must not dlopen a non-CET program, so I’m not sure

5

Yes.

Regarding --enable-cet=permissive, let’s keep the discussion of that flag on the PR, I’m easily confused. Answering there.

6

An update on this:

A few things are pushing me back towards the second option (flags named after actual features such as shadowstack)

Some initial testing has suggested that packages broken by Shadow-Stack and PAC-RET don’t really have an overlap (in fact it’s hard to find anything broken by PAC-RET, while quite a few have problems with Shadow-Stack). ARM v9.4 is introducing its own very similar Shadow-Stack feature (though technically called Guarded Control Stack), and while concrete information is hard to come across, it doesn’t look like compilers are making the two mutually incompatible. In fact it may be desirable to enable both together as PAC-RET will work with processors all the way back to ARM v8.3, whereas at time of writing there isn’t even any released hardware with Guarded Control Stack.

RISC-V also looks like it’s getting a Shadow-Stack feature, though with more explicit instructions.

So right now I’m minded to add separate shadowstack and pacret hardening flags. Perhaps there’s an argument that we should prefix all hardware-based features with e.g. hw, because there also exist various attempts at software implementations of most of these protections and we can’t necessarily rule out one of those becoming dominant.

Following on from this there are also hardware CFI features for protecting the forward edge, known by Intel as Indirect Branch Tracking and elsewhere (ARM v8.5) as Branch Target Indicators (sometimes Identification). Here my suggestion would be to follow the Intel naming of IBT because BTI is an overloaded term with Branch Target Injection (a spectre variant).

If we did end up with flags that are mutually incompatible I would expect to resolve the conflict in add-hardening.sh alongside the fortify/fortify3 handling. If we ever decided to get really fancy we could add mechanisms to specify which flags should take priority. But let’s cross that particular bridge if we ever get to it.