Pre-RFC: Gradual Transition of NixOS x86_64 Baseline to x86-64-v3 with an Intermediate Step to x86-64-v2

Development RFCs
61
62

So even for computational demanding software the performance difference between v2 and v3 is mostly within the error margins.

My takeaway: Stick with current settings and add a good guide that:

  • explains how to optimize individual packages by setting architecture feature flags
  • explains how to confirm that there is an actual speed-up (which is not guaranteed, see zstd compression in the CachyOS benchmarks) on the target machine and with the expected workload.

But I guess people that have a real requirement for these optimizations (e.g. in the HPC context) are already tuning their packages.

5 Likes
63

Came here to post this too.

General observation seeems to be not much difference except some particular cases, which matches discussions so far generally.

But who would have guessed that PHP was one of those cases?!

64

Yep, I had this intuition when I was doing performance diagnostics on Nextcloud deployments and looking at perf for a while. I think PHP is not exploiting the hardware in any serious way via cpuid alas.

But this is an interesting observation, this makes it really compelling to encourage people to use PHP on a higher tier if they need that performance boost on modest CPUs albeit having advanced CPU instruction sets.

Though, I guess now it’d be interesting to do real world benchmarks on PHP applications. :slight_smile:

65

Just a footnote:

image
image1453×465 58.8 KB

66

performance difference between v2 and v3 is mostly within the error margins.

Ok, but we’re not even enabling v2 yet IIRC. It seems an easy win to enable compilation for v2 given its broad compatibility (any post~2009 cpu). This would allow us to test the waters and identify misbehaving packages/toolchains early. instead of relying on random end user builds to discover packages that break with arch specific flags, and have to patch in exceptions for those packages.

Also eventually I’d love to see PGO/BOLT enabled packages on NixOS but it seems unlikely we’d be able to get there if we can’t stabilize builds enabling post-2009 cpu features.

67

I don’t see any data suggesting a significant “win” if you account for the insane error margins that the measurement bias imposes.

PGO/BOLT don’t exclude any hardware to my knowledge. They are an entirely tangential topic.

2 Likes
68

Pre-RFC: Gradual Transition of NixOS x86_64 Baseline to x86-64-v3 with an Intermediate Step to x86-64-v2 - #38 by riceicetea
we are not using O3 because sometimes decreases performance (by agressive unrolling, etc).

But, has anyone ever benchmarked a full-o3 system to an o2 system?

I’ve asked the CachyOS people (ricers, I know!) about this kind of thing over telegram, and their response is that 2020s CPUs actually respond well to -O3, but only if you also use x86-64-v3.

See also Sunnyflunk.

Pre-RFC: Gradual Transition of NixOS x86_64 Baseline to x86-64-v3 with an Intermediate Step to x86-64-v2 - #39 by Atemu
O3 includes potentially optimisations that produce potentially unsafe and/or wrong code.

Are you confusing -O3 with -Ofast, or are you talking about undefined behavior getting surfaced? Either way, it’s not totally the compiler’s fault, and buggy packages can be marked as O2-only anyways until more investigation is done.

(There are some more commonly used UBs, but compilers have accommodating flags like -fwrapv. Some use fno-strict-aliasing too.)

69

That’s nice but that’s not data.

There is a ton of things you can unknowingly do wrong when producing such data (see my previous post), so even if it was data, it’d have to be extremely clear and plentiful, not just a minor difference in a subset of benchmarks.

I’m not entirely sure where I have that from, so it might not be true (anymore?) but I don’t mean -Ofast. -Ofast enables things that knowingly break certain aspects of specifications/standards.

70

Long ago gcc -O3 really used to produce buggy code relatively commonly IIRC. I don’t think that should hold anymore. But if -O3 was a good default in general (e.g. for a whole distro), I wonder why gcc is still keeping -O2 as the default.

4 Likes
71

I don’t think I quite understand the argument around variation due to memory layout, and controlling for it. If the benefits observed from bumping the x86 microarchitecture level are random variation due to different memory layout, shouldn’t it be essentially random whether bumping the microarchitecture level helps or hurts performance? But what we observe is slightly improved performance, quite consistently. I don’t see how such random variation would produce an upward bias in the data. If the benchmarks so far show that this consistently increases performance, why do we need to control for random variation if that variation is indeed random?

2 Likes
72

Another datapoint (article in German): Ubuntu 25.10 führt optimierte Pakete für moderne x86-64-CPUs ein | heise online
The article is basted on this posting: Introducing architecture variants: amd64v3 now available in Ubuntu 25.10 - Foundations - Ubuntu Community Hub
TL;DR: Ubuntu provides packages for x86_64-v3 via opt-in, average expected performance improvement is ~1% (numbers from December 2023…)

4 Likes
73

For anyone interested in testing x86_64-v3 and providing some useful data: NixOS 26.05 is pretty easy to get running in full x86-64-v3 mode: A bunch of issues and flaky test from 25.11 have been fixed, including opencolorio, openvswitch, numpy and more.

I have a branch of my config that successfully builds, boots and runs with gcc.arch = “x86-64-v3”; set globally and only 3 simple overrides: NixOS/desktop at v3-fix · Momi-V/NixOS · GitHub

It has a full KDE Plasma Desktop, Steam, Blender (with ROCm), Firefox, Chromium, Electron Apps, Cemu, Lutris and more, 1957 (nix-system) and 1403 (nix-user) packages in total.

Quick tutorial for anyone interested in testing this:

  1. (required) Set nix.settings.system-features = [ “gccarch-x86-64-v3” ]; to whatever your CPU can run and you want to test. You can include more like [ “gccarch-znver3” “gccarch-x86-64-v3” “gccarch-x86-64-v2” “gccarch-x86-64” ], this just tells nix your local machine has the capability to build and run these. These can stay set even during normal use, it will not affect your system’s daily use in any way. I have these in my hardware-configuration.nix
  2. (recommended) If you have many cores set nix.settings.max-jobs = 16; (or less) and nix.settings.cores = 16; (or less) to reduce memory usage. 16 and 16 worked fine for me with 64GB RAM and 64GB zSwap, mostly using 30GB - 55GB with a few spikes to ~80GB. Prefer lowering max-jobs while keeping cores higher to prioritize finishing compiles, not starting new ones. I used nix.settings.max-jobs = 1; nix.settings.cores = 31; at some points to quickly test changes to make failing jobs succeed without wasting time waiting on other jobs. I have these options in my configuration.nix.
  3. (REQUIRED) run nixos-rebuild switch to apply these options BEFORE setting gcc.arch = “x86-64-v3”; and starting to compile stuff. If you change anything from 1. and 2. comment out gcc.arch and related options to apply the changes with nixos-rebuild switch, then uncomment them again to do the build.
  4. (required) comment out (or delete) nixpkgs.hostPlatform = lib.mkDefault “x86_64-linux”; in hardware-configuration.nix
  5. (required) add to your config (I have this in my hardware-configuratioon.nix):
  nixpkgs.localSystem = {
    gcc.arch = "x86-64-v3";
    gcc.tune = "generic";
    system = "x86_64-linux";
  };

Fixes (I have this in my configuration.nix):

Then just sudo nixos-rebuild build, wait for it to complete, monitor RAM usage, restart if it fails (there are still a few flaky tests that just time out under high CPU load), potentially decrease nix.settings.max-jobs and when it’s done run sudo nixos-rebuild boot and restart.

5 Likes
74

What’s still required is some benchmark data: I tried getting phoronix-test-suite running, but that’s really annoying on NixOS (even without any gcc.arch stuff in the mix). I tried playing with buildFHSEnv but didn’t get far: NixOS/desktop at v3-phoronix · Momi-V/NixOS · GitHub .

If anyone has ideas on different ways to benchmark things or methods to get phoronix running please let me know. If you hit errors feel free to ping me, maybe we can build a set of overlays to get even more stuff working fine. You can use something like this to build a package on the CLI standalone and use nix log <store-path-in-error-message.drv> to get details on what went wrong:

nix build \
  --impure \
  --keep-failed \
  --expr '
    let
      pkgs = import <nixpkgs> {
        localSystem = {
          system = "x86_64-linux";
          gcc = {
            arch = "x86-64-v3";
            tune = "generic";
          };
        };
      };
    in pkgs.assimp.overrideAttrs (old: {
      NIX_CFLAGS_COMPILE = (old.NIX_CFLAGS_COMPILE or "") + " -ffp-contract=on";
    })
  '
1 Like
75

Long term I’d like to try spinning up a Hydra instance to build x86-64-v3 stuff and see what’s working and what isn’t, that is if I can somehow find a way to feed gcc.arch = “x86-64-v3”; into the build process as a global override.

2 Likes
76

This was reported today:

This is just a single test case of one test suite and even this large of a difference could probably still be explained by measurement bias.

I’m not sure this has been explored much though?

Tuning compile-time assumptions for modern hardware certainly sounds like something that could provide considerable benefit in basically any task worth considering; a large part of the advances of hardware performance in the past ~10 years have been due to e.g. improved branch prediction.

We can adjust the tune without restricting the hardware the result can run on. It might run slower on older hardware but I’d argue that, if you cared about performance, you should not be running ancient hardware.

I’d consider setting a “modern” tune a possible avenue. I’d also consider the bar for required evidence of benefit quality quite a bit lower as we’re not excluding anyone by doing so. Though I also see it possible for a “modern” tune to actually regress performance even on modern hardware and would personally prefer strong evidence that that isn’t the case – that, in the worst case, we keep the status quo.

I’d also like to request any future research into the benefit of -march to normalise the tune in order to measure only the benefit we get from requiring modern instruction set extensions and thereby excluding users of old hardware.

7 Likes