# Pre-RFC: Gradual Transition of NixOS x86\_64 Baseline to x86-64-v3 with an Intermediate Step to x86-64-v2
**URL:** https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924
**Category:** RFCs
**Created:** [November 23, 2023, 5:45pm UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924 "2023-11-23T17:45:01Z")
**Posts on this page:** 20
**Page:** 3
### Author:  [@nyanbinary](https://discourse.nixos.org/u/nyanbinary)
#### Post date: [December 12, 2023, 10:52pm UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/41 "2023-12-12T22:52:29Z")
> **[Ubuntu Linux Evaluating x86-64-v3 Based Build - AVX & Newer Intel/AMD CPUs](https://www.phoronix.com/news/Ubuntu-x86-64-v3-Experiment)**
>
> Canonical is experimenting with x86\_64 micro-architecture feature levels! They have produced an experimental build of Ubuntu Server using x86\_64\_v3 for requiring basically Intel and AMD CPUs with AVX capabilities
---
### Author:  [@Atemu](https://discourse.nixos.org/u/Atemu)
#### Post date: [December 13, 2023, 8:41am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/42 "2023-12-13T08:41:51Z")
TL;DR Ubuntu is looking into this topic aswell and have come to the same conclusion as we have: There is no reliable data on the performance benefit and we need actual real-world people to test their workloads on real systems to find out.
---
### Author:  [@reckenrode](https://discourse.nixos.org/u/reckenrode)
#### Post date: [December 17, 2023, 4:05am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/43 "2023-12-17T04:05:22Z")
> [@Atemu](#):
>
> The thought that compiler optimisations such as AVX only exclude “older systems” (as in: decades old) is wrong.
>
> There is hardware as recently released as _this year_ which does not support AVX of any kind: [Tremont (microarchitecture) - Wikipedia](https://en.wikipedia.org/wiki/Tremont_(microarchitecture)). Moving to v3 in 2027 would exclude this hardware 4 years after release.
>
> Such low power Celerons are somewhat popular in the homelab scene for their extreme power efficiency and low prices (commonly available in used thin clients).
>
> My NAS uses a Celeron J4105 from 2017 ([Goldmont Plus](https://en.wikipedia.org/wiki/Goldmont_Plus)) and I know that @musicmatze uses a similar chip aswell.
My router is firewall appliance with a Celeron J3060 running NixOS. I could set up a builder if the change is made, but I’d really prefer not to have to do that.
---
### Author:  [@Atemu](https://discourse.nixos.org/u/Atemu)
#### Post date: [December 27, 2023, 3:59pm UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/44 "2023-12-27T15:59:34Z")
Phoronix has a decent set of benchmarks on Ubuntu’s experimental x86\_64-v3 ISO:
> **[Benchmarking The Experimental Ubuntu x86-64-v3 Build For Greater Performance...](https://www.phoronix.com/review/ubuntu-x86-64-v3-benchmark)**
>
> One of the exciting innovations currently being explored by Canonical ahead of the Ubuntu 24.04 LTS release is an x86-64-v3 build of the OS / packages.
Summary:
> The Ubuntu x86-64-v3 performance benefits overall were typically small but consistent. In some workloads the x86-64-v3 applications obtained from the archive could be a great deal faster but ultimately it comes down to a subset of software that will really benefit.
---
### Author:  [@nyanbinary](https://discourse.nixos.org/u/nyanbinary)
#### Post date: [December 30, 2023, 7:17pm UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/45 "2023-12-30T19:17:47Z")
Now Fedora is looking into providing optimized packages too.
> **[Fedora 40 Looks To Provide Optimized x86\_64 Binaries For Different HWCAPs](https://www.phoronix.com/news/Fedora-40-Faster-x86-64)**
>
> In addition to Ubuntu exploring the possibility of x86\_64-v3 builds/packages, a proposal has been raised for Fedora Linux with its current Fedora 40 cycle to provide the ability to offer optimized x86\_64 (AMD64) binaries based upon the CPU's x86\_64...
---
### Author:  [@vcunat](https://discourse.nixos.org/u/vcunat)
#### Post date: [January 4, 2024, 6:50am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/46 "2024-01-04T06:50:05Z")
RHEL 10 will also go for it, apparently: [https://developers.redhat.com/articles/2024/01/02/exploring-x86-64-v3-red-hat-enterprise-linux-10](https://developers.redhat.com/articles/2024/01/02/exploring-x86-64-v3-red-hat-enterprise-linux-10)
But note that distros with very long support cycles (5–10 years) can be more aggressive about this than e.g. NixOS/NixPkgs, as just after several months you’d be left without compatible binaries on any maintained version.
---
### Author:  [@Atemu](https://discourse.nixos.org/u/Atemu)
#### Post date: [January 4, 2024, 8:41am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/47 "2024-01-04T08:41:31Z")
From the Fedora 40 announcement:
> Systemd will be modified to insert the additional directories into the $PATH environment variable (affecting all programs on the system) and the equivalent internal mechanism in systemd (affecting what executables are used by services). Individual packages can provide optimized libraries via the glibc-hwcaps mechanism and optimized executables via the extended search path.
This is an interesting workaround for the “doesn’t work with static executables” limitation of glibc-hwcaps that we should be able to trivially replicate.
---
### Author:  [@nyanbinary](https://discourse.nixos.org/u/nyanbinary)
#### Post date: [January 4, 2024, 10:28pm UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/48 "2024-01-04T22:28:40Z")
tbf we could also do that but just announce the changes now and implement them 5 years later (eg announcing a v2 baseline this year and then actually implementing the baseline in 2029)
---
### Author:  [@vcunat](https://discourse.nixos.org/u/vcunat)
#### Post date: [January 5, 2024, 8:19am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/49 "2024-01-05T08:19:18Z")
I’m afraid I fail too see an advantage in announcing such a thing years in advance.
---
### Author:  [@boomshroom](https://discourse.nixos.org/u/boomshroom)
#### Post date: [January 7, 2024, 12:24am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/50 "2024-01-07T00:24:59Z")
Providing specific packages that make good use of the optimizations would be valuable for saving build time and storage space. However, being Nix, it’s already possible to build packages for specific CPU micro-architectures. This prevents using the cache as would be expected, but it’s also just really confusing to use, and depending on how you do it, it could involve rebuilding everything all the way down to the bootstrap tools just to optimize _one_ package.
Before changing the default, I’d suggest we make it easier for users to select which micro-architecture or level to build for to apply to individual packages. If the package in question is a library, then dependant packages could use it with overlays, though this would also require rebuilding those dependent packages.
Another advantage we have being Nix is that changing the default does not mean completely dropping support. Much like how you can already set a micro-architecture higher than the baseline, increasing the baseline should still allow using older hardware at the cost of specifying so explicitly and less ability to use the caches. Assuming we can actually make setting the micro-architecture simple enough so people would actually understand how to do it, we could just say to do so in case one has older hardware. They’d need to compile manually, but it should be possible for Hydra to build at least stdenv for multiple micro-architectures, even if not all of nixpkgs.
TLDR: Increasing the baseline isn’t a compatibility problem for us like it is with other distros, but rather a documentation problem.
Also, Nixpkgs will simply deny v4 optimizations when using GCC 12, which is the default on unstable right now. Staging has updated to GCC 13, but it could be some time before that propogates to other channels.
---
### Author:  [@Atemu](https://discourse.nixos.org/u/Atemu)
#### Post date: [January 7, 2024, 1:39am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/51 "2024-01-07T01:39:31Z")
> [@boomshroom](#):
>
> depending on how you do it, it could involve rebuilding everything all the way down to the bootstrap tools just to optimize _one_ package.
There’s rarely such a thing as “optimising one package”. The vast majority of packages use libraries for much (if not most) of their work. This means these libraries also influence the package’s performance.
I.e. optimising ffmpeg would likely gain very little (if any) vorbis encoding performance because ffmpeg uses a dedicated library for the performance-critical part of that.
> [@boomshroom](#):
>
> changing the default does not mean completely dropping support. Much like how you can already set a micro-architecture higher than the baseline, increasing the baseline should still allow using older hardware at the cost of specifying so explicitly and less ability to use the caches.
\*no ability to use the caches.
You likely wouldn’t even be able to boot an installer ISO on an unsupported CPU.
Raising the default baseline will exclude the majority of users of then-unsupported hardware. There’s no way around it.
> [@boomshroom](#):
>
> They’d need to compile manually, but it should be possible for Hydra to build at least stdenv for multiple micro-architectures, even if not all of nixpkgs.
Just the stdenv wouldn’t really help anyone. If you’re going to be building your average desktop closure, having the stdenv cached or not doesn’t make a huge difference.
Suffice to say, this is not a feasible option if you want users of then-unsupported hardware to still be able to reasonably use NixOS. Especially considering their hardware likely being highly unsuited for compiling modern software.
If anyone should be building their own packages, it’s the users of powerful modern CPUs who need the handful of % mean improvement for …what reason exactly?
* * *
Anyhow, I think it’s reasonably well established that:
- There is a point to this. With v3, the increase is small and highly package dependant but it’s there, it’s real and likely desirable, especially for those few packages.
- There are many users who would be effectively excluded from using the distro were the baseline raised to v3 and even some who would be excluded by v2.
- We’re can have our cake and eat it too at the expense of a closure size increase (how much?) using glibc-hwcaps in addition to the `$PATH` workaround for the executables themselves.
The next step is to build code infrastructure for glibc-hwcaps usage and use it on packages&libraries that benefit.
Until that is done and well established, I think we can pause any discussion on raising the baseline.
---
### Author:  [@reckenrode](https://discourse.nixos.org/u/reckenrode)
#### Post date: [January 9, 2024, 7:02am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/52 "2024-01-09T07:02:43Z")
Would this change only apply to Linux or also include Darwin?
Darwin has two levels of support:
- x86\_64: Implies SSE3 but not necessarily SSE4.2.
- x86\_64h: This appears equivalent to x86-64-v3. It targets Haswell CPUs or newer.
The only caveat is Rosetta 2 does not support AVX, so it can’t really benefit from x86\_64h. However, the most likely approach would be to build fat binaries supporting both architectures, so it could continue to run x86\_64 code. And if you really concerned about performance on an aarch64 Mac, you should be using native code anyway.
(Whether it’s worth the effort is a separate question. My inclination is to say it’s not.)
---
### Author:  [@nyanbinary](https://discourse.nixos.org/u/nyanbinary)
#### Post date: [January 21, 2024, 12:02am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/53 "2024-01-21T00:02:58Z")
> **[Serpent OS To Require x86\_64-v2 CPUs While Offering x86\_64-v3 Packages Too](https://www.phoronix.com/news/Serpent-OS-January-2024)**
>
> Serpent OS as the latest Linux distribution project of well known developer Ikey Doherty is off to a great start for 2024
Seems SerpentOS is going to use a custom superset of v3 called v3x.
---
### Author:  [@yshui](https://discourse.nixos.org/u/yshui)
#### Post date: [January 21, 2024, 3:29pm UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/54 "2024-01-21T15:29:23Z")
Can we add an option, perhaps to stdenv?, that selects the x86\_64 baseline? Then we can have multiple versions of each package with different baselines in the cache, and let the user choose based on their hardware.
---
### Author:  [@raphi](https://discourse.nixos.org/u/raphi)
#### Post date: [January 21, 2024, 3:36pm UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/55 "2024-01-21T15:36:29Z")
The setting exists, [`gcc.arch = "x86_64-v2"`](https://nixos.wiki/wiki/Build_flags). Caching multiple architecture levels requires a lot of additional cache space and build farm time.
---
### Author:  [@jorgesgk](https://discourse.nixos.org/u/jorgesgk)
#### Post date: [February 17, 2024, 6:27pm UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/56 "2024-02-17T18:27:58Z")
Considering Nix is source-based, wouldn’t it be possible to still support older microarchitectures by just not using the cache?
---
### Author:  [@raphi](https://discourse.nixos.org/u/raphi)
#### Post date: [February 17, 2024, 7:04pm UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/57 "2024-02-17T19:04:49Z")
Older microarchitectures have less processing power for compiling software. Therefore they need cached builds the most.
---
### Author:  [@Atemu](https://discourse.nixos.org/u/Atemu)
#### Post date: [February 18, 2024, 8:54am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/58 "2024-02-18T08:54:30Z")
See discussion in [Pre-RFC: Gradual Transition of NixOS x86\_64 Baseline to x86-64-v3 with an Intermediate Step to x86-64-v2 - #51 by Atemu](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/51)
---
### Author:  [@Atemu](https://discourse.nixos.org/u/Atemu)
#### Post date: [February 21, 2024, 5:45pm UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/59 "2024-02-21T17:45:58Z")
It appears I must retract a statement of mine again in light of new data.
I’ve recently been reminded of a talk I saw a few years ago where evidence was presented that LLVM 3.1’s `-O3` shows no significant improvement over `-O2` if you continuously randomise the memory layout.
`-O2` vs `-O3` is not the topic of this thread of course but it’s closely related. An important detail I had forgotten was just how significant the measurement bias caused by memory layout is and how easily it’s caused.
In short: Even extremely minor changes such as a different length of UNIX user name or different working directory can cause very significant changes in performance of the same binary.
If @nyanbinary and I ran the same executable on the same hardware and software environment, we could realistically see something on the order of a 10% performance difference just because my UNIX user name is shorter.
This is a huge issue because the slightly different memory layout due to slightly different code generation (i.e. different `-march`) can cause significant performance differentials even if that slightly different code itself causes no significant difference at all.
[The paper which first reported on this issue](https://users.cs.northwestern.edu/~robby/courses/322-2013-spring/mytkowicz-wrong-data.pdf) claims variance of upwards of 20% in industry standard benchmarks (though usually lower) just due to such measurement biases.
Any experiment which shows a significant but not immense increase by changing the compiler optimisation µ-arch target could be fully explained by just changes in memory layout alone. Something that looks like a slight increase could be a slight regression that is masked by the measurement bias in actuality.
The authors of the original paper do propose a method to control for some of the biases and have shown that a sufficiently diverse set of benchmarks can mitigate the measurement bias to some degree.
The presenter of the talk I watched and author of the paper it’s based upon has developed an effective technique to control this bias called [Stabilizer](https://github.com/ccurtsinger/stabilizer) which continuously randomises the memory layout down to the function at runtime; thereby removing any possibility of their position to impact performance.
Sadly the original project has not been developed further and is not compatible with modern clang or gcc. There are two forks which have attempted to support newer versions of clang ([12](https://github.com/magras/stabilizer), [16](https://github.com/parsa/stabilizer)) but they do not appear to be very mature.
Stabilizer talk recording on Youtube: [https://youtu.be/r-TLSBdHe1A](https://youtu.be/r-TLSBdHe1A) (well worth watching)
Stabilizer paper: [https://people.cs.umass.edu/~emery/pubs/stabilizer-asplos13.pdf](https://people.cs.umass.edu/~emery/pubs/stabilizer-asplos13.pdf)
“Producing Wrong Data Without Doing Anything Obviously Wrong!”: [https://users.cs.northwestern.edu/~robby/courses/322-2013-spring/mytkowicz-wrong-data.pdf](https://users.cs.northwestern.edu/~robby/courses/322-2013-spring/mytkowicz-wrong-data.pdf)
* * *
Given how none of the data presented so far even attempts to control for this bias, I am no longer sure this is worth pursuing at all, even in the “lite” form of glibc-hwcaps.
I now also need to question the majority of benchmarks of software I see anywhere because, to my knowledge, none of those control for this bias either.
---
### Author:  [@twoolie](https://discourse.nixos.org/u/twoolie)
#### Post date: [February 26, 2024, 4:58am UTC](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924/60 "2024-02-26T04:58:35Z")
I decided to try and rebuild my laptop with various arch flags and found a lot of build failures, such as libvorbis segfaulting on a zen4 arch, x509 certificate validators failing on x86\_64-v3 and more. I think having a parallel build pipeline for at least a core subset of packages for different architectures would be useful to shake out these failures early.
> [@raphi](#):
>
> The setting exists, [`gcc.arch = "x86_64-v2"`](https://nixos.wiki/wiki/Build_flags). Caching multiple architecture levels requires a lot of additional cache space and build farm time.
There’s a lot of derivations in the store that don’t care what platform they’re running on (think pure-python, pure-ruby, npm packages), which get these implied dependency on the architecture because they’re force-built against a particular version of of the host interpreter just in case it relies on a c lib, which itself is arch dependent.
The design of Nix makes this really hard to do, but surely there’s got to be a way to split these derivations out into architecture-agnostic package-sets. Perhaps content-addressed packages may provide a way to collapse truly-arch-independent packages down automatically, by collapsing N different derivations that all build to the same set of files into a single output?
[Previous page](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924.md?page=2)
[Next page](https://discourse.nixos.org/t/pre-rfc-gradual-transition-of-nixos-x86-64-baseline-to-x86-64-v3-with-an-intermediate-step-to-x86-64-v2/35924.md?page=4)