I’d like to start a discussion around eventually raising the default x86_64 baseline used by NixOS/nixpkgs.
The rough proposal is to do this in two stages:
- move from the current x86-64 baseline to x86-64-v2
- later move to x86-64-v3, tentatively around 2027
The main reason for doing this incrementally rather than jumping directly to v3 is to give us time to find compatibility problems, work out the nixpkgs/toolchain implications, and decide what we want long-term support for older x86_64 hardware to look like.
There are still some fairly fundamental questions around implementation and whether microarchitecture levels should be represented as separate systems, system features, or something else.
Background info
The current x86_64 baseline lets NixOS run on a very wide range of hardware, which is obviously useful. The downside is that the compiler has to assume a pretty old CPU and can’t make use of instructions that have been common for years.
Other distributions have started looking at, or already shipping, newer x86-64 microarchitecture levels. I think it’s worth discussing whether NixOS should eventually do the same.
The x86-64 psABI defines several commonly discussed levels:
- x86-64-v1: effectively the original x86_64 baseline
- x86-64-v2: adds instructions such as SSSE3, SSE4.1, SSE4.2, POPCNT, etc.
- x86-64-v3: adds AVX, AVX2, BMI1/2, FMA, and related features
- x86-64-v4: primarily AVX-512-era extensions
I don’t think v4 makes sense as a general-purpose NixOS baseline in the foreseeable future since it’s so new and would cut off a lot of people systems out, so this proposal is specifically about v2 and v3.
Phase 1: x86-64-v2
The first step would be switching the default x86_64 build baseline to something equivalent to:
-march=x86-64-v2
It cuts off some older x86_64 CPUs, but still supports considerably more hardware than v3. It would mostly serve as an intermediate baseline that lets us exercise the infrastructure required for changing the architecture level without immediately dropping everything that lacks AVX2.
During this period we could evaluate:
- build failures caused by the new baseline
- packages that make incorrect CPU-feature assumptions
- Hydra/ofborg implications
- binary cache compatibility
- how architecture levels should be represented inside nixpkgs
- how well the performance gains justify the compatibility cost
It would also give users and maintainers some advance warning before any eventual move to v3.
Phase 2: x86-64-v3
The longer-term goal would be moving the default baseline to:
-march=x86-64-v3
A possible target would be around 2027, although I don’t think the date should be considered fixed before we have actual data and agreement on the migration path. v3 is considerably more interesting from a performance perspective because it makes AVX2, BMI1/2, FMA and several other instruction set extensions universally available to the compiler.
At the same time there is still perfectly usable hardware which does not support v3. Before making such a transition, we would therefore need a reasonably clear answer to the question of what will happen to machines that don’t support v3
That could mean retaining a legacy package set, supporting multiple architecture levels in Hydra, relying on community-maintained configurations, or some other solution. I don’t think this pre-RFC needs to decide that yet, but we probably need to solve it before v3 can realistically become the default.
Why bother?
The obvious argument is performance.
If every package has to assume an original x86_64-era CPU, compilers cannot freely generate instructions which are available on almost all modern systems. Individual projects can and sometimes do perform runtime CPU dispatch, but that isn’t free and many packages simply don’t bother.
Moving the baseline allows those optimizations to happen across the distribution rather than only in specially optimized packages.
There is also an ecosystem argument. Several Linux distributions and distribution projects have been experimenting with newer x86-64 architecture levels, including Arch-related projects, RHEL/CentOS, openSUSE and others. CachyOS in particular has useful experience distributing packages optimized for newer architecture levels.
Compatibility
Biggest downside to this change by far is comp. Every increase in the baseline makes some hardware unable to run binaries from the normal NixOS binary cache. The v2 transition would affect older systems, while v3 would exclude a substantially larger group of machines.
NixOS has users running servers, workstations and unusual systems for much longer than the typical consumer replacement cycle. f we raise the baseline, we need to support these folks rather than abandoning those users.
Another complication is determining whether architecture levels should actually be exposed as different Nix system values.
For example, should we eventually have something conceptually like:
x86_64-linux
x86_64-v2-linux
x86_64-v3-linux
or should the architecture remain x86_64-linux with the microarchitecture expressed through features or another mechanism (If so what exactly)?
There are implications either way for package evaluation, binary substitutes, Hydra, flakes, cross compilation and ofborg.
Build infrastructure
things we’d need to look at:
- stdenv and compiler configuration
- bootstrap binaries
- Hydra build platforms
- binary caches
- ofborg
- package tests
- cross compilation
- packages with handwritten assembly
- packages performing their own CPU detection
- reproducibility between builders
ofborg in particular would need to understand whichever model we adopt so that PR evaluation and tests happen against the correct architecture level.
If multiple levels remain supported simultaneously, that also increases build and maintenance costs.
Benchmarking
One weakness of this proposal right now is that there isn’t enough NixOS-specific benchmark data.
I’ve seen claims and measurements from other distributions showing meaningful gains from v3 builds, including results in roughly the 10-20% range for some workloads, but that shouldn’t be interpreted as translating to actual NixOS performance since well distros can be vastly different from one another.
The effect is extremely workload-dependent.
Some applications will benefit substantially. Others will barely change at all. Projects which already perform runtime CPU dispatch may show almost no improvement from changing the distribution baseline.
Before making any decision I’d like to see repeatable NixOS/nixpkgs benchmarks comparing:
x86-64
x86-64-v2
x86-64-v3
on identical hardware.
Ideally the test set would contain a mixture of:
- compilation
- compression/decompression
- cryptography
- scientific/numerical workloads
- multimedia encoding
- databases
- interpreters
- common desktop applications
- representative server workloads
CachyOS and similar projects could also be useful references, but they obviously shouldn’t substitute for measurements done with nixpkgs.
Migration strategy
A very rough migration could look something like this:
- Establish proper support for representing/building multiple x86-64 architecture levels.
- Add CI and Hydra coverage for those levels.
- Benchmark v1/v2/v3 using representative nixpkgs workloads.
- Document an easy way for users to check which level their CPU supports.
- Move the default to x86-64-v2.
- Keep collecting compatibility and performance data.
- Decide on a sustainable legacy-support mechanism.
- Consider moving the default to x86-64-v3 once the ecosystem and hardware distribution make that reasonable.
Probably the most important part is that each transition should be reversible during testing, rather than just plainly committing to v3.
Alternatives
Jump directly to x86-64-v3
This avoids spending time on an intermediate baseline and gets the largest potential optimization benefit immediately.
The downside is that the compatibility break is much larger and it gives us less opportunity to test the infrastructure incrementally.
Switch to x86-64-v2 only
Pretty much what it says on the tin. Move to v2 and leave v3 as an optional target instead of making it the future default.
Stay on the existing baseline
This gives us maximum compatibility and avoids additional build infrastructure.
The cost is that the default nixpkgs package set remains constrained by an architecture baseline dating back to the beginning of x86_64.
Maintain multiple baselines indefinitely
We could provide v1/v2/v3 package sets in parallel.
This would offer the best compatibility and optimization options for users, but would also have by far the largest Hydra, cache, CI and maintenance cost, which makes this option nonviable unless there some sort of miracle sponsor.
There may be a compromise where one architecture level is officially built by Hydra while older levels remain buildable but are not necessarily provided by the main binary cache.
Keep v1 and selectively build performance-sensitive packages for v3
Another option would be to keep the general nixpkgs baseline at x86-64-v1 while building selected performance-sensitive packages for x86-64-v3.
That could include things like compilers, compression tools, crypto libraries, multimedia codecs, numerical/scientific software, databases, and other packages where AVX2/FMA/BMI actually produce measurable gains.
This would preserve compatibility for most of nixpkgs while still getting some of the practical benefit of v3 where it matters.
The main problem is dependency handling. If a supposedly v1-compatible package ends up depending on a library built for v3, then that whole closure effectively requires v3. Maybe we can cache a v1 and v3 version of these packages?
We’d also need to decide how optimized variants are exposed and cached. This could be done through separate package sets, overlays, or some other mechanism rather than changing the entire system .
This approach would be interesting to benchmark, since it might capture a large part of the real-world performance benefit without requiring the entire distribution to move to v3.
Questions
The questions I’d like feedback on are:
- Does moving the default baseline make sense at all?
- Is x86-64-v2 a useful transition step, or should we eventually go directly from the current baseline to v3?
- How should microarchitecture levels be represented in Nixpkgs: separate systems, system features, or something else?
- What level of legacy hardware support should NixOS commit to?
- Would building multiple architecture levels in Hydra be practical?
- What should the benchmark suite look like?
- What existing work in nixpkgs could be reused for this?
- What would need to change in ofborg?
- What is the best way to let users determine whether their system supports v2 or v3?
I’m particularly interested in input from people familiar with stdenv/bootstrap, Hydra, ofborg, cross compilation, and the existing gcc.arch / platform CPU-feature machinery.