Nixos-rebuild system-features error when recompiling everything with gcc.arch set

Help
1

Hey, new user, no chance of losing me. I’ve searched nixos wiki (Build flags) and the web but it can’t pinpoint this problem.

This is a fresh installation on which I initially added a bunch of unconfigured package names (userland mostly) that do nothing yet.

I’ve tried setting gcc.arch and system-features. Running “nixos-rebuild switch --use-remote-sudo” failed before build starts as follows, which is the main problem I’m having:

error: a 'x86_64-linux' with features {gccarch-broadwell} is required to build '/nix/store/4szk0nzwps00kf8wrv6g2rzwrjn8wfdb-git-2.47.2.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

The relevant /etc/nixos/configuration.nix added for these attempts looks like this:

  # Build from source, when false
  nix.settings.substitute = false;
  # Included sources, when true
  system.includeBuildDependencies = true;

  # Include the 4 default features plus the gccarch-* feature
  nix.settings.system-features = [
    "benchmark"
    "big-parallel"
    "nixos-test"
    "kvm"
    "gccarch-broadwell"
  ];

  nixpkgs.hostPlatform = {
    gcc.arch = "broadwell";
    gcc.tune = "broadwell";
    system = "x86_64-linux";
  };

Following that error I’m able to start compilation using…

sudo -Hi
nixos-rebuild switch --option system-features gccarch-broadwell

… which, to start compilation, required both the dedicated root user and the options… but it fails after many hours with a combination of two errors, one similar to the above but asking for features {gccarch-broadwell, big-parallel} complaining it only provides {gccarch-broadwell}, and then a second error right after which matches exactly the prior error (so two sets of feature incompatibilities, 2 errors, sorry lost the log, and the second error implied the --option system-features gccarch-broadwell part was still insufficient to address the original issue).

I’ve also tried the following, which failed with the original error:

nixos-rebuild switch --option system-features "[ gccarch-broadwell benchmark big-parallel nixos-test kvm ]"

In order of importance:

  1. What are the current official procedures to recompile the entire OS and packages from source using gcc.arch? (wonderful feature)
  2. Do I have to manually clean the store somehow before rebuilding from source? Missing step?
  3. Do you need nix.settings.system-features in /etc/nixos/configuration.nix at all?
  4. Why does setting nix.settings.system-features in /etc/nixos/configuration.nix do nothing in this case?
  5. If you do need nix.settings.system-features, do you have to specify the 4 default ones, or can you just put gccarch-* alone and not destroy the default configuration? (I thought the list appends, but examples on the web add the 4 defaults, which seems distasteful)
  6. Is there a way to do this where you don’t have to pass --option system-features gccarch-broadwell on the command-line? It seems required to set up the compiler.
  7. Was this syntax correct: nixos-rebuild switch --option system-features "[ gccarch-broadwell benchmark big-parallel nixos-test kvm ]"?

Thanks if anyone knows what I missed

2

When you are activating that in the same step like you want to rebuild, the running nix daemon does not yet know that config.
So try to set that first, switch, and then set gcc.arch, that should work/worked last time when I played around with it.

Passing the option should also work but I am not sure how the verbatim is. Possibly you can just set it as extra-* but for that one has to dig into the nix config reading (I am not sure about it)

Once you have the System-Features enabled, you will no longer need to pass it.

Also fyi, setting gcc.arch implies setting the system-feature, you just need to “bootstrap” it (afaik).

I can recheck today in the evening and report back that’s needed for me (I was building x86_64-v3)

3

I see what you mean. I’m commenting out nixpkgs.hostPlaftorm and all my packages for now but leave the nix.settings.system-features in, not sure if I still needed the --option system-features gccarch-* then though

Note: That system was stuttering hard on the desktop for awhile after the last failure. I think something about a partial recompile (linux-firmware?) affected the state, but it’s going again.

[This is a really well-designed system. I used Debian for years and ended up making grsec-hardened whonix live CDs on arch, and this would have blew that all away, so in the end everyone will just use a nixos-qubes. I saw they were making Nix FreeBSD, I was hoping for a redox-os version someday]

4

It made sense if you have to recompile the toolchain for that arch first, the other people said similar things

5

If you set nix.settings.system-features you dont need any --option system-feature.

So i was checking --option * variants and was not able to make it happen.

nix show-config --option extra-system-features gccarch-x86-64-v3 does show that it should be active, but trying to build it ends in still asking for a builder with that system capabilities.

So i think that there is currently no way around than setting nix.settings.system-features explicitly once, and then build the system.

In case you are later looking to exclude packages from the builds, you can to something like

 nixpkgs.config.packageOverrides = pkgs: {
        inherit (unoptimized) openexr_3;
        haskellPackages = pkgs.haskellPackages.override {
          overrides = haskellPackagesNew: haskellPackagesOld: {
            inherit (unoptimized.haskellPackages) cryptonite hermes-json hermes-json_0_2_0_1;
          };
        };

        inherit (unoptimized) portfolio libreoffice-qt krita;
      };

which overrides pkgs.openexr_3, pkgs.haskellPackages.{cryptonite,hermes-json,hermes-json_0_2_0_1}, pkgs.portfolio, etc pp. With that you should be able to do all you will need for the first steps.

I hope that help a bit in customizing your system.

6

Thanks, more configurations needed before even first-pass compile will succeed. It was already going better - then too many open files failure

7

I can at least confirm that passing --option extra-system-features alone as suggested on the wiki simply does not work here (this is without setting gcc.arch):

error: a 'x86_64-linux' with features {big-parallel} is required to build '/nix/store/v63sw7k91laj0lqnn6ir79pa90zjg3a3-llvm-18.1.8.drv', but I am a 'x86_64-linux' with features {gccarch-broadwell}

It seems to override the default system-features for some of the build, so I guess it only works when building individual packages.

Using

  nix.settings.system-features = [
    "benchmark"
    "big-parallel"
    "nixos-test"
    "kvm"
    "gccarch-broadwell"
  ];

under sudo nixos-rebuild switch - without settings gcc.arch - appears to allow compilation to proceed, although mine is failing for other reasons.

I managed one successful build without gcc.arch (with gcc-arch-* system-feature), but could not get subsequent builds with gcc.arch to work due to obscure failures. Then, after removing gcc.arch again, I keep running into this timeout:

90% tests passed, 2 tests failed out of 20

Total Test time (real) = 1574.72 sec

The following tests FAILED:
         16 - exhaustive1_test (Timeout)
         19 - exhaustive_test (Timeout)
Errors while running CTest
FAILED: CMakeFiles/test.util
cd /build/source/build && /nix/store/w5ccvkai5qqrla7k10zd718drb44m2km-cmake-3.30.5/bin/ctest --force-new-ctest-process
ninja: build stopped: subcommand failed.
error: builder for '/nix/store/83ba3hjqrii840rjiyslk7vh2k5pqrf3-re2-2024-07-02.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/phf7llgdw1gaydv29rifwq6dq16vndv5-closure-info.drv' failed to build
error: 1 dependencies of derivation '/nix/store/cmgx6k1c77yhrpi9wqv5d1rw1yxj8f1h-nixos-system-cathbg-24.11.716687.bdb91860de2f.drv' failed to build

Anyhow this seems to devolve into various other build issues I’ll have to sort through (but any ideas to prevent the above error welcome - might be a timeout setting somewhere).

Otherwise the original question was addressed.

8

When I was doing such custom builds I made two things:

  1. I noticed few builds that are just unstable and checked if they are also unstable on default settings by searching for builds on the official hydra instance. For those I just kept restarting until they were fine, sadly that’s something then you have to repeat regularly.
  2. There had been some builds that are just not building or working with optimizations, those I did override to use the default build flags. The way I was doing it was by overriding the packages with the snippet I already posted. A package I remember for my closure was simdjson, that just did not build.

With that as combination I was able to build a system closure that contained a plasma shell and my daily drivers, so I was using the system.

Tbh, at some point I dropped the full rebuild, for my setup the difference was between not noticable and maybe very little noticable (depending on what I did) [I never ran actual benchmarks, but I also did not optimize for the specific CPU, but a x86_64 level].
I ended then in building just the kernel and the mesa stack with flags, which I used quite a while.

Since I dropped my custom kernel build I am back to full default packages.

Personal opinion: If you are looking for more performant binaries, investigation to LTO and PGO might be more worthy results.
Sadly I did not (yet) dig down that rabbit hole, as I am currently on limited times due other personal stuff.