According to the hydra log, it failed with “Fatal Python error: Illegal instruction”. I believe that this is could have been caused by the build happening to run on an older x86 machine without some instruction sets (AVX?), and it’s possible that the package doesn’t compile properly on such a machine.
Is there a general nixpkgs policy on supported instruction sets?
What’s the minimum set of x86_64 ISA extensions supported by the hydra fleet?
When hydra builds a package for x86_64, which ISA extensions should be assumed (SSE{3,4,4.1,4.2}, AVX, AVX-2, …)?
Is it a bug if a package only builds or works on a machine with some minimal set of ISA extensions that is apparently not shared by the entire hydra fleet?
Apologies if these questions have already been answers. I’m still pretty new here.
at a guess it will probably compile fine, as the compiler itself probably doesn’t use CPU optimisation sets, it’s more likely to fail during the unit or integration test phases.
Hydra is a venerable and loveable old beast, my limited experience with it, I’ve not seen any options for allocating build slaves depending on their physical configuration. Hydra kinda assumes that every thing can build and test everything AFIAK. (please correct me if i’m missing something). Hydra even in it’s current form is still one of the best CI/CD servers I’ve ever come across for a multitude or reasons.
It would be nice to get more package sets being compiled with hardware optimisation enabled , very similar to what clear linux does with it’s package sets. Hydra only has a finite resources available to it, but i hope to see trustnix being able to build more and more specialist packages with optimisations enabled, that can be cached and use by everyone.
Yes, this is the reason and it’s even worse, some of the builders do not even support SSE 4.1.There are two common workarounds. (1) As @nixinator says, this can usually be worked around by disabling tests, so that the compiled code does not get executed (though disabling tests is of course not nice). See e.g.:
(2) The software may build with -march or other flags, that need to be disabled.
I think the oldest CPU in the Hydra build farm is an Opeteron 6100 series, which supports SSE3:
I’d consider this an upstream bug, software should use CPUID and select code paths or kernels based on that. But there is a lot of machine learning software out there that doesn’t do this and simply relies on -march=native plus autovectorization.
I think my remaining question though is still: for upstreams that only support compile-time selection of the x86 ISA extensions (e.g. -march=...), which, as you point out is pretty common, what level should we compile them at by default in nixpkgs and on hydra?
It seems like the most practical answer is to compile them for a “lowest common denominator” ISA that:
(a) is supported by all CPUs in the hydra build farm.
(b) is supported by almost all CPUs used “in the wild” by users of the hydra cache.
Perhaps the answer is “up to and including SSE3”, but it’d be good to have some guidance or a policy on that.
I wanted to look into nixpkgs integration next. However, it won’t be of much use until Nix on Hydra has the right version. Also, in nixpkgs, we’d need to gracefully handle this on machines that do not have a new-enough version of Nix yet. Stay tuned!
We should probably have a word about how these work in the Nix docs. Right now it is an undocumented feature and I’d raise that with any nixpkgs PR that uses this.