While exploring the possibility of incorporating some Rust in a cmake-managed C++ project, I am trying to follow this template. It works like a charm with -DENABLE_LTO=OFF (cmake, make and ./main all succeed), but I’m not having any luck with -DENABLE_LTO=ON, always getting
-- IPO / LTO not supported: <Change Dir: /tmp/cxx-cmake-example/build/CMakeFiles/_CMakeLTOTest-CXX/bin
at the cmake step
This issue still being open, look ominous. Variations on this theme haven’t born any fruit.
Can you offer any advice on how to get LTO to work in this template, with nix-provided tools (Rust/C++ compilers/linkers)?
This doesn’t work because LTO transfers the grunt work of the compilation to the linker. I’m not exactly sure if LLD invokes the compiler again for that or applies the passes in process, but nevertheless LLD doesn’t know how to deal with the LTO intermediate format of your compiler, which seems to be GCC.
To make it work you can create a shell with clang as the default compiler with this:
Dodging the issue of what has happened to lldClang in unstable, I pinned nixpkgs to 20.09 of today. This allowed me to make more progress, with the cmake step now reporting
-- IPO / LTO enabled
However, the make step failed with (buried among lots of noise)
Replacing llvmPackages_11 with llvmPackages_12 led to a successful make. Unfortunately ./main reports
Calling rust function, time elapsed: 4477595 ns.
Calling c++ function, time elapsed: 90 ns.
that is to say, the rust version is orders of magnitude slower than the C++ version, which is expected without LTO, but with LTO they should be comparable.
In other words, even though it compiles, the optimization appears not to work.
Starting with the version that compiled an ran, but where the LTO seems to have no effect on the speed of the Rust version; changing the pinned nixpkgs version to unstable, and making the change you suggested:
(in case it’s any use, I’ve pushed the whole lot here)
it now fails at the cmake stage with:
cmake -DENABLE_LTO=ON -DCMAKE_BUILD_TYPE=Release ..
-- The C compiler identification is Clang 12.0.0
-- The CXX compiler identification is Clang 12.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /nix/store/x9hgvxa06b03n3j282dpj5bv68hz5mws-clang-wrapper-12.0.0/bin/clang
-- Check for working C compiler: /nix/store/x9hgvxa06b03n3j282dpj5bv68hz5mws-clang-wrapper-12.0.0/bin/clang - broken
CMake Error at /nix/store/zzqzl5lm59mc04fd8f1wbwl7nmxvv1bb-cmake-3.19.7/share/cmake-3.19/Modules/CMakeTestCCompiler.cmake:66 (message):
The C compiler
"/nix/store/x9hgvxa06b03n3j282dpj5bv68hz5mws-clang-wrapper-12.0.0/bin/clang"
is not able to compile a simple test program.
It fails with the following output:
Change Dir: /home/jacek/src/cxx-cmake-example/build/CMakeFiles/CMakeTmp
Run Build Command(s):/nix/store/g2fna66r9m081w1h1zj857j06jigx6cq-gnumake-4.3/bin/make cmTC_c6a62/fast && /nix/store/g2fna66r9m081w1h1zj857j06jigx6cq-gnumake-4.3/bin/make -f CMakeFiles/cmTC_c6a62.dir/build.make CMakeFiles/cmTC_c6a62.dir/build
make[1]: Entering directory '/home/jacek/src/cxx-cmake-example/build/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_c6a62.dir/testCCompiler.c.o
/nix/store/x9hgvxa06b03n3j282dpj5bv68hz5mws-clang-wrapper-12.0.0/bin/clang -o CMakeFiles/cmTC_c6a62.dir/testCCompiler.c.o -c /home/jacek/src/cxx-cmake-example/build/CMakeFiles/CMakeTmp/testCCompiler.c
Linking C executable cmTC_c6a62
/nix/store/zzqzl5lm59mc04fd8f1wbwl7nmxvv1bb-cmake-3.19.7/bin/cmake -E cmake_link_script CMakeFiles/cmTC_c6a62.dir/link.txt --verbose=1
/nix/store/x9hgvxa06b03n3j282dpj5bv68hz5mws-clang-wrapper-12.0.0/bin/clang -rdynamic CMakeFiles/cmTC_c6a62.dir/testCCompiler.c.o -o cmTC_c6a62
clang-12: error: invalid linker name in argument '-fuse-ld=lld'
make[1]: *** [CMakeFiles/cmTC_c6a62.dir/build.make:106: cmTC_c6a62] Error 1
make[1]: Leaving directory '/home/jacek/src/cxx-cmake-example/build/CMakeFiles/CMakeTmp'
make: *** [Makefile:140: cmTC_c6a62/fast] Error 2
CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
CMakeLists.txt:5 (project)
If this to be used in a project which depends on a Nix-provided C++ library, this library will have to be recompiled with clang, if the LTO is to work. How would one express this in a shell.nix?
Of course, there is no point in doing this unless the remaining problems are resolved, which I summarize here:
On 20.09, the LTO-enabled example compiles and runs, but the optimization fails to speed up the Rust version to be on a par with the C++ version, which is the whole point of the LTO exercise.
On unstable, where lldClang has been replaced with clangUseLLVM, regardless of whether LTO is ON or OFF, the linker is not found:
clang-12: error: invalid linker name in argument '-fuse-ld=lld'
I did a few more tests on current nixos and replicated the same on debian buster with clang-11 / lld-11 from backports and rustc 1.51 from rustup:
First off: We can get lld into scope with llvmPkgs.bintools. With that, the unstable channel works as well.
Now if I compare 2 builds of the demo project with opposite settings for ENABLE_LTO I get a slight discrepancy:
# With LTO
$ hyperfine build/main hyperfine nix-shell
Benchmark #1: build/main
Time (mean ± σ): 3.4 ms ± 0.7 ms [User: 2.8 ms, System: 0.7 ms]
Range (min … max): 2.3 ms … 5.2 ms 390 runs
# Without LTO
$ hyperfine off/main hyperfine nix-shell
Benchmark #1: off/main
Time (mean ± σ): 3.9 ms ± 0.6 ms [User: 3.0 ms, System: 0.9 ms]
Range (min … max): 2.8 ms … 6.5 ms 358 runs
But as you observed, the run-time of the rust codepath is much higher in both cases.
I don’t know how to verify that optimizations are actually happening at link time, but I figured inspecting the object files might give a hint. I extracted release/librust_part.a which contains a lot of code, presumably dependencies of cxx. Interestingly most of those contain executable code, only rust_part and cxx contain actuall LLVM IR.
I suspect that either the demo project is incorrectly set up or cross-language LTO isn’t able to inline everything at the moment.
I stopped at that point but I hope it’ll help you to make further progress.