Rust/C++ link-time-optimization

jacg · May 26, 2021, 10:10am

While exploring the possibility of incorporating some Rust in a cmake-managed C++ project, I am trying to follow this template. It works like a charm with -DENABLE_LTO=OFF (cmake, make and ./main all succeed), but I’m not having any luck with -DENABLE_LTO=ON, always getting

-- IPO / LTO not supported: <Change Dir: /tmp/cxx-cmake-example/build/CMakeFiles/_CMakeLTOTest-CXX/bin

at the cmake step

This issue still being open, look ominous. Variations on this theme haven’t born any fruit.

Can you offer any advice on how to get LTO to work in this template, with nix-provided tools (Rust/C++ compilers/linkers)?

tobim · May 26, 2021, 10:57am

This doesn’t work because LTO transfers the grunt work of the compilation to the linker. I’m not exactly sure if LLD invokes the compiler again for that or applies the passes in process, but nevertheless LLD doesn’t know how to deal with the LTO intermediate format of your compiler, which seems to be GCC.

To make it work you can create a shell with clang as the default compiler with this:

 $ cat shell.nix                                                                                                                                                     nix-shell
{ pkgs ? import <nixpkgs> {} }:
let
  llvmPkgs = pkgs.buildPackages.llvmPackages_11;
  mkShell = pkgs.mkShell.override { stdenv = llvmPkgs.stdenv; };
in mkShell {
  buildInputs = [
    llvmPkgs.lldClang.bintools
  ];
  hardeningDisable = [ "fortify" ];
  NIX_CFLAGS_LINK = "-fuse-ld=lld";
}

Note that you also have to modify the RUST_FLAGS in your example project:

--- a/rust_part/CMakeLists.txt
+++ b/rust_part/CMakeLists.txt
@@ -7,7 +7,7 @@ else ()
 endif ()

 if(ENABLE_LTO)
-    set(RUST_FLAGS "-Clinker-plugin-lto" "-Clinker=clang-11" "-Clink-arg=-fuse-ld=lld-11")
+    set(RUST_FLAGS "-Clinker-plugin-lto" "-Clinker=clang" "-Clink-arg=-fuse-ld=lld")
 endif()

 set(RUST_PART_LIB "${CMAKE_CURRENT_BINARY_DIR}/${TARGET_DIR}/librust_part.a")

jacg · May 26, 2021, 11:52am

lldClang seems not to exist (neither at that location, nor anywhere in the nixpkgs source (using master from sometime earlier today)).

I’ve tried poking around …

llvmPkgs.lld does exist, but does not contain bintools
llvmPkgs.bintools does exist
etc.

but haven’t found anything that works.

jacg · May 26, 2021, 12:24pm

Dodging the issue of what has happened to lldClang in unstable, I pinned nixpkgs to 20.09 of today. This allowed me to make more progress, with the cmake step now reporting

-- IPO / LTO enabled

However, the make step failed with (buried among lots of noise)

(Producer: 'LLVM12.0.0-rust-1.52.1-stable' Reader: 'LLVM 11.0.0')

Replacing llvmPackages_11 with llvmPackages_12 led to a successful make. Unfortunately ./main reports

Calling rust function, time elapsed: 4477595 ns.
Calling c++ function, time elapsed: 90 ns.

that is to say, the rust version is orders of magnitude slower than the C++ version, which is expected without LTO, but with LTO they should be comparable.

In other words, even though it compiles, the optimization appears not to work.

tobim · May 26, 2021, 12:25pm

My apologies, it seems lldClang has been removed in 18c38f8aee732b6202042383fc35997a39361830. Can you try with clangUseLLVM instead?

jacg · May 26, 2021, 12:52pm

Sure …

Starting with the version that compiled an ran, but where the LTO seems to have no effect on the speed of the Rust version; changing the pinned nixpkgs version to unstable, and making the change you suggested:

-  commit-id = "76ed24ceab9ec8b520f977a2803181f0c1d86b4d";
+  commit-id = "ea7d4aa9b8225abd6147339f0d56675d6f1f0fd1";
 
-    llvmPkgs.lldClang.bintools
+    llvmPkgs.clangUseLLVM.bintools

(in case it’s any use, I’ve pushed the whole lot here)

it now fails at the cmake stage with:

cmake -DENABLE_LTO=ON -DCMAKE_BUILD_TYPE=Release .. 
-- The C compiler identification is Clang 12.0.0
-- The CXX compiler identification is Clang 12.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /nix/store/x9hgvxa06b03n3j282dpj5bv68hz5mws-clang-wrapper-12.0.0/bin/clang
-- Check for working C compiler: /nix/store/x9hgvxa06b03n3j282dpj5bv68hz5mws-clang-wrapper-12.0.0/bin/clang - broken
CMake Error at /nix/store/zzqzl5lm59mc04fd8f1wbwl7nmxvv1bb-cmake-3.19.7/share/cmake-3.19/Modules/CMakeTestCCompiler.cmake:66 (message):
  The C compiler

    "/nix/store/x9hgvxa06b03n3j282dpj5bv68hz5mws-clang-wrapper-12.0.0/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/jacek/src/cxx-cmake-example/build/CMakeFiles/CMakeTmp
    
    Run Build Command(s):/nix/store/g2fna66r9m081w1h1zj857j06jigx6cq-gnumake-4.3/bin/make cmTC_c6a62/fast && /nix/store/g2fna66r9m081w1h1zj857j06jigx6cq-gnumake-4.3/bin/make  -f CMakeFiles/cmTC_c6a62.dir/build.make CMakeFiles/cmTC_c6a62.dir/build
    make[1]: Entering directory '/home/jacek/src/cxx-cmake-example/build/CMakeFiles/CMakeTmp'
    Building C object CMakeFiles/cmTC_c6a62.dir/testCCompiler.c.o
    /nix/store/x9hgvxa06b03n3j282dpj5bv68hz5mws-clang-wrapper-12.0.0/bin/clang    -o CMakeFiles/cmTC_c6a62.dir/testCCompiler.c.o -c /home/jacek/src/cxx-cmake-example/build/CMakeFiles/CMakeTmp/testCCompiler.c
    Linking C executable cmTC_c6a62
    /nix/store/zzqzl5lm59mc04fd8f1wbwl7nmxvv1bb-cmake-3.19.7/bin/cmake -E cmake_link_script CMakeFiles/cmTC_c6a62.dir/link.txt --verbose=1
    /nix/store/x9hgvxa06b03n3j282dpj5bv68hz5mws-clang-wrapper-12.0.0/bin/clang -rdynamic CMakeFiles/cmTC_c6a62.dir/testCCompiler.c.o -o cmTC_c6a62 
    clang-12: error: invalid linker name in argument '-fuse-ld=lld'
    make[1]: *** [CMakeFiles/cmTC_c6a62.dir/build.make:106: cmTC_c6a62] Error 1
    make[1]: Leaving directory '/home/jacek/src/cxx-cmake-example/build/CMakeFiles/CMakeTmp'
    make: *** [Makefile:140: cmTC_c6a62/fast] Error 2
    
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:5 (project)

jacg · May 27, 2021, 8:21am

If this to be used in a project which depends on a Nix-provided C++ library, this library will have to be recompiled with clang, if the LTO is to work. How would one express this in a shell.nix?

Of course, there is no point in doing this unless the remaining problems are resolved, which I summarize here:

On 20.09, the LTO-enabled example compiles and runs, but the optimization fails to speed up the Rust version to be on a par with the C++ version, which is the whole point of the LTO exercise.
On unstable, where lldClang has been replaced with clangUseLLVM, regardless of whether LTO is ON or OFF, the linker is not found:
```
clang-12: error: invalid linker name in argument '-fuse-ld=lld'
```

I have no idea how to tackle these problems.

tobim · May 30, 2021, 10:31pm

I did a few more tests on current nixos and replicated the same on debian buster with clang-11 / lld-11 from backports and rustc 1.51 from rustup:

First off: We can get lld into scope with llvmPkgs.bintools. With that, the unstable channel works as well.
Now if I compare 2 builds of the demo project with opposite settings for ENABLE_LTO I get a slight discrepancy:

# With LTO
$ hyperfine build/main                       hyperfine nix-shell
Benchmark #1: build/main
  Time (mean ± σ):       3.4 ms ±   0.7 ms    [User: 2.8 ms, System: 0.7 ms]
  Range (min … max):     2.3 ms …   5.2 ms    390 runs

# Without LTO
$ hyperfine off/main                         hyperfine nix-shell
Benchmark #1: off/main
  Time (mean ± σ):       3.9 ms ±   0.6 ms    [User: 3.0 ms, System: 0.9 ms]
  Range (min … max):     2.8 ms …   6.5 ms    358 runs

But as you observed, the run-time of the rust codepath is much higher in both cases.

I don’t know how to verify that optimizations are actually happening at link time, but I figured inspecting the object files might give a hint. I extracted release/librust_part.a which contains a lot of code, presumably dependencies of cxx. Interestingly most of those contain executable code, only rust_part and cxx contain actuall LLVM IR.

I suspect that either the demo project is incorrectly set up or cross-language LTO isn’t able to inline everything at the moment.

I stopped at that point but I hope it’ll help you to make further progress.

zyansheep · July 10, 2021, 3:50pm

(in case it’s any use, I’ve pushed the whole lot here )

Woh, thanks for posting that, I’ve been trying to get lld to work after lldClang was removed and this configuration worked!
(My config)