How to **not** use libs from nix

I have a machine with Ubuntu 22.04 and I am using Nix (not NixOS) in conjunction with home-manager to easily install tools.

When trying to build a rust project from source, I have the following errors:

error: /nix/store/90illc73xfs933d06daq6d41njs8yh66-glibc-2.32-37/lib/libc.so.6: version `GLIBC_2.33' not found (required by /projects/kbct/target/debug/deps/libproc_macro_error_attr-0723a4c084984c9e.so)
   --> /home/oliv/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-error-1.0.4/src/lib.rs:284:9
    |
284 | pub use proc_macro_error_attr::proc_macro_error;
    |         

From various searches, I understand that this stems from mix os libs and nix libs.
Particularly, this github issue recommends not to use nix to install compilers, and advice I plan to use not to bother too much.

However:

  • even with rustup, rustc, cargo, etc installed locally,
  • without anything set for LD_LIBRARY_PATH or this var set explicitly to /usr/lib:/usr/lib32:/usr/lib64
    I still see the libc installed by nix being used to compile.

Not being very versed into compiler options, I seek for help to properly configure my env not to use these libs to build my project.

Thanks very much to all that will answer

Some notes:

  • I have read about patchself to fix hard-coded lib paths in .so, but these files being built by cargo, I am not sure I can automate the fix after every build.
1 Like

How did you install your toolchain, and what is the content of LD_PRELOAD ?

Rust toolchain has been installed using rustup, following rustup.rs procedure. That is curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh (at the time of the writing).

Currently, either in fishshell and bash (mostly using fishshell), LD_PRELOAD is empty.

However, even if I am not expert in what ldd does, seeing it regularly used in these questions, I tested it upon rustc installed by rustup and I see:

❯ ldd rustc 
	linux-vdso.so.1 (0x00007ffcd43be000)
	librustc_driver-2c37baf9e5f6ef22.so => /home/oliv/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/./../lib/librustc_driver-2c37baf9e5f6ef22.so (0x00007f145bc00000)
	libstd-05b39ac0cb4c5688.so => /home/oliv/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/./../lib/libstd-05b39ac0cb4c5688.so (0x00007f145b800000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f14600ec000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f14600e7000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f14600e2000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f145b5d8000)
	libLLVM-14-rust-1.61.0-stable.so => /home/oliv/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/./../lib/../lib/libLLVM-14-rust-1.61.0-stable.so (0x00007f1456078000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f14600c0000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f145fb19000)
	/nix/store/90illc73xfs933d06daq6d41njs8yh66-glibc-2.32-37/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f146010a000)
	libz.so.1 => /nix/store/6hhqb94ilrdmh0cx5bhdqb92m6bqkvdj-zlib-1.2.11/lib/libz.so.1 (0x00007f14600a1000)

I can see that both libz and libm are coming from nix, while the ohers are local or from rustup.
Conversely, ld-linux-x86-64.so.2 points to my os version.

I apologize because I haven’t enough knowledge of these, but could it be the source of the issue?

did you install gcc, stdenv, clang or binutils with nix (either nix-env or home-manager) ? What does which cc ld rustc cargo report ?

Also did you install nixpkgs’s rustup in the past, then uninstalled it, and installed the official rustup afterwards without nuking .rustup in-between ?

1 Like

It may be excessive but dear @symphorien, you are a life-saver.
Indeed, I did uninstall nix rustup without nuking .rustup. Performing this cleaning and re-installing the toolchain solved the issues.
I managed to compile my project without any error this time :tada:

So, once more, thanks for your prompt answers today and for your previous helpful answers in various threads.

Before closing this discussion, if I may ask, is there something specific to configure how libraries are used (as it was the OP). Or, because some are hard-coded into binaries, should one just wind up tools until a clean state is restored?

1 Like

An executable hardcodes the so-called ELF interpreter it needs: nixpkgs-provided stuff uses a nix-provided interpreter (part of glibc) and “normal” executables hardcode /lib/ld-linux.so.2. Optionally, executables can also hardcode where libraries they use should be looked for with RUNPATH or RPATH. This is the case of all nixpkgs-provided executables, but is more rare for “normal” executables. When the RUNPATH or RPATH are not set, libraries are looked in the environment variable LD_LIBRARY_PATH or in system-configured path like /usr/lib. So LD_LIBRARY_PATH can somewhat influence where libraries are picked, but not fully, and it cannot change what ELF interpreter is used.

The issue here is that nixpkgs’s rustup is patched to use patchelf to point executables downloaded by rustup to some nixpkgs-provided ELF interpreter and patch their RPATH to some working libs. This is required to make rustup work on NixOS where the default ELF interpreter does not exist. As these changes are persistent (because they modify the executable) removing rustup and reinstalling another that does not have this behavior is not enough.

1 Like

I’m glad to find the solution to this here! Arguably, rustup installed via Nix should alter the location of ~/.rustup so as not to run into these issues.

1 Like

I wonder how often tools/users hardcode the path to the compiler in ~/.rustup

That is one solution, but honestly I prefer a project specific devshell which pulls in the required Rust version, either just from nixpkgs or from fenix. That way the rust version built by Nix for the package, and the rust version used in development are identical.

1 Like