[Solved] Using Nix on CentOS 8; Problems with outdated glibc

Help
1

I have been maintaining installation scripts for several bioinformatic programs using nix. Nix works beautifully on the latest version of Archlinux and other Linux distributions.

Now, I am trying to use nix to compile programs in an HPC environment with CentOS 8. Obviously, I have no root privilege, and I get around it with nix-user-chroot. Nix used to work beautifully on CentOS 7 and 8, but as Nix gets updated, the glibc and libstdc++ on older Linux distributions are now causing compilation issues.

My first attempt to install nix on CentOS 8 was thwarted by a glibc and libstdc++ issue.
I tried installing updated versions of glibc and libstdc++ and running the nix install script, but then the system’s bash, dirname, and mkdir, etc. were not compatible with the version of glibc and libstdc++ that the nix binary required.

So, I obtained the static binary of nix from Hydra - nix:master:buildStatic.x86_64-linux and I replaced the standard version of the nix binary with this static binary. I then set up symlinks from nix-env, nix-channel to the static nix binary. I managed to run the nix install script.
nix-channel complained about missing nix-env at a specific /nix/store location, which was solved with a few symlinks.

I even managed to compile a program using nix, and I thought things were looking really good…

$ nix-env -i fastp

Then, I tried to run the program compiled with nix:

$ fastp
fastp: /usr/lib64/libc.so.6: version `GLIBC_2.33' not found (required by fastp)
fastp: /usr/lib64/libc.so.6: version `GLIBC_2.34' not found (required by fastp)

This was the same glibc linkage error that I started with when the nix installation script tried to run nix-env during installation…

Sure, I can set up my LD_LIBRARY_PATH to point to the updated versions of glibc, but I’d run into the same issue as before where my system binaries depend on older versions of glibc whereas programs compiled with nix depend on new versions of glibc.

So, can anyone suggest a viable way to use nix on a Linux distribution (e.g. CentOS 8) that relies on outdated versions of glibc and libstdc++?

2

Nix packages should be using a glibc from the nix store, but your output shows the package is trying to use the system’s glibc. Is this intentional and something you configured?

1 Like
3

The way nix is designed, packages installed through nix are supposed to avoid interacting with the system’s libraries and even ld.so. It shouldn’t matter what distro you’re on, or if there’s even a viable gnu base system available outside the nix store at all. If the outside system is interfering, you should be trying to figure out how to cut off nix-installed programs from it. Try running them with LD_LIBRARY_PATH=""? Perhaps try to find out if this distro is doing something to avoid binaries specifying a non-standard ELF interpreter?

1 Like
4

I didn’t configure nix to use the system’s glibc. In fact, I don’t have a custom nix configuration file.

The nix binary that is installed by the nix installation script also depends on the system’s glibc:

$ sh <(curl -L https://nixos.org/nix/install) --no-daemon
5

This is strange… clearing LD_LIBRARY_PATH actually worked:

LD_LIBRARY_PATH='' fastp

I usually don’t have system libraries in my LD_LIBRARY_PATH, but my account on the HPC server is set up with system library paths added to LD_LIBRARY_PATH:

$ echo $LD_LIBRARY_PATH
/usr/lib64:/usr/lib:/usr/lib64:/usr/lib:/usr/lib64:/usr/lib

It looks like system paths in LD_LIBRARY_PATH took precedence over the the /nix/store paths…

$ ldd $(which fastp)
/home/dshih/.nix-profile/bin/fastp: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /home/dshih/.nix-profile/bin/fastp)
/home/dshih/.nix-profile/bin/fastp: /usr/lib64/libc.so.6: version `GLIBC_2.33' not found (required by /home/dshih/.nix-profile/bin/fastp)
/home/dshih/.nix-profile/bin/fastp: /usr/lib64/libc.so.6: version `GLIBC_2.34' not found (required by /home/dshih/.nix-profile/bin/fastp)
        linux-vdso.so.1 (0x00007ffffb3f7000)
        libisal.so.2 => /nix/store/g8hl1xqih8dyf81459080yx2k7s2kxmh-isa-l-2.30.0/lib/libisal.so.2 (0x00007fc5a8690000)
        libdeflate.so.0 => /nix/store/wm8j5lpylysqy97pd6f918bvr8qkrrk4-libdeflate-1.8/lib/libdeflate.so.0 (0x00007fc5a8681000)
        libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007fc5a8294000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fc5a7eff000)
        libm.so.6 => /usr/lib64/libm.so.6 (0x00007fc5a7b7d000)
        libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007fc5a7965000)
        libc.so.6 => /usr/lib64/libc.so.6 (0x00007fc5a75a0000)
        /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc5a84b4000)

$ LD_LIBRARY_PATH='' ldd $(which fastp)
        linux-vdso.so.1 (0x00007ffdad3a7000)
        libisal.so.2 => /nix/store/g8hl1xqih8dyf81459080yx2k7s2kxmh-isa-l-2.30.0/lib/libisal.so.2 (0x00007fe9f526b000)
        libdeflate.so.0 => /nix/store/wm8j5lpylysqy97pd6f918bvr8qkrrk4-libdeflate-1.8/lib/libdeflate.so.0 (0x00007fe9f525c000)
        libpthread.so.0 => /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/libpthread.so.0 (0x00007fe9f5257000)
        libstdc++.so.6 => /nix/store/wprxx5zkkk13hpj6k1v6qadjylh3vq9m-gcc-11.3.0-lib/lib/libstdc++.so.6 (0x00007fe9f4e79000)
        libm.so.6 => /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/libm.so.6 (0x00007fe9f5175000)
        libgcc_s.so.1 => /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/libgcc_s.so.1 (0x00007fe9f515b000)
        libc.so.6 => /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/libc.so.6 (0x00007fe9f4c70000)
        /nix/store/ayfr5l52xkqqjn3n4h9jfacgnchz1z7s-glibc-2.35-224/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fe9f508f000)

Thanks! I guess I’ll clear LD_LIBRARY_PATH going forward.

6

Alright. Problem solved.

Library paths in LD_LIBRARY_PATH take precedence over paths in /nix/store.

So, if your LD_LIBRARY_PATH contains system library paths, you need to unset LD_LIBRARY_PATH before installing nix and also when you run nix or any program compiled by nix.

If you don’t need system library paths in LD_LIBRARY_PATH, the easiest solution would be to strip the system library paths from LD_LIBRARY_PATH in the .bashrc

7

(FYI, Discourse allows the thread creator to mark a post as the “solution”, which also marks the thread as a whole as solved.)