According to the ld.so manual, the dynamic linker should look for libraries in
- rpath
LD_LIBRARY_PATH- runpath
- cache
- defaults (/usr/lib)
However for some reason, the dynamic linker on nix doesn’t seem to look in the default directories. Because of that, buildFHSUserEnv sets LD_LIBRARY_PATH to include /usr/lib. But that results in a different behaviour, because now /usr/lib is considered first instead of last (rpath is deprecated).
Why doesn’t the dynamic linker look in the default locations on nixos? Is there some way to get around that behaviour?
1 Like
This is the code that does it:
Perhaps we can get rid of it? It may obsolete now that sandboxing is available.
1 Like
Isn’t sandboxing optional? Especially on macOS where it doesn’t work well.
Thanks, that is very helpful! So at least currently, there is no way to get the default ld behaviour in a FHS environment without rebuilding the world 
Sandboxing only solves the issue at build time though right? I’m not sure if we want it solved at runtime too.
An alternative solution for the FHS environments would be to make up some new system dir (NATIVE_LIB_DIRS=/idontcareaboutimpurities-lib) and then symlink /usr/lib to that location in the builder for the FHS environment. That should emulate default behaviour without a realistic possibility of causing impurity on a normal system.
I stumbled upon this a while back trying to make FHS shells more ergonomic and was surprised to see that removing LD_LIBRARY_PATH worked (as I was too under the impression that ld didn’t look for things in /lib etc.). Not quite sure what’s up, but in particular conda-shell works at least, and eg. the local python it installs resolve libraries correctly.
Probably the relevant binaries have /lib or other relevant directories in their runpath? You can check with readelf -d <binary>.
Otherwise those dirs should not be checked because of the bit of code @matthewbauer linked.
Nope, already checked that, it’s a bit strange.
> readelf --dynamic .conda/bin/python
0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]
0x0000000000000001 (NEEDED) Shared library: [libutil.so.1]
0x0000000000000001 (NEEDED) Shared library: [librt.so.1]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x000000000000000f (RPATH) Library rpath: [$ORIGIN/../lib]
...
While ldd .conda/bin/python3.6 shows correctly resolved libraries (eg. libpthread) to the nix store (which aren’t linked in ~/.conda/lib).
So yeah, I’m a bit confused 
Do you have a test case where things break without LD_LIBRARY_PATH.
The binutils dynamic linker is just used on Linux. But that’s definitely a concern for unsandboxed Linux. My argument would be that since this is just a fallback, it doesn’t really effect impurity. It will be looked at last after all of the other paths. It just helps us in cases like this where you want the linker to look in /usr/lib.
My test case is the sage build system, but that takes a while to build. Are you using linux? Does ldd show any libraries resolved to /lib? If so, in which order was /lib checked (export LD_DEBUG=all)?
1 Like
I could only see this become an issue when people test packages (after build, so not in sandbox) and they work because of system impurities.
Cheers, didn’t know about LD_DEBUG_ALL. Looks like glibc stuff is resolved through the cache or system search path:
8020: search cache=/nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/etc/ld.so.cache
8020: search path=/nix/store/681354n3k44r8z90m35hm8945vsp95h1-glibc-2.27/lib (system search path)
It does indeed not find stuff from /lib. Thanks for clearing up my confusion 