A double edge sword: LD_LIBRARY_PATH no longer works.
But LD_PRELOAD still does. 
There should be a better gesture if you want to override a library rather than using LD_LIBRARY_PATH in Nix
After thinking about it for a while I realized that Nix does not do “dynamic linking” but “deferred linking”. Obviously we should take a cautious approach to changing the link process but at the same time we should recognize we are doing something fundamentally different.
Exactly!
In fact I proposed in our paper submission the thought experiment:
Imagine if the dynamic loader validated the content address so that you could not replace the link with another shared object file.
It really starts to get blurry what is static linking and dynamic linking from a theoretical perspective.
Dynamic linking is supposed to let you swap ABI compatible pieces but what happens when you disallow that but you still load shared objects.
Guix implemented a solution to this problem in their package repository last year, relying on dynamic loader caches instead: Taming the ‘stat’ storm with a loader cache — 2021 — Blog — GNU Guix. It’d be interesting to compare notes and if they considered your solution before implementing it.
In a way their solution is more conservative (which is good for such large impact changes), but it seems that the loader cache is a specific feature of the glibc dynamic loader (could be wrong here). Though, we probably need to pay attention to the loader implementation always: Did you test your solution with musl’s loader? Does it translate to macOS’ dylibs?
I recommend you read my previous post that has this information as well:
Seems to me that you only link the Guix article as an “alternative approach”, but don’t discuss the respective advantages or drawbacks which would interest me.
Also, musl-support is crucial for nixpkgs arguably, having an approach that supports the musl dynamic linker would be nice.
This seems like a good place to link OP’s recent talk on this and the subsequent Q&A for further discussion! I had some questions about the talk, which I’ll hopefully return to here soon. 
Thanks for the link!
Ok, here are a couple of questions:
-
What are the main applications of extending this approach to create a new executable format fhat excite you?
-
Can this ‘stamping’ approach be extended to Darwin? (I assume yes)
-
Could we use something like this to make (most) Nix packages relocatable without container-y hacks on the whole Nix store? Could a new executable format relax some of the current constraints on path rewriting in Nix?
-
Introspection and debugging. The current ELF format is incredibly terse and challenging to make changes.
I imagine for instance, a much simpler way to make edits to symbol resolution that would maintain provenance and do away with LD_PRELOAD for systems such as Nix. -
Yes! Spack just merged this into their upstream for support for all binaries.
-
There is also support for $ORIGIN that we should use rather than absolute paths from /nix/store
using $ORIGIN and relative paths make all executables relocatable anywhere on the filesystem as long as the store tree moves with it.
I believe it would also allow re-use of binaries across different store installations.
I saw @thufschmitt express some interest in solving the stat storm issue in Nix/Nixpkgs. @fzakaria, do you feel like Shrinkwrap is more or less ready for widespread use in Nixpkgs?
Is there some identifiable milestone or test set of packages we should try to hit or something?
Shrinkwrap is easy to apply individually to packages.
Including it in stdenv I think is challenging by default because at the moment it’s in Python which I don’t think is in the bootstrap. I use the awesome LIEF library and we could convert it to pure C++.
We can also apply GitHub - fzakaria/nix-harden-needed: Bubble up the correct paths to your shared object libraries in Nix which needs patchelf which is in the bootstrap.
When, if ever, is Shrinkwrap preferable to nix-harden-needed? When you just want to cheaply create a stamped binary wrapping something that’s already built and so already in all your binary caches?
Could either tool be an alternative (in some use cases— maybe not for Steam where games bring their own binaries into the picture in an ad-hoc way) to long-lived FHSUserEnvs? If so, they could bring similar functionality for packaging ‘foreign’ proprietary software with Nix to platforms that don’t have Linux’s sandboxing capabilities, right? (Instead of worrying about user-mode chrooting in a platform-native way, you worry about lifting and stamping in a way that’s native to the executable format(s) you want to use on the platform.)
Yes, I believe Shrinkwrap will set the dynamic libraries to the full path in the NEEDED. For nix-harden-needed only dependencies with the nix-harden-needed setup-hook will use the full path. In the nix-harden-needed example, the libc.so is just the filename in NEEDED and the parent directory in RUNPATH.
I’m don’t see how either could replace FHSUserEnvs. Both still require the dynamic library paths to be known at build time. It might help if there was some strange conflict between different runpaths? I think most of the use of buildFHSUserEnv outside of running foreign applications is just to avoid complexity patching some software.
I’m testing the nix-harden-needed method, I’m trying to provide a modified stdenv so that rebuilding a single package and dependencies can be done without having to override every packages buildInputs.
Shrinkwrap is done post-build time FWIW – which is the benefit and the drawback.
The simplicity of nix-harden is it “automatically” propagates through the Nixpkgs system whereas Shrinkwrap is a post-action you can apply to “stamp”.