Can $ORIGIN be used to make Nix prebuilt binaries relocatable?

Hi there,

According to my understanding, the reason Nix prebuilt binaries have to be installed in ‘/nix/store’ is because of the use of absolute paths in RPATH. I think $ORIGIN can be used to turn all those absolute paths into relative paths. Since this is really obvious, I assume it has been considered by the Nix developers already. But I can’t think of reasons to not use $ORIGIN, so I’m curious what’s the reasoning behind using absolute paths instead of $ORIGIN-based relative paths. Any thoughts?

It wouldn’t just be RPATHs, it would also be any kind of embedded reference to a path. For example, if I have a binary that invokes another binary via absolute path, I can’t really change that to be relative to the binary’s install location.

You are right that RPATH is not the only problem. But I think with RPATH fixed, a lot of packages will become relocatable, and that will be helpful to users who for one reason or another can’t install Nix to /nix/store.

Of course, if something as fundamental as glibc can not be made relocatable, then by transitivity pretty much no other packages can be made relocatable, whether $ORIGIN is used or not. But glibc is pretty much self-contained, so it shouldn’t be too hard to make it relocatable.

This is what Bazel does and definitely can work. But, there are significant downsides to it. For instance-

  • Cannot move binaries after they’ve been linked. Since everything is relative you have to move the whole tree along with it. Things like multiple outputs would not work correctly.
  • There is no way to specify to build systems that $ORIGIN linking should be used. Nixpkgs tries to defer as much as possible to the package’s build system. These usually expect a hardcoded path in the form of --prefix and similar. No build system that I’m aware of cares about “$ORIGIN”.
  • $ORIGIN does not work on macOS or probably any other system but Linux (`$ORIGIN` in RPATH on macOS · Issue #4480 · bazelbuild/bazel · GitHub). It sounds like there are close parallels in macOS, but there is no certainty the behavior will be the same. With absolute paths, we are pretty sure this works on any Unix system.
  • Larger rpaths can reduce reproducibility. Each rpath is looked up by the linker in order, so having lots of them like Bazel does can harm reproducible execution. If the library exists in one rpath for one system but not another, we may have different versions of libraries being used. With Nix’s absolute hash+name scheme, we can be certain that all binaries use the same libraries, and those same libraries are included in the closure.
  • Adding on to that, the fundamental difference here is how Nix’s idea of “closure” vs. Bazel’s idea of “runfiles”. Nix’s closures are automatically managed, while runfiles are manually managed. This benefits Nix users by not making them care about what is used at runtime, only what absolute path references are transitively included matter. On the other hand, you end up relying on the absolute Nix store path. Nix cares about both build time and run time while Bazel only really care about build time.
1 Like

Don’t think this is too bad. We already have different code for Darwin and Linux . Currently Darwin and Linux load libraries differently already in Nixpkgs. For Linux the lookups of libraries is quadratic whilst for Darwin it isn’t, iirc.

For large package sets (like nixpkgs) this isn’t worth it, IMO. Packages embed lots of absolute paths detected during build, even to various data files (not just executables). On Linux you can still use e.g. mount namespaces to create a virtual FS layout, often even without having elevated privileges.

1 Like