Runtime alternative to `patchelf --set-interpreter`

Env variable $LD_LIBRARY_PATH is a runtime alternative to patchelf --set-rpath.
Is there something similar for pathelf --set-interpreter ?

The case is executables inside jar files downloaded from Maven.
For example https://github.com/os72/protoc-jar has executables with interpreter set to /lib64/ld-linux-x86-64.so.2 so they fail to run on NixOS.
patchelfing would be weird here, in that case the files under $HOME/.m2/ will have nix store paths, which could be garbage collected on the next upgrade.
I’d rather patch kernel or glibc to make such executables work than would make a cron job updating executables inside jars to the current valid interpreter.

Upstream ticket is WONTFIX: https://github.com/os72/protoc-jar/issues/26

It was tempting to use statifier to fix executables inside jars (I am mostly using a custom Maven resolver/downloader, so there is a single point for such a postprocessing), but unfortunately statifier fails to process those binaries.

Also ld-linux-x86-64.so.2 does not appear under /run/current-system so there is no “stable” path like /run/current-system/sw/lib64/ld-linux-x86-64.so.2 which could substitute /lib64/ld-linux-x86-64.so.2 :frowning:

Turning the question the other way around, why isn’t NixOS more compatible with the other Linux distributions?

I believe the answer is mostly historical. NixOS used to not have sandboxed builds and so the best way to catch missing build dependencies was to have them on custom paths.

Having to patchelf every executable makes it quite difficult to use NixOS when downloading custom binaries from the internet, npm, rubygems, python wheels. It adds a lot of friction to new users.

So I started playing with the idea of making NixOS compatible with the other distros. I haven’t gotten very far for now, this is how my nixos module looks like:

# Make NixOS compatible
#
# TODO: symlink /lib64/ld-linux-x86-64.so.2 to
# /run/current-system/sw/lib/ld-linux-x86_64.so.2
#
# TODO: symlink /usr/lib to /run/current-system/sw/lib
#
# TODO: symlink /bin/bash to /run/current-system/sw/bin/bash
{ pkgs, lib, ... }:
{
  environment.systemPackages = [
    pkgs.glibc.out

    # Common /usr/lib dependencies
    pkgs.zlib.out
  ];

  # FIXME: something else is overriding that setting
  # error: The unique option `environment.sessionVariables.LD_LIBRARY_PATH' is defined multiple times,
  environment.sessionVariables.LD_LIBRARY_PATH =
    lib.mkForce [
      "/run/opengl-driver/lib"
      "/run/current-system/sw/lib"
    ];
}

I believe the answer is mostly historical. NixOS used to not have sandboxed builds and so the best way to catch missing build dependencies was to have them on custom paths.

Not having to care if some programs need different versions of the same library is also useful in runtime

Having to patchelf every executable makes it quite difficult to use NixOS when downloading custom binaries from the internet, npm, rubygems, python wheels. It adds a lot of friction to new users.

So basically you want the default environment to be a FHSUserEnv ?

     system.activationScripts.ldso = lib.stringAfter [ "usrbinenv" ] ''                       
       mkdir -m 0755 -p /lib64                                                                
       ln -sfn ${pkgs.glibc.out}/lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2.tmp   
       mv -f /lib64/ld-linux-x86-64.so.2.tmp /lib64/ld-linux-x86-64.so.2 # atomically replace 
     '';                                                                                      

does the trick, making /lib64/ld-linux-x86-64.so.2 the third FHS exception after /bin/sh and /usr/bin/env
Currently I see no better solution

1 Like

Another tool that might be useful in that space: autopatchelf <bin_path> wrapper script that would lookup all the missing .so files in the nix index and patch the binary automagically. It would probably work 99% of the time.

Then use the boot.binfmt facility to replace all the binary executions to first go through autopatchelf :-p

2 Likes

If that facility is flexible enough, it could detect the need for /lib64/ld-linux-x86-64.so.2 in the elf and only pass those through the wrapper.

It would be glorious :smiley:

  • keep a cache of rewritten binaries based on the executable hash
  • forward all the rewrites to a journald log for inspection

The only issue I think is that it’s a kernel-level setting so it would also apply inside of the nix builders. Which we definitely don’t want.

If it’s not turned on by default, I wouldn’t see much problems; though having binfmt tweakable in the sandbox is probably a good idea, considering otherwise it’s an impurity.

Example of such executables with /lib64/ld-linux-x86-64.so.2 interpreter:
Outside jar http://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/1.9.0/
Inside jar https://repo1.maven.org/maven2/com/github/os72/protoc-jar/3.8.0/

Although their rpath is empty, they somehow manage to find libpthread.so.0, libm.so.6 and libc.so.6 in nix store (I do not have systemwide $LD_LIBRARY_PATH as @zimbatm and set | grep glibc returns no result)

I’m not sure if it applies to your scenario, but here is an example that uses $NIX_CC/nix-support/dynamic-linker to find the linker.

https://bitbucket.org/geoscienceaustralia/geodesy-archive/src/master/nix/gfzrnx/default.nix

NIX_CC is also set in nix-shell.

No, it is not relevant to Nix build process.
It is about how to run on NixOS unmodified Linux binaries when is it difficult to patch them, when they are distributed via Maven or another weird way.

I’ve had the hunch for a while that nix starts breaking down when having to deal with any sort of “packed” binaries (e.g. dependencies are calculated by looking at what hashes can be found in a result). I think it would be good to improve compatibility in ways that don’t involve modifying executables.

Though I guess the fallback continues to be FHSUserEnv or using a VM. Better seamless integration with FHSUserEnv might also be interesting?

Sorry for the useless reply :sweat_smile:

Might it make sense to have nix fhs-run in addition to nix run, that
would make a FHSUserEnv of the given derivations and run in this shell?

I’m kind of unsure of which nesting behavior we should be expecting from
such a command, but…

2 Likes

I expect a more general command like nix container that allow users to run commands in an ad-hoc container.

2 Likes

I expect a more general command like nix container that allow users to run commands in an ad-hoc container.

The nix container idea creates a large risk of creating a systemd-only container interface…

1 Like

Are you sure that ld-linux isn’t looked up in LD_LIBRARY_PATH? I thought that was the case

Are you sure that ld-linux isn’t looked up in LD_LIBRARY_PATH?

Dynamic interpreter in the ELF header is an absolute path.

I made a PR with an option to enable ld-linux at its usual FHS location

2 Likes

You can run executables with your interpreter of choice by passing the executable as an argument to ld-linux. Like this:

./my-ld-linux.so.2 path/to/exe ...

This means you’d have to patch the source of the calling library instead of the executable so it might not be a better solution… But this is the runtime alternative to using --set-interpreter.