Executables with shared libraries fail to run with err "no such file or directory"

[eljamm]

Alright, so I found an interesting project nix-alien which allows you to run unpatched binaries.

Inside the project directory, run:

$ nix --extra-experimental-features "nix-command flakes" run github:thiagokokada/nix-alien -- ./main

If that worked, the following command will show you which libs are needed:

$ nix --extra-experimental-features "nix-command flakes" run github:thiagokokada/nix-alien#nix-alien-find-libs -- ./main

I tried this on a random non-NixOS program and it worked nicely.

If after all of this and it still doesn’t run, I think this might be an issue with the project itself or something else like a permission error…

[Cequallium]

@eljamm

nix --extra-experimental-features "nix-command flakes" run github:thiagokokada/nix-alien -- ./main

oh my god, it works, tysm!
also running the second command doesnt output anything… oddly.

[eljamm]

When you run the first command, does it show you a list of libraries and asks you to choose or does it just launch the program?

[Cequallium]

no it just launches the program

*i wrote dependency instead of library

[eljamm]

Try running the next command:

$ nix --extra-experimental-features "nix-command flakes" run github:thiagokokada/nix-alien#nix-alien-ld -- ./main

It will create a cached shell in '~/.cache/nix-alien and give you its path. When you open it, you should see the libraries it loads in NIX_LD_LIBRARY_PATH:

Example

{ pkgs ? import
    (builtins.fetchTarball {
      name = "nixpkgs-unstable-20240509145238";
      url = "https://github.com/NixOS/nixpkgs/archive/f1010e0469db743d14519a1efd37e23f8513d714.tar.gz";
      sha256 = "sha256-doPgfj+7FFe9rfzWo1siAV2mVCasW+Bh8I1cToAXEE4=";
    })
    { }
}:

let
  inherit (pkgs) lib stdenv;
  NIX_LD_LIBRARY_PATH = with pkgs; lib.makeLibraryPath [
    gmp.out
    libz.out
  ];
  NIX_LD = lib.fileContents "${stdenv.cc}/nix-support/dynamic-linker";
in
pkgs.writeShellScriptBin "anki-panky" ''
  export NIX_LD_LIBRARY_PATH='${NIX_LD_LIBRARY_PATH}'${"\${NIX_LD_LIBRARY_PATH:+':'}$NIX_LD_LIBRARY_PATH"}
  export NIX_LD='${NIX_LD}'
  /home/user/Downloads/anki-panky "$@"
''

For this example, it’s libz and gmp.

[Cequallium]

nope, totally blank.

let
  inherit (pkgs) lib stdenv;
  NIX_LD_LIBRARY_PATH = with pkgs; lib.makeLibraryPath [
    
  ];
  NIX_LD = lib.fileContents "${stdenv.cc}/nix-support/dynamic-linker";
in

[eljamm]

How odd… Well, since we’ve gotten this far, we might as well see what is being loaded with LD_DEBUG=libs when it does work:

$ LD_DEBUG=libs nix --extra-experimental-features "nix-command flakes" run github:thiagokokada/nix-alien -- ./main

[Cequallium]

The log is too long to post here so i created a pastebin.

[eljamm]

Thanks, if you could do this as well, what does the following give you?

$ nix-shell -p patchelf --run "patchelf --print-needed ./main"

[Cequallium]

@eljamm

nix-shell -p patchelf --run "patchelf --print-needed ./main"

libresolv.so.2
libpthread.so.0
libc.so.6

[Cequallium]

i added glibc under nix-ld.libraries and it started working, bruh.

[eljamm]

I was gonna say, those three libraries appear in the following derivations:

$ nix-locate libc.so.6 | grep glibc
(zsnes.out)                                   2,550,384 x /nix/store/80h8wkvzf60012mnkik703ifzc1w1kyl-glibc-2.39-31/lib/libc.so.6
(hare.out)                                    2,122,232 x /nix/store/s5wxag65m62a2bg2lg82jb3nqc42wyi6-glibc-aarch64-unknown-linux-gnu-2.39-31/lib/libc.so.6
(hare.out)                                    1,759,184 x /nix/store/brnz0ga886s09gkmjd0nwhvirjwaph7w-glibc-riscv64-unknown-linux-gnu-2.39-31/lib/libc.so.6
iconv.out                                     2,285,600 x /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
glibc_multi.out                                       0 s /nix/store/s4gcddkm2kvb02qnrm2rq22lj8yynqhw-glibc-multi-2.39-31/lib/libc.so.6
glibc_memusage.debug                                  0 s /nix/store/ssai8vvm9hvhjc7i6acpma6fv66dz19a-glibc-gd-2.39-31-debug/lib/debug/libc.so.6
iconv.debug                                           0 s /nix/store/0k9qgiqxjlhh71pj7a2czz33ir3nv720-glibc-2.39-31-debug/lib/debug/libc.so.6
glibc_memusage.out                            2,285,600 x /nix/store/bimkvwhg1wrg4fmzf6aqhy68m8b12q9a-glibc-gd-2.39-31/lib/libc.so.6

I think just installing glibc_multi will just solve the issue.

[Cequallium]

Is this what you had in mind?

programs.nix-ld.libraries = with pkgs; [
 glibc_multi 
];

[eljamm]

I meant just as a normal package, if only to test, but if that doesn’t work then using nix-ld would indeed be the way to go.

[Cequallium]

Thank you, this solution works perfectly. I will now mark this as resolved.

[eljamm]

Which one solved it, installing as a package or nix-ld?

[Cequallium]

I installed it as a global package in configuration.nix
environment.systemPackages = with pkgs; [ glibc_multi ];

[eljamm]

On my system, I had it installed as a dependency for steam, that’s why the program was running normally for me.

$ nix why-depends /run/current-system nixpkgs#glibc_multi
/nix/store/4a4lz5fj7s31wc5giw6g72cis03qjhvi-nixos-system-nixos-24.05.20240509.f1010e0
└───/nix/store/lix00h8v7jl9fv5aavl9ykfxz6lagwx9-system-path
    └───/nix/store/4slxy5jsmypyi30rr5lzh6ja94kk53qz-steam-run
        └───/nix/store/v0lwlgnb1jc4mdlzzyh77zszakkyf8q7-steam-run-bwrap
            └───/nix/store/8zl8x3ijicd3shdc4lydzc5l80xlfjw4-steam-run-fhs
                └───/nix/store/5ddanrp2i7g7fvpbpzk6s93ah4vg7ycz-steam-run-usr-target
                    └───/nix/store/v6ayhasssr0wj8906zihf6lc1dnsx9ag-glibc-multi-2.39-31-bin

It would be nice if we could somehow run inside a shell/environment where no libraries are loaded by default from the system, but turns out it’s not hard to debug the libraries the program needs once you get the hang of it.

Congrats, anyways

[rgoulter]

While I’m glad it’s working,

I’m still not sure why it wasn’t working.

e.g. those libs are already in glibc:

ls /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/ | grep 'lib\(pthread\|resolv\|c\)'

and upthread, ldd seemed to be able to find them:

@Cequallium

they all seem to exist.

I notice the dynamic linker ld-linux-x86-64.so.2 resolves to a different path for some reason /nix/store/35pq4hr29c3sl79lgfwgsvd9nwzyp4am-glibc-2.39-5/lib/ld-linux-x86-64.so.2 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib64/ld-linux-x86-64.so.2.

Sure, setting LD_LIBRARY_PATH results in a core dump, but that wasn’t the observed error. Using patch-elf --set-interpreter with this /nix/store/35pq....p4am-glib-2.39-5/ didn’t result in the same error, either.

[eljamm]

@rgoulter

I notice the dynamic linker ld-linux-x86-64.so.2 resolves to a different path for some reason

The same thing happened in:

Build go code within a nix shell

We’ve been investigating since yesterday about the issue, without much luck. I made a quick reproducer here: Go in a Nix Shell · GitHub Running this code on my own NixOS laptop but also on my own Amazon Workspace (based on RedHat) works pretty fine. I don’t think the issue is related to Nix at all since the beginning. [image] However, running exactly the same steps on his Amazon Workspace (based on Ubuntu) doesn’t work: ❯ git clone https://gist.github.com/drupol/e8db5a7e22611d3aa9cd94ca5ac…

So there is definitely something wrong there. Both programs deal with a "net/http" server so I wonder how that relates.

@rgoulter

Using patch-elf --set-interpreter with this /nix/store/35pq....p4am-glib-2.39-5/ didn’t result in the same error, either.

That’s true. I couldn’t reproduce the error with this either, but perhaps this is not the main cause, but an rather an effect.

Here is my take on why it wasn’t working based on these two threads:

• wrong version of glibc gets picked as an interpreter
• this version doesn’t exist (anymore? after gc?) on the machaine
• user tries to run the binary, but interpreter is not found
• no such file or directory error

How to reproduce?

• $ nix run nixpkgs#patchelf – --set-interpreter “/hello/world” ./main
• $ ./main
• $ zsh: no such file or directory: ./main
• now, replace “/hello/world” with a glibc that no longer exists and you’ll have the same issue

Solutions?

• re-install the system → correct linker gets picked
• install another glibc → its linker gets picked? fixes the first issue?