Executables with shared libraries fail to run with err "no such file or directory"

[Cequallium]

Hi, I tried this and i checked all the links, they all seem to exist.
As with gopls it used to work perfectly as did all the other executables and they all broke randomly at the same time. So i included it as i felt it was related.

$ ldd --verbose ./main
        linux-vdso.so.1 (0x00007ffdfafba000)
        libresolv.so.2 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libresolv.so.2 (0x00007f598a59c000)
        libpthread.so.0 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libpthread.so.0 (0x00007f598a597000)
        libc.so.6 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6 (0x00007f598a3aa000)
        /nix/store/35pq4hr29c3sl79lgfwgsvd9nwzyp4am-glibc-2.39-5/lib/ld-linux-x86-64.so.2 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib64/ld-linux-x86-64.so.2 (0x00007f598a5af000)

        Version information:
        ./main:
                libc.so.6 (GLIBC_2.34) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_2.32) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
        /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libresolv.so.2:
                libc.so.6 (GLIBC_ABI_DT_RELR) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_2.14) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_2.4) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_2.34) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_2.3) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
        /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libpthread.so.0:
                libc.so.6 (GLIBC_ABI_DT_RELR) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
        /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib64/ld-linux-x86-64.so.2

[eljamm]

A few questions / observations:

Example 1

With go run, the files are copied to a temporary location, so that’s not weird. Printing the executable path for catsay-go shows the following:

$ go run Catsay.go
/tmp/go-build596412124/b001/exe
 Usage: catsay <Phrase> <Color (1 to 8)>
 Colors:  1  2  3  4  5  6  7  8 

Catsay.go

import (
	"fmt"
	"os"
	"path/filepath"
)

func main() {
	// print current path
	ex, err := os.Executable()
	if err != nil {
		panic(err)
	}
	exPath := filepath.Dir(ex)
	fmt.Println(exPath)
    ...

Could this be an issue with the project? Does it build normally?

Example 2

Mason won’t work normally with Nix, so you need to either install the tools you need in your system or add them to neovim:

Cannot install gopls using mason.nvim - #2 by eljamm

However, don’t forget to remove the tools that have already been downloaded by going to the Mason menu, hovering over them, then pressing X. running :MasonUninstallAll

If you want to share your config between NixOS and other distros, you can do something like this:

NeoVim cannot start lsp because clangd is not found - #4 by eljamm

So in NixOS, nvim will use the system/home-manager tools while in other distros, Mason will install the binaries as usual.

Example 3

This works on my machine. What happens when you delete target/ and run cargo run again?

[Cequallium]

Hi thanks for taking the time to contribute

• The project from Example 1 is GitHub - Cequallium/whispers: a plaintext blog
  it built perfectly in its current state (latest commit)
• Example 2:
  I’ll certainly do that from now on.
• Example 3:
  huh, it does start working. Is this a go specific issue?

[eljamm]

@Cequallium

Example 3:
huh, it does start working. Is this a go specific issue?

The person in this comment faced a similar issue and suggested deleting the target/ folder for Rust. Does the Go project from example 1 work when you delete whispers and rebuild?

[Cequallium]

No, it does not work.

[eljamm]

Comparing with your previous ldd --verbose ./main, the main difference is the following:

- /nix/store/35pq4hr29c3sl79lgfwgsvd9nwzyp4am-glibc-2.39-5/lib/ld-linux-x86-64.so.2 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib64/ld-linux-x86-64.so.2 (0x00007f598a5af000)
+ /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/ld-linux-x86-64.so.2 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib64/ld-linux-x86-64.so.2 (0x00007f37efd52000)

You have glibc-2.39-5 pointing to glibc-2.39-31. Would this probably be the cause of the issue?

Edit:

$ LD_PRELOAD="/nix/store/35pq4hr29c3sl79lgfwgsvd9nwzyp4am-glibc-2.39-5/lib/ld-linux-x86-64.so.2" ./whispers
Floating point exception (core dumped)

$ LD_PRELOAD="/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/ld-linux-x86-64.so.2" ./whispers
whispering on http://localhost:8080/

This might be related to this:

Local Builds and System (AUR) Packages Linking Nix Libraries

I have found the culprit. When I was fiddling with LSPs and neovim, I put clang under home.packages, which also put ld (and many other compiler-related applications not supposed to be there) under PATH. Moving them into programs.neovim.extraPackages prevents such pollution.

In effect, one of the packages in your home.nix/dev.nix might be interfering with go.

If you try this, does it work?

$ nix-shell --pure -p go
$ go run ./main.go

[rgoulter]

@Cequallium

Is this a go specific issue?

Just to rule out anything weird, which go are you using for Example 1? which go? (& how was it installed).

My understanding of the issue is it’s something to do with shared libraries & dynamic linking.

(e.g. Without net/http, the go programs don’t link against shared libraries for that case).

[Cequallium]

You’re totally right, it is related to shared libraries

@Cequallium

also one more thing, i noticed your reproducer imported net/http and my project did too, so i tested by removing it and compiling it. it ran perfectly without any errors.

$ which go
/home/cql/.nix-profile/bin/go

$ whereis go
go: /nix/store/vqwb13qqvqnmqksxpv5s7b1kf0vd50cf-home-manager-path/bin/go

@Cequallium

i have installed go via home-manager through programs.go.enable = true;
cargo is also installed via home-manager
i am using nixpkgs-unstable

[Cequallium]

No, it unfortunately does not.

[Cequallium]

Just to add more information, i did have nix-ld enabled in my configuration.nix a while ago, but i have since removed it.

@rgoulter

My understanding of the issue is it’s something to do with shared libraries & dynamic linking.

[eljamm]

What does running type -a gcc give you?

[rgoulter]

@Cequallium

You’re totally right, it is related to shared libraries

Looking at the nix-ld readme, it suggests running an executable with the environment variable LD_DEBUG=libs to figure out what libs are missing.

[Cequallium]

it doesnt seem to work

$ LD_DEBUG=libs ./main
zsh: no such file or directory: ./main

[Cequallium]

$ type -a gcc
gcc is /home/cql/.nix-profile/bin/gcc

[eljamm]

It could be useful to run a nested shell in this case:

$ LD_DEBUG=libs zsh
$ ./main

It will show you the libraries each command will try to load.

[Cequallium]

heres everything after pressing enter on ./main

zsh: no such file or directory: ./main
     19838:     find library=libc.so.6 [0]; searching
     19838:      search path=/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/glibc-hwcaps/x86-64-v3:/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/glibc-hwcaps/x86-64-v2:/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib          (system search path)
     19838:       trying file=/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/glibc-hwcaps/x86-64-v3/libc.so.6
     19838:       trying file=/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/glibc-hwcaps/x86-64-v2/libc.so.6
     19838:       trying file=/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6
     19838:
     19838:     find library=libgcc_s.so.1 [0]; searching
     19838:      search path=/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib              (system search path)
     19838:       trying file=/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libgcc_s.so.1
     19838:      search path=/nix/store/p3ffjixpnfgkqh20nsrc13vrj3yfi0nj-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v3:/nix/store/p3ffjixpnfgkqh20nsrc13vrj3yfi0nj-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v2:/nix/store/p3ffjixpnfgkqh20nsrc13vrj3yfi0nj-gcc-13.2.0-lib/lib               (RUNPATH from file /home/cql/.nix-profile/bin/starship)
     19838:       trying file=/nix/store/p3ffjixpnfgkqh20nsrc13vrj3yfi0nj-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v3/libgcc_s.so.1
     19838:       trying file=/nix/store/p3ffjixpnfgkqh20nsrc13vrj3yfi0nj-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v2/libgcc_s.so.1
     19838:       trying file=/nix/store/p3ffjixpnfgkqh20nsrc13vrj3yfi0nj-gcc-13.2.0-lib/lib/libgcc_s.so.1
     19838:
     19838:     find library=libm.so.6 [0]; searching
     19838:      search path=/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib              (system search path)
     19838:       trying file=/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libm.so.6
     19838:
     19838:
     19838:     calling init: /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/ld-linux-x86-64.so.2
     19838:
     19838:
     19838:     calling init: /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6

[eljamm]

Alright, so I found an interesting project nix-alien which allows you to run unpatched binaries.

Inside the project directory, run:

$ nix --extra-experimental-features "nix-command flakes" run github:thiagokokada/nix-alien -- ./main

If that worked, the following command will show you which libs are needed:

$ nix --extra-experimental-features "nix-command flakes" run github:thiagokokada/nix-alien#nix-alien-find-libs -- ./main

I tried this on a random non-NixOS program and it worked nicely.

If after all of this and it still doesn’t run, I think this might be an issue with the project itself or something else like a permission error…

[Cequallium]

@eljamm

nix --extra-experimental-features "nix-command flakes" run github:thiagokokada/nix-alien -- ./main

oh my god, it works, tysm!
also running the second command doesnt output anything… oddly.

[eljamm]

When you run the first command, does it show you a list of libraries and asks you to choose or does it just launch the program?

[Cequallium]

no it just launches the program

*i wrote dependency instead of library