Understanding ldd output

Help
1

I’m trying to create a somewhat FHS compatible NixOS installation by symlinking /run/current-system/sw to /usr.

I’m using the configuration

{ pkgs, ... }: {
  systemd.tmpfiles.rules = [ "L+ /usr - - - - /run/current-system/sw" ];
  environment.systemPackages = [ pkgs.python3 pkgs.python3Packages.pip ];
}

and start a virtual machine built with this configuration with nixos-shell which mounts my $HOME folder to the VM.

For testing, I used a Hello World binary compiled in a Debian Bookworm chroot, that I copied to ~/hello_compiled_on_bookworm.

On my host system I am using nix-ld, so I can execute that file. On my host system, ldd shows me

ldd ~/hello_compiled_on_bookworm
	linux-vdso.so.1 (0x00007ffd2bbab000)
	libc.so.6 => /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6 (0x00007f6d9e4b3000)
	/lib64/ld-linux-x86-64.so.2 => /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib64/ld-linux-x86-64.so.2 (0x00007f6d9e6a0000)

On my experimental VM with the above config, when trying to run that binary I get

~/hello_compiled_on_bookworm
zsh: no such file or directory: /home/jan/hello_compiled_on_bookworm

In that VM, ldd shows me

~/mydocuments/computer/nix/nixos-fhs: ldd ~/hello_compiled_on_bookworm
	linux-vdso.so.1 (0x00007ffea77ac000)
	libc.so.6 => /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/libc.so.6 (0x00007f7a508c7000)
	/lib64/ld-linux-x86-64.so.2 => /nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib64/ld-linux-x86-64.so.2 (0x00007f7a50ab4000)

I don’t understand how ldd in the VM understands that it has to search for files in the Nix store. LD_LIBRARY_PATH is not set. The directory /lib64 does not exist. I suspect that nixos-shell is leaking parts of my environment into the VM but I don’t understand which parts could be responsible for this.

In the Debian Bookworm chroot, ldd yields

ldd a.out
	linux-vdso.so.1 (0x00007ffea3f99000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd45f87a000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd45fa66000)

And I would have expected a similar output from ldd inside the VM.

2

On a sidenote, does someone know a small Python package on PyPI that contains binary code? Ultimately I’d like to test if my FHS solution works with pip-installed Python packages that contain binary code. I could use the pandas package, but it’s quite large, and a smaller package would be nice for testing.

3

Your ldd has the glibc somewhere in its own RPATH, and as ldd assumes that every lib it knows about itself is also available to any program it inspects, it prints its own glibc here. Your a.out is not aware of the library.

Same is actually true about the linker… Your binary does not see it.

For semi-FHS-compliance you need to also provide the dynamic linker.

PS: as lib is usually not part of the system, I’d create another abstraction, that explicite is only for libs and perhaps headers, and link that one rather than the system, to keep some seperation of concerns.

1 Like
4

For semi-FHS-compliance you need to also provide the dynamic linker.

It should be possible to just copy the statically linked /lib64/ld-linux-x86-64.so.2 from another distro, right?

PS: as lib is usually not part of the system, I’d create another abstraction, that explicite is only for libs and perhaps headers, and link that one rather than the system, to keep some seperation of concerns.

Also I found that for example libc.so.6 is missing from /run/current-system/sw/lib. What do we actually have that folder for, given that on NixOS binaries are usually patchelfed and shouldn’t require a lib folder outside of the Nix store?

5

I found that I can invoke the dynamic linker directly, and run the executable like that:

/var/chroots/bookworm/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --library-path /var/chroots/bookworm/lib/x86_64-linux-gnu /var/chroots/bookworm/hello_world_c/a.out
Hello

However I can not run it when setting the library PATH to a NixOS store path:

/var/chroots/bookworm/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --library-path /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/ /var/chroots/bookworm/hello_world_c/a.out
*** stack smashing detected ***: terminated
zsh: IOT instruction (core dumped)  /var/chroots/bookworm/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --library-pat