"Could not start dynamically linked executable" on cargo run

So I want to do embedded programming on an esp32. Therefore I made a shell that installs different tooling etc. Here is the shell.nix

{ pkgs ? import <nixpkgs> {} }:
  let
    overrides = (builtins.fromTOML (builtins.readFile ./rust-toolchain.toml));
    libPath = with pkgs; lib.makeLibraryPath [
      # load external libraries that you need in your rust project here
    ];
in
  pkgs.mkShell rec {
    buildInputs = with pkgs; [
      clang
      # Replace llvmPackages with llvmPackages_X, where X is the latest LLVM version (at the time of writing, 16)
      espflash
      cargo-binstall
      llvmPackages.bintools
      rustup
      cargo
    ];
    RUSTC_VERSION = overrides.toolchain.channel;
    # https://github.com/rust-lang/rust-bindgen#environment-variables
    LIBCLANG_PATH = pkgs.lib.makeLibraryPath [ pkgs.llvmPackages_latest.libclang.lib ];
    shellHook = ''
      export PATH=$PATH:''${CARGO_HOME:-~/.cargo}/bin
      export PATH=$PATH:''${RUSTUP_HOME:-~/.rustup}/toolchains/$RUSTC_VERSION-x86_64-unknown-linux-gnu/bin/
      '';
    # Add precompiled library to rustc search path
    RUSTFLAGS = (builtins.map (a: ''-L ${a}/lib'') [
      # add libraries here (e.g. pkgs.libvmi)
    ]);
    LD_LIBRARY_PATH = libPath;
    # Add glibc, clang, glib, and other headers to bindgen search path
    BINDGEN_EXTRA_CLANG_ARGS =
    # Includes normal include path
    (builtins.map (a: ''-I"${a}/include"'') [
      # add dev libraries here (e.g. pkgs.libvmi.dev)
      pkgs.glibc.dev
    ])
    # Includes with special directory paths
    ++ [
      ''-I"${pkgs.llvmPackages_latest.libclang.lib}/lib/clang/${pkgs.llvmPackages_latest.libclang.version}/include"''
      ''-I"${pkgs.glib.dev}/include/glib-2.0"''
      ''-I${pkgs.glib.out}/lib/glib-2.0/include/''
    ];
  }

(This template is from the official NixOS Wiki)

Now I create a file called rust-toolchain.toml in the same Directory as the shell.nix file. And inside of the rust-toolchain.toml file I put:

[toolchain]
channel = "stable"

After entering the shell I do:
esp-generate --chip esp32 blinky I cd into the newly created project cd blinky and try runnint cargo run but I get this error message:

[nix-shell:~/Personal/Rust/embedded/blinky]$ cargo run --release
Could not start dynamically linked executable: /home/kronix/.rustup/toolchains/esp/bin/cargo
NixOS cannot run dynamically linked executables intended for generic
linux environments out of the box. For more information, see:
https://nix.dev/permalink/stub-ld

I don’t understand what I am doing wrong? I really love NixOS but it is depressing to say the least to encounter so many errors that you don’t have on more conventional Distros. But this uniqueness makes NixOS really great, too!

Hi Kronix,
I could not reproduce your issue, may I know how did you install the esp toolchain?

I used the espup package from nix packages to install the toolchain, and inside the blinky project the toolchain was set to “esp”, and cargo run works for me (at least it compiles without cargo issue).

I also see a message telling me to activate the environment setup by esp up by running:

. ~/export-esp.sh

Can you try with this, add espup to your buildInputs, then uninstall the toolchain you previously install, and reinstall with espup from nix package.

The issues stems from dynamic linked binaries being compiled in such a way that Nix itself disagrees with.
After the build is complete with cargo, you’ll probably still need to strip and link the bins to make them fully nix compliant.

Without configuring in the ld as part of the nix main config, another way is to use either autoPatchelf or patchelf to strip and link the bins.
Ive never tinkered with an esp board personally but i can make some guesses as to what method will probably work, an if you are compiling from x86 cross to the little esp board then static linking might be a better option, but if you are building and running on esp then dynamic linking or static linking would be aight.

The easy solution is autopatchelf, you can add it to the rec statement above buildInputs like so:

  nativeBuildInputs = [
    autoPatchelfHook
  ];

Most of the time, it should link and execute at run time since the shell you are making is basically a wrapper.
If you want a static linked solution, you’d need to include the relevant libraries at the very top and then add them as a build libPath which would look about like this:

  preFixup = let
    # we prepare our library path in the let clause to avoid it become part of the input of mkDerivation
    libPath = lib.makeLibraryPath [
      qt5.qtbase        # libQt5PrintSupport.so.5
      qt5.qtsvg         # libQt5Svg.so.5
      stdenv.cc.cc.lib  # libstdc++.so.6
      saneBackends      # libsane.so.1
    ];
  in ''
    patchelf \
      --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
      --set-rpath "${libPath}" \
      $out/opt/master-pdf-editor-4/masterpdfeditor4
  '';

source: Packaging/Binaries - NixOS Wiki
But the caveat is you’d need to call the program using a wrapper that exec’s out to start it so env variables get passed to the child process that spawns from the script.

There are a few ways to do this, you could simply call the execution at the end of your build with exec but thats not as flexible. My solution was to package the bins as dynamically linked with the lib paths being built in the drv, and then a writing a pkg.mkShellApplication i could call from the command line directly that invokes the script that calls exec where the lib paths of the drv get inherited by the child process that spawns.

Heres my drv which does build and run as intended, although its got its own problems but i think it would serve as a useful template for your project. Please ignore the colorful debug language.

{ stdenv, lib, fetchurl, patchelf, makeWrapper, jack2, alsa-lib, writeShellApplication, makeDesktopItem, gtk3, glib, xorg, remarkable2-toolchain, bash, autoPatchelfHook, copyDesktopItems, vulkan-loader, libva, wayland }:

let

  libPath = lib.makeLibraryPath [ autoPatchelfHook alsa-lib jack2 stdenv.cc.cc stdenv.cc.cc.lib remarkable2-toolchain glib gtk3 xorg.libX11 xorg.libXrandr vulkan-loader libva wayland ];

  acesx86_64 = writeShellApplication {
  name = "acesx86_64";
  runtimeInputs = [ bash wayland libva vulkan-loader alsa-lib jack2 ];

  text = ''
    #!/bin/bash

    STORE_PATH=$(dirname "$(dirname "$(readlink -f "$(which launcher)")")")

    echo "$STORE_PATH"
    echo "DEBUG::: STORE_PATH = $STORE_PATH"
    echo "Check for home directory, create if not present."
    export ACES64_DIR=$HOME/.aces64/War-Thunder-086d99e/
    echo "DEBUG::: Print LD env variables prior to setting them."
    echo "$LD_LIBRARY_PATH"
    echo "DEBUG::: Print libPath inherited by drv on realization."
    echo ${libPath}

    echo "DEBUG::: Setting libPath for runtime enviroment."
    export LD_LIBRARY_PATH="${libPath}:$LD_LIBRARY_PATH"
    echo "DEBUG::: Print libPath: $LD_LIBRARY_PATH"

    if [ ! -d "$HOME/.aces64/War-Thunder-086d99e" ]; then

      echo "DEBUG::: Directory not found. Fuck."

      mkdir -p "$HOME/.aces64/War-Thunder-086d99e"
      cd "$HOME/.aces64/War-Thunder-086d99e"

      echo "DEBUG::: Directory created and CD'ed."
      echo "DEBUG::: Setting Enviroment variables for runtime again."
      export ACES64_DIR=$HOME/.aces64/War-Thunder-086d99e/
      else
        echo "DEBUG::: Directory exits, fucking good.."
    fi
    echo "Installing launcher, bpreport, and selfupdater scripts to the user directory"

    install -m755 -D \
    "$STORE_PATH"/War-Thunder-086d99e/launcher "$ACES64_DIR/launcher"

    install -m755 -D \
    "$STORE_PATH"/War-Thunder-086d99e/gaijin_selfupdater "$ACES64_DIR/gaijin_selfupdater"

    install -m755 -D \
    "$STORE_PATH"/War-Thunder-086d99e/bpreport "$ACES64_DIR/bpreport"

    cp -f "$STORE_PATH"/War-Thunder-086d99e/ca-bundle.crt "$ACES64_DIR/ca-bundle.crt"
    cp -f "$STORE_PATH"/War-Thunder-086d99e/launcherr.dat "$ACES64_DIR/launcherr.dat"
    cp -f "$STORE_PATH"/War-Thunder-086d99e/libsciter-gtk.so "$ACES64_DIR/libsciter-gtk.so"
    cp -f "$STORE_PATH"/War-Thunder-086d99e/libsteam_api.so "$ACES64_DIR/libsteam_api.so"
    cp -f "$STORE_PATH"/War-Thunder-086d99e/package.blk "$ACES64_DIR/package.blk"
    cp -f "$STORE_PATH"/War-Thunder-086d99e/yupartner.blk "$ACES64_DIR/yupartner.blk"





    cd "$ACES64_DIR" || { echo "cd command rejected, breaking"; exit 1; }
    echo "DEBUG::: Changing root directory to $ACES64_DIR"
    echo "DEBUG::: Where the fuck am i. $PWD"

    exec "$ACES64_DIR/launcher"
    '';}; in
stdenv.mkDerivation rec {
  name = "WarThunder";
  pname = "War-Thunder";
  version = "086d99e";

  src = fetchurl {
    url = "https://github.com/Mephist0phel3s/War-Thunder/archive/refs/tags/086d99e.tar.gz";
    hash = "sha256-vqpx85ZT1AzKk7dkZvMDMJf9GWalDM/F2JhaiMybMoY=";
  };


  sourceRoot = "./${pname}-${version}";
  unpackPhase = false;
  dontConfigure = true;
  dontBuild = true;
  nativeBuildInputs = [ copyDesktopItems ];
  buildInputs = [ autoPatchelfHook stdenv.cc.cc stdenv.cc.cc.lib remarkable2-toolchain glib gtk3 xorg.libX11 xorg.libXrandr vulkan-loader libva wayland ];

#  patchPhase = let





  installPhase = let

  desktopItem = makeDesktopItem {
    name = "WarThunder";
    exec = "acesx86_64";
    icon = "launcher";
    desktopName = "War Thunder";
    genericName = "War Thunder";
    categories = [ "Game" ];
    }; in ''
  runHook preInstall
    mkdir -p $out/${pname}-${version}
    mkdir -p $out/share/pixmaps
    mkdir -p $out/bin

    if [ -f "${acesx86_64}/bin/acesx86_64" ]; then
      install -m755 -D ${acesx86_64}/bin/acesx86_64 $out/bin/acesx86_64
    else
      echo "FATAL ERROR DEBUG::: acesx86_64 script not found at ${acesx86_64}, breaking."
      exit 1
    fi

    install -m755 -D launcher $out/${pname}-${version}/launcher
    install -m755 -D gaijin_selfupdater $out/${pname}-${version}/gaijin_selfupdater
    install -m755 -D bpreport $out/${pname}-${version}/bpreport

    cp ca-bundle.crt $out/${pname}-${version}/ca-bundle.crt
    cp launcherr.dat $out/${pname}-${version}/launcherr.dat
    cp libsciter-gtk.so $out/${pname}-${version}/libsciter-gtk.so
    cp libsteam_api.so $out/${pname}-${version}/libsteam_api.so
    cp package.blk $out/${pname}-${version}/package.blk
    cp yupartner.blk $out/${pname}-${version}/yupartner.blk

    echo "STORE_PATH=\"$out/${pname}-${version}\"" > $out/${pname}-${version}/store_path.txt

    echo "DEBUG::: Running patchelf to link binaries"
    patchelf \
      --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
      --set-rpath ${libPath} launcher
    patchelf \
      --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
      --set-rpath ${libPath} bpreport
    patchelf \
      --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" \
      --set-rpath ${libPath} gaijin_selfupdater
    echo "DEBUG::: Done, proceeding."

    install -m755 -D launcher.ico $out/share/pixmaps/launcher.png
    echo "DEBUG::: Installing the fucking desktop file"
    mkdir -p "$out/share/applications"
    echo "INFO: Skpipping cp -rf desktopItem"
    echo "INFO: sym linking aces to WarThunder for purposes of desktop execution."
    ln -s ${acesx86_64}/bin/acesx86_64 $out/bin/WarThunder

    echo "INFO: Installing bins to out/bin"
    install -m755 -D launcher $out/bin
    install -m755 -D gaijin_selfupdater "$out/bin"
    install -m755 -D bpreport "$out/bin"
    echo "INFO: Done"

  runHook postInstall
  '';


  meta = with lib; {
    homepage = "https://warthunder.com/";
    description = "Military Vehicle PVP simulator, tanks, planes, warships. Report bugs with this nix pacakge as an issue on github @https://github.com/Mephist0phel3s/nixpkgs/issues";
    platforms = platforms.linux;
    maintainers = with maintainers; [ Mephist0phel3s ];
  };
}

Side note, there really should be a section in the wiki or nix.dev for this thats easier to find but there isnt unfortunately.
the nix-ld is not enabled by default, and might be what you need to help debug your project.

TLDR, add this to your config with libraries you need to run off the shelf linux bins in nixos without having to using patchelf on the terminal to make it work every time you unpack a tarball.

programs.nix-ld.enable = true;
programs.nix-ld.libraries = with pkgs; [
xorg.libX11 #example, this is a library needed for the aces project i just posted
  ];

src for nix-ld: GitHub - nix-community/nix-ld: Run unpatched dynamic binaries on NixOS [maintainer=@Mic92]

EDIT: Fixed grammar

I installed the esp toolchain version via espup.
For reproducibility I added it to the shell.nix

{ pkgs ? import <nixpkgs> {} }:
  let
    overrides = (builtins.fromTOML (builtins.readFile ./rust-toolchain.toml));
    libPath = with pkgs; lib.makeLibraryPath [
      # load external libraries that you need in your rust project here
    ];
in
  pkgs.mkShell rec {
    buildInputs = with pkgs; [
      clang
      # Replace llvmPackages with llvmPackages_X, where X is the latest LLVM version (at the time of writing, 16)
      espflash
      cargo-binstall
      espup
      llvmPackages.bintools
      rustup
      cargo
    ];
    RUSTC_VERSION = overrides.toolchain.channel;
    RUSTUP_TOOLCHAIN = "esp";
    # https://github.com/rust-lang/rust-bindgen#environment-variables
    LIBCLANG_PATH = pkgs.lib.makeLibraryPath [ pkgs.llvmPackages_latest.libclang.lib ];
    shellHook = ''
      export PATH=$PATH:''${CARGO_HOME:-~/.cargo}/bin
      export PATH=$PATH:''${RUSTUP_HOME:-~/.rustup}/toolchains/$RUSTC_VERSION-x86_64-unknown-linux-gnu/bin/



        echo "Installing ESP toolchain..."
        espup install

      if [ -f "$HOME/export-esp.sh" ]; then
        echo "Sourcing export-esp.sh..."
        . $HOME/export-esp.sh
      fi
      '';
    # Add precompiled library to rustc search path
    RUSTFLAGS = (builtins.map (a: ''-L ${a}/lib'') [
      # add libraries here (e.g. pkgs.libvmi)
    ]);
    LD_LIBRARY_PATH = libPath;
    # Add glibc, clang, glib, and other headers to bindgen search path
    BINDGEN_EXTRA_CLANG_ARGS =
    # Includes normal include path
    (builtins.map (a: ''-I"${a}/include"'') [
      # add dev libraries here (e.g. pkgs.libvmi.dev)
      pkgs.glibc.dev
    ])
    # Includes with special directory paths
    ++ [
      ''-I"${pkgs.llvmPackages_latest.libclang.lib}/lib/clang/${pkgs.llvmPackages_latest.libclang.version}/include"''
      ''-I"${pkgs.glib.dev}/include/glib-2.0"''
      ''-I${pkgs.glib.out}/lib/glib-2.0/include/''
    ];
  }

And changed the content int the rust-toolchain.toml to

[toolchain]
channel = "esp"

And I still get the exact same error:

Could not start dynamically linked executable: /home/kronix/.rustup/toolchains/esp/bin/cargo
NixOS cannot run dynamically linked executables intended for generic
linux environments out of the box. For more information, see:
https://nix.dev/permalink/stub-ld

Even if I just do cargo --version

You didnt include your libraries in the call for libPath, so no lib packages are actually being linked during the build. CC your libs needed for runtime to here

Can you elaborate? What libraries do I need for this cargo error? I am new to nixos, so sorry if the question is dumb

No worries, not a dumb question at all, took me a week to figure it out on my own so no sweat.

The short version is you need to ldd and patchelf --print-needed the executables you are pulling down for the toolchain.

The nix linker is Black Magic. I am going to show you the manual method so the autoPatchelfHook makes sense, as it does the same thing its just an automated process you use during build instead of having to hunt down missing libs in bins.

This is a long one so get some coffee or a coke before you start reading.

Starting point:
We are going to need a few things, notably nix-index and nix-locate. You can do this as a nix-shell but if dev is your thing then you’ll want it for later as the db can take a while to populate.
Edit your /etc/nixos/configuration.nix file and add the pkg nix-index and nix-locate to system packages, rebuild and run:

nix-index

This will take a minute, so do this now while you are reading.

Now we need to know what libs the bin needs to do stuff. Im going to use my aces project from earlier as an example.
My project was an already packaged program native to linux, just needed to be ported so the process should mirror pretty close to what you are tryn for.
Here, im downloading a fresh copy of the bins from the official website for the purpose of demonstration.

[jason@Beast:~/tmp]$ tar -xf wt_launcher_linux_1.0.3.22.tar.gz

[jason@Beast:~/tmp]$ cd WarThunder/

[jason@Beast:~/tmp/WarThunder]$ ls
bpreport ca-bundle.crt gaijin_selfupdater launcher launcher.ico launcherr.dat libsciter-gtk.so libsteam_api.so package.blk yupartner.blk

[jason@Beast:~/tmp/WarThunder]$ ./launcher
./launcher: error while loading shared libraries: libglib-2.0.so.0: cannot open shared object file: No such file or directory

[jason@Beast:~/tmp/WarThunder]$

No worky, as expected. Its not been stripped or linked yet so lets start/

First, we query the bin for what it wants:

[jason@Beast:~/tmp/WarThunder]$ patchelf --print-interpreter launcher
/lib64/ld-linux-x86-64.so.2

[jason@Beast:~/tmp/WarThunder]$ patchelf --print-needed launcher
libXrandr.so.2
librt.so.1
libpthread.so.0
libdl.so.2
libX11.so.6
libgtk-3.so.0
libglib-2.0.so.0
libgobject-2.0.so.0
libgio-2.0.so.0
libgdk-3.so.0
libstdc++.so.6
libm.so.6
libgcc_s.so.1
libc.so.6

[jason@Beast:~/tmp/WarThunder]$ 

This lists all the libs needed for execution.
Now we need to see which are linked and which arent, we need to use ldd.

[jason@Beast:~/tmp/WarThunder]$ ldd launcher
linux-vdso.so.1 (0x00007ffe8d9fa000)
libXrandr.so.2 => not found
librt.so.1 => /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/librt.so.1 (0x000070754dcdf000)
libpthread.so.0 => /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libpthread.so.0 (0x000070754dcda000)
libdl.so.2 => /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libdl.so.2 (0x000070754dcd5000)
libX11.so.6 => not found
libgtk-3.so.0 => not found
libglib-2.0.so.0 => not found
libgobject-2.0.so.0 => not found
libgio-2.0.so.0 => not found
libgdk-3.so.0 => not found
libstdc++.so.6 => not found
libm.so.6 => /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libm.so.6 (0x000070754dbea000)
libgcc_s.so.1 => /nix/store/2d5spnl8j5r4n1s4bj1zmra7mwx0f1n8-xgcc-13.3.0-libgcc/lib/libgcc_s.so.1 (0x000070754dbc5000)
libc.so.6 => /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6 (0x000070754d9cc000)
/lib64/ld-linux-x86-64.so.2 => /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib64/ld-linux-x86-64.so.2 (0x000070754dce6000)

[jason@Beast:~/tmp/WarThunder]$

Now we know which libs are missing. Lets link them.

Before we do, we need to build lib paths in the shell for them and thats where we will start to use nix-index and nix repl.

You’ve probably not been introduced yet to nix repl, so the brief explanation is nix repl is a shell you can launch that will accept commands like a nix shell, you can build an entire drv and install it only using nix repl. Its a powerful tool thats woven into the very core of nix so its worth taking some time to get familiar with it.

Nix repl can also be the most infuriating piece of dog shit software written since windows ME but i digress.
running nix repl will just drop you in a shell with no variables or pkgs imported so you will need to import the pkg repo directly or you can get a copy of the repo as a git and execute nix repl from that directory to import all of them that way. Your pick on that.

But for the sake of simplicity, you can just run (as normal user, no need for root)

[jason@Beast:~/tmp/WarThunder]$ nix repl
Nix 2.24.11
Type :? for help.
nix-repl> :l <nixpkgs>
Added 22928 variables.

nix-repl> 

By this point, your nix-index should be done.
Take all the libs you found in print-missing statement from the elf interpreter and do this with them.
Im going to use the lib libXrandr.so.2 for this example.
Here we are going to use nix-locate to find what pkg contains this library.

[jason@Beast:~/tmp/WarThunder]$ nix-locate --top-level -w lib/libXrandr.so.2
xorg.libXrandr.out 0 s /nix/store/k4wbhldwg6lfsiaxmp6fjzma8qdh3vm5-libXrandr-1.5.4/lib/libXrandr.so.2

[jason@Beast:~/tmp/WarThunder]$

Coo, the pkg xorg.libXrandr contains the lib we need. So lets build a path for it, back to nix repl.

nix-repl> with pkgs; lib.makeLibraryPath [ xorg.libXrandr ] 
"/nix/store/k4wbhldwg6lfsiaxmp6fjzma8qdh3vm5-libXrandr-1.5.4/lib"

nix-repl> 

Thats our new library path, now we can take and link that path to the bin using patchelf on the terminal.
First, we need to spawn a nix-shell with a standard dev env, then set the link.

[jason@Beast:~/tmp/WarThunder]$ nix-shell -p stdenv.cc.cc stdenv.cc.cc.lib
bash: .: /run/current-system/sw/etc/bash_completion.d/bash-completion: is a directory

[nix-shell:~/tmp/WarThunder]$ patchelf --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" --set-rpath /nix/store/k4wbhldwg6lfsiaxmp6fjzma8qdh3vm5-libXrandr-1.5.4/lib launcher

[nix-shell:~/tmp/WarThunder]$ 

Now lets check with the interpreter again and see if this lib is still listed as missing.

[nix-shell:~/tmp/WarThunder]$ ldd launcher
linux-vdso.so.1 (0x00007ffd2e9f4000)
libXrandr.so.2 => /nix/store/k4wbhldwg6lfsiaxmp6fjzma8qdh3vm5-libXrandr-1.5.4/lib/libXrandr.so.2 (0x000070cd8a983000)
librt.so.1 => /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/librt.so.1 (0x000070cd8a97e000)
libpthread.so.0 => /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/libpthread.so.0 (0x000070cd8a979000)
libdl.so.2 => /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/libdl.so.2 (0x000070cd8a974000)
libX11.so.6 => not found
libgtk-3.so.0 => not found
libglib-2.0.so.0 => not found
libgobject-2.0.so.0 => not found
libgio-2.0.so.0 => not found
libgdk-3.so.0 => not found
libstdc++.so.6 => not found
libm.so.6 => /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/libm.so.6 (0x000070cd8a889000)
libgcc_s.so.1 => /nix/store/2d5spnl8j5r4n1s4bj1zmra7mwx0f1n8-xgcc-13.3.0-libgcc/lib/libgcc_s.so.1 (0x000070cd8a864000)
libc.so.6 => /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/libc.so.6 (0x000070cd8a66b000)
libXext.so.6 => /nix/store/hi1mbz7s24l63w1m9sy9ynl8yjvb2f46-libXext-1.3.6/lib/libXext.so.6 (0x000070cd8a654000)
libXrender.so.1 => /nix/store/bnfmimcfan36gg0g488c5hksg8nyr16y-libXrender-0.9.11/lib/libXrender.so.1 (0x000070cd8a647000)
libX11.so.6 => /nix/store/zcq4irdcgn3ljqdnlpm2zjp7f1kw9jvm-libX11-1.8.10/lib/libX11.so.6 (0x000070cd8a4fb000)
/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/ld-linux-x86-64.so.2 => /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib64/ld-linux-x86-64.so.2 (0x000070cd8a992000)
libxcb.so.1 => /nix/store/c25p9xs9n6grwx4i4l4kmz09scgcav4b-libxcb-1.17.0/lib/libxcb.so.1 (0x000070cd8a4cf000)
libXau.so.6 => /nix/store/6a49zj2wva8nxw7sidw9j9bp2nifscbw-libXau-1.0.11/lib/libXau.so.6 (0x000070cd8a4ca000)
libXdmcp.so.6 => /nix/store/1x0fg2bhf14lk83458c4iir92hc1njjh-libXdmcp-1.1.5/lib/libXdmcp.so.6 (0x000070cd8a4c0000)

[nix-shell:~/tmp/WarThunder]$

And we see, libXrandr is now linked for execution.

This is the long version of how patchelf works to ‘nixify’ bins.

2 Likes

Wow man… I can’t express how thankful I am that you took your time and explained everything so clearly. So what I have to do now is to link to .rustup/toolchains/esp/bin/cargo, right?

1 Like

Correct, there are probably some other gotchas mixed in there given you are assembling a rust program to run on another board but probably yeah.

Side note, you can build multiple lib paths at the same time in the repl by including more packages in the [ libs ] statement separated by a space, itll spit out a copy pastable path you can use for the linker the same way you did before.

Once you get everything linked and you can run the program from the terminal directly, then you can translate the nix repl into the drv directly by calling the buildlibPath function, and then taking that as a variable you can call within another tested function like so :

export LD_LIBRARY_PATH="${libPath}:$LD_LIBRARY_PATH

The extra set of {} around the lib path nested inside another function will call that function one level higher, calling $libPath within a nested function will return bad data in this scenario.

1 Like

I am completely honest I tried it for the past hours and I just don’t get it. I am just not clever enough for NixOS. It works for other people but not for me and after scattering the Internet for a template for my specific esp32 + the correct esp-hal it has an compiling error from a lib that was deprecated last year. I am just depressed and going to sleep. Still thanks for the patience and help… I am clueless at this point

It took me nearly a full month to build the drv i shared earlier, and many of those nights were full of bugs and compile/build errors that were infuriatingly non-descriptive bullshit.

I had to re-re-re-re-re-re-re-re-re-re-read the manual and wiki. Alot. A fucking ton to really understand anything that was going on under the hood so dont feel too bad.

Get a good nights sleep, and take a crack at it again tomorrow. Im on the forums every day so if you need help just send me a message or reply to this thread.

I gotchu bro.

And if ever you need a laugh, remember this.

One time i nearly died to a bot in PUBG, but i didnt know it was a bot. I proceeded to shit talk this bot for a solid 40 seconds after i killed it before one of my teammates mentioned its not in fact a player, that it was a bot.
I growled after that.

Everybody has bad days bro.

Can you enter your nix-shell and run which espup?

which espup

Did you install espup via cargo at first? That might be the reason the toolchain installed is not in nix way.

❯ which espup
/nix/store/dz7xndhdm53618gikzr13r678gdyg6q7-espup-0.13.0/bin/espup

also run readelf on the cargo that having the issue like this:

❯ readelf -d ~/.rustup/toolchains/esp/bin/cargo

Dynamic section at offset 0x1ca6280 contains 31 entries:
  Tag        Type                         Name/Value
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000001e (FLAGS)              ORIGIN BIND_NOW
 0x000000006ffffffb (FLAGS_1)            Flags: NOW ORIGIN PIE
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000007 (RELA)               0x33b0
 0x0000000000000008 (RELASZ)             1766136 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffff9 (RELACOUNT)          73366
 0x0000000000000017 (JMPREL)             0x1b26a8
 0x0000000000000002 (PLTRELSZ)           4512 (bytes)
 0x0000000000000003 (PLTGOT)             0x1cafcd0
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000006 (SYMTAB)             0x338
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0x24d4
 0x000000000000000a (STRSZ)              3804 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x24b8
 0x0000000000000019 (INIT_ARRAY)         0x1b7a398
 0x000000000000001b (INIT_ARRAYSZ)       24 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x1b7a390
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000000000000c (INIT)               0x1b78750
 0x000000000000000d (FINI)               0x1b78770
 0x000000006ffffff0 (VERSYM)             0x2048
 0x000000006ffffffe (VERNEED)            0x22b4
 0x000000006fffffff (VERNEEDNUM)         4
 0x0000000000000000 (NULL)               0x0

And I agree nix is hard to grasp the concept, it need a long time to even know what you are doing, a lot of time there will be no documentation on how to do things, and you will be forced to read the source code from nixpkgs itself.

1 Like

Oh I know why I could not reproduce your issue, because I had nix-ld enabled. Just add

programs.nix-ld.enable = true;

to your configuration.nix or else you will need to patch the linker path like @Mephist0phel3s mentioned in his reply.

Okay I am either tripping or just witnessing black magic. Why does this work?!?!?! Can I somehow add this to my flake/shell so it is easier to reproduce? And can you or @Mephist0phel3s maybe explain why this works?

This is the shell it works with:

{ pkgs ? import <nixpkgs> {} }:
  let
    overrides = (builtins.fromTOML (builtins.readFile ./rust-toolchain.toml));
    libPath = with pkgs; lib.makeLibraryPath [
      # load external libraries that you need in your rust project here
    ];
in
  pkgs.mkShell rec {
    buildInputs = with pkgs; [
      clang
      # Replace llvmPackages with llvmPackages_X, where X is the latest LLVM version (at the time of writing, 16)
      espflash
      cargo-binstall
      espup
      llvmPackages.bintools
      rustup
      esp-generate
      cargo
    ];
    RUSTC_VERSION = overrides.toolchain.channel;
    RUSTUP_TOOLCHAIN = "esp";
    # https://github.com/rust-lang/rust-bindgen#environment-variables
    LIBCLANG_PATH = pkgs.lib.makeLibraryPath [ pkgs.llvmPackages_latest.libclang.lib ];
    shellHook = ''
      export PATH=$PATH:''${CARGO_HOME:-~/.cargo}/bin
      export PATH=$PATH:''${RUSTUP_HOME:-~/.rustup}/toolchains/$RUSTC_VERSION-x86_64-unknown-linux-gnu/bin/



        echo "Installing ESP toolchain..."
        espup install

      if [ -f "$HOME/export-esp.sh" ]; then
        echo "Sourcing export-esp.sh..."
        . $HOME/export-esp.sh
      fi
      '';
    # Add precompiled library to rustc search path
    RUSTFLAGS = (builtins.map (a: ''-L ${a}/lib'') [
      # add libraries here (e.g. pkgs.libvmi)
    ]);
    LD_LIBRARY_PATH = libPath;
    # Add glibc, clang, glib, and other headers to bindgen search path
    BINDGEN_EXTRA_CLANG_ARGS =
    # Includes normal include path
    (builtins.map (a: ''-I"${a}/include"'') [
      # add dev libraries here (e.g. pkgs.libvmi.dev)
      pkgs.glibc.dev
    ])
    # Includes with special directory paths
    ++ [
      ''-I"${pkgs.llvmPackages_latest.libclang.lib}/lib/clang/${pkgs.llvmPackages_latest.libclang.version}/include"''
      ''-I"${pkgs.glib.dev}/include/glib-2.0"''
      ''-I${pkgs.glib.out}/lib/glib-2.0/include/''
    ];
  }

Elf binary in linux that requires dynamic linking often hardcode the dynamic liner binary path in the meta data. You can use readelf - a | grep interpreter to see that the path is:

❯ readelf -a .rustup/toolchains/esp/bin/cargo | grep interpreter
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2] 
      # this path will work for fhs compliance system

The result is likely similar to /lib64/ld-linux-x86-64.so.2. This will work for other fhs compliance distro but nixos is not fhs compliance, meaning that the libraries are not installed at those location, including the linker.

The packages you installed from nixpkgs are already had the interpreter path patched, take bash for example you can see the linker point to somewhere in the nix store:

❯ readelf -a $(which bash) | grep interpreter
      [Requesting program interpreter: /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/ld-linux-x86-64.so.2]
      # this path will work for nixos but not other fhs compliance system

There is a great article explaining this:

What nix-ld do is not magic, it just put the liner at the expected location in the fhs lib directory. You can try find that directory before and after enabling nix-ld.

ls /lib64/

Patching the binary is basically telling the binary it to find the linker at other place hence also solve the issue.

Other dynamic library will also behave the same, you will often see runtime error about unable to load certain shared library is also caused by the same issue because the linker could not find the library at the rpath, hence patching the binary with additional rpath will solve the issue.

1 Like

The nix-ld option can only be enabled in your configuration file. If you use single configuration it is configuration.nix or if you use flake to configure your system, you will need to enable it inside your flake.

As far as I know you can’t just enable this inside your development environments using shell.nix or flake, because it is not intended to work this way.

The way of using rustup and espup to install a tool chain is already quite imperative, this setup is actually not very reproducible anyway

1 Like

Thank you so much for your help and explanation!

Just got out of bed, hit wrong button

1 Like

Sure.

And yes, it is black magic as i mentioned earlier. Been a linux user for nearly 20 years and nix is the most magical ive used yet and i love it

let overrides =

in
pkgs.mkShell rec {

the let and in in this statement more or less translate to english as
“let = take this context/attribute set and use it as context for in”

So when you build libPaths at the top of a drv or flake and use the let and in statements leading into the pkgs.mkShell rec {, the libPaths were built and since the pkgs.mkShell inclues the “rec” operator, this means that the function defined in that block can use the libPaths DURING despite the fact they havent ‘technically’ been built yet

But after taking another sip of coffee and double checking that, you didnt actually do that.

You enabled the ld-enable statement in your nixos config and it worked?

Ah yeah there it is

To add to this tho, even with the ld enabled, if you dont include any libs with it in your config, your linker will be active but it wont point to anything interesting except maybe stdc

if you dont plan to merge this into a public repo or what have you, and you just wanna do flakes or local configuring or whatever these are both options.

The nix way is to install it as a package via system packages, and you can do this with custom software packs actually.

Grab a copy of nixpkgs and nixos src, and you can set your own git repo as a channel for nixos and nixpkgs if you’d like to, so when you do a pull/push from the repo, you can run a nix-channel --update ; nixos-rebuild switch to pull down the recent commits and install them to your local bin cache.

This is how i did the War Thunder package actually, its not been merged yet but installed like:

pkgs.WarThunder under system packages in the main config.

just a thought.

Also, about the linker.

You can add runtime libs to prevent needing to link them at all in future by hand:

programs.nix-ld.enable = true;
programs.nix-ld.libraries = with pkgs; [
pkgs.xorg.libX11
pkgs.libgtkflow3
pkgs.libgtkflow4
pkgs.gtk3-x11
pkgs.gtk3
pkgs.libglibutil
pkgs.xorg.libXrandr
  ];
sudo -s 
nixos-rebuild switch

Will both activate the linker and add these libs to the system pack under the nix ld at the same time, you can add all your rust stuff here and run it right out of the tarball if its setup correctly.

Remember there are always about 50 ways to solve a problem, if you find yourself getting frustrated with tunnel vision, walk away for like half an hour and rethink your approach.
Sometimes you just need out of your own head to see the problem.

Its worth mentioning that doing the manual link method and creating a shell wrapper with the env variables set correctly makes it unnecessary to enable the ld.

DRV’s do the same nixify process during the build to make them executable so there was likely a step i didnt explain properly enough or a step you missed, and the wiki leaves much to be desired on this front.

EDIT: Yeah, seems i screwed something up in my original walkthrough since i also had the ld enabled. TODO: disable nix-ld and write a guide for nixifying bins by hand on the terminal
theoretically if the patchelf was done correctly within the proper shell env, you shouldnt have needed the ld at all using that walk through

According to the .dev book on the ld topic: Frequently Asked Questions — nix.dev documentation

NixOS cannot run dynamically linked executables intended for generic Linux environments out of the box. This is because, by design, it does not have a global library path, nor does it follow the Filesystem Hierarchy Standard (FHS).

There are a few ways to resolve this mismatch in environment expectations:

  • Use the version packaged in Nixpkgs, if there is one. You can search available packages at NixOS Search.
  • Write a Nix expression for the program to package it in your own configuration.There are multiple approaches to this:
    • Build from source.Many open-source programs are highly flexible at compile time in terms of where their files go. For an introduction to this, see Packaging existing software with Nix.
    • Modify the program’s ELF header to include paths to libraries using autoPatchelfHook.Do this if building from source isn’t feasible.
    • Wrap the program to run in an FHS-like environment using buildFHSEnv.This is a last resort, but sometimes necessary, for example if the program downloads and runs other executables.
  • Create a library path that only applies to unpackaged programs by using nix-ld. Add this to your configuration.nix:

programs.nix-ld.enable = true;

programs.nix-ld.libraries = with pkgs; [

Add any missing dynamic libraries for unpackaged programs

here, NOT in environment.systemPackages

];

Then run nixos-rebuild switch, and log out and back in again to propagate the new environment variables. (This is only necessary when enabling nix-ld; changes in included libraries take effect immediately on rebuild.)

Create a library path that only applies to unpackaged programs by using [nix-ld](https://github.com/Mic92/nix-ld). Add this to your configuration.nix: implies you didnt patch the bins correctly or drv on realization didnt patch them correctly either

1 Like