How do I compile a Rust binary on NixOS that works on other Linux distros?

I’m not using carnix or buildRustPackage, I just have a normal stdenv.mkDerivation in default.nix that up until now has worked well. It gets me Rust, Cargo, the Diesel CLI, and dependencies like GStreamer without much fuss or without deviating much from normal Rust workflow. My project builds with normal cargo build --release without issue and runs fine on the NixOS machine it was built on.

with import <nixpkgs> { };

stdenv.mkDerivation {
  name = "music-dev-env";
  buildInputs = [
    cargo
    cargo-outdated
    diesel-cli
    glib
    gst_all_1.gstreamer
    gst_all_1.gst-plugins-base
    openssl
    pkg-config
    sqlite
  ];
}

However when I copy the built binary to an Arch Linux machine I get No such file or directory when trying to run it. Obviously the file is actually there so I think this is due to the loader. Building on the Arch machine works and that binary runs fine. I notice the built binary from NixOS references /nix/store which of course won’t be there unless Nix is installed.

> $ ldd music                                                                                                                                                                                                                              
	linux-vdso.so.1 (0x00007ffc345c3000)
	libgstpbutils-1.0.so.0 => /usr/lib/libgstpbutils-1.0.so.0 (0x00007f4ac1b5a000)
	libgstreamer-1.0.so.0 => /usr/lib/libgstreamer-1.0.so.0 (0x00007f4ac1a17000)
	libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0 (0x00007f4ac19bf000)
	libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007f4ac1896000)
	libssl.so.1.1 => /usr/lib/libssl.so.1.1 (0x00007f4ac1804000)
	libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0x00007f4ac1527000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f4ac1360000)
	/nix/store/aqq6367snc1zh3fs1pc4j4zm5h80vkkz-glibc-2.31/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f4ac2d6e000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f4ac135a000)
	libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x00007f4ac121f000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f4ac11fd000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f4ac11e1000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f4ac109c000)
	libgstvideo-1.0.so.0 => /usr/lib/libgstvideo-1.0.so.0 (0x00007f4ac0ff4000)
	libgstaudio-1.0.so.0 => /usr/lib/libgstaudio-1.0.so.0 (0x00007f4ac0f7a000)
	libgsttag-1.0.so.0 => /usr/lib/libgsttag-1.0.so.0 (0x00007f4ac0f3c000)
	libgstbase-1.0.so.0 => /usr/lib/libgstbase-1.0.so.0 (0x00007f4ac0ec0000)
	libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0x00007f4ac0eb9000)
	libunwind.so.8 => /usr/lib/libunwind.so.8 (0x00007f4ac0e9f000)
	libdw.so.1 => /usr/lib/libdw.so.1 (0x00007f4ac0e01000)
	libffi.so.7 => /usr/lib/libffi.so.7 (0x00007f4ac0df5000)
	libpcre.so.1 => /usr/lib/libpcre.so.1 (0x00007f4ac0d83000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f4ac0d69000)
	liborc-0.4.so.0 => /usr/lib/liborc-0.4.so.0 (0x00007f4ac0ce2000)
	liblzma.so.5 => /usr/lib/liblzma.so.5 (0x00007f4ac0cba000)
	libelf.so.1 => /usr/lib/libelf.so.1 (0x00007f4ac0ca0000)
	libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007f4ac0c8d000)
> $ readelf -a music | grep interpreter
      [Requesting program interpreter: /nix/store/aqq6367snc1zh3fs1pc4j4zm5h80vkkz-glibc-2.31/lib/ld-linux-x86-64.so.2]

I figured OK, maybe carnix does some magic to avoid this, I’ll just give in to what’s in the Nixpkgs manual and use that. But I ran into this issue.

I did more searching and found someone with the exact same issue, and the fix was to move to static linking (which for Rust seems to mean using musl). But when I tried that it failed on building cdparanoia (a dependency of gst-plugins-base.

❯ nix-build '<nixpkgs>' --arg crossSystem '{ config = "x86_64-unknown-linux-musl"; }' -A cdparanoia
these derivations will be built:
  /nix/store/cnx2biisdalb5d2ijj4qkhrgjag367cm-cdparanoia-III-10.2-x86_64-unknown-linux-musl.drv
building '/nix/store/cnx2biisdalb5d2ijj4qkhrgjag367cm-cdparanoia-III-10.2-x86_64-unknown-linux-musl.drv'...
unpacking sources
unpacking source archive /nix/store/xwzjiyay9rkgr4vrnlhz9v1dkk1xs1q7-cdparanoia-III-10.2.src.tgz
source root is cdparanoia-III-10.2
setting SOURCE_DATE_EPOCH to timestamp 1221168321 of file cdparanoia-III-10.2/debian/rules
patching sources
applying patch /nix/store/dm271zf5jh357wa43l1yh7z4rsb5j27k-utils.patch
patching file interface/utils.h
patching file utils.h
applying patch /nix/store/j6a2glhywhbiia1cfrp91qgqg7nr3b6j-fix_private_keyword.patch
patching file interface/cdda_interface.h
patching file interface/cooked_interface.c
patching file interface/interface.c
patching file interface/scan_devices.c
patching file interface/scsi_interface.c
Hunk #39 succeeded at 1626 (offset 4 lines).
Hunk #40 succeeded at 1674 (offset 4 lines).
Hunk #41 succeeded at 1743 (offset 17 lines).
patching file interface/test_interface.c
updateAutotoolsGnuConfigScriptsPhase
configuring
configure flags: --prefix=/nix/store/071km90a59952gbbhsx28468xjrjwg7l-cdparanoia-III-10.2-x86_64-unknown-linux-musl --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-musl
checking build system type... x86_64-unknown-linux-gnu
checking host system type... Invalid configuration `x86_64-unknown-linux-musl': machine `x86_64-unknown-linux' not recognized
configure: error: /nix/store/j8vysakw78bpgngba32hfwwikqda9yx2-bash-4.4-p23/bin/bash ./config.sub x86_64-unknown-linux-musl failed
builder for '/nix/store/cnx2biisdalb5d2ijj4qkhrgjag367cm-cdparanoia-III-10.2-x86_64-unknown-linux-musl.drv' failed with exit code 1
error: build of '/nix/store/cnx2biisdalb5d2ijj4qkhrgjag367cm-cdparanoia-III-10.2-x86_64-unknown-linux-musl.drv' failed

So how do I build the binary on NixOS in a way that it’s generally portable for Linux? I think static linking would be ideal, but I’d be happy at least just not having ld-linux-x86-64.so.2 pointing to the Nix store. I feel like this is the opposite of the more common problem, which is running binaries built on “normal” Linux on NixOS and having to use patchelf or something.

3 Likes

Use patchelf to set any interpreter you want. On most other linuxes /usr/lib64/ld-linux-x86-64.so.2 should do.

3 Likes

It seems that needs to be run on the Arch machine, not the NixOS one, right? In which case, I guess the answer to how to compile on NixOS for “common” Linux is ultimately is either 1) cross-compiling or 2) compiling in Docker?

patchelf --set-intepreter /usr/lib64/ld-linux-x86-64.so.2 target/release/music
patchelf: getting info about '--set-intepreter': No such file or directory

It is --set-interpreter, the p beeing surrounded by rs.

And it has to be done on nixOS, it usually has patchelf available, other Linux distros rarely have it packaged at all…

Oh duh! Sorry, the “no such file” threw me off and Arch does have patchelf in fact.

To confirm, this works: patchelf --set-interpreter /usr/lib64/ld-linux-x86-64.so.2 target/release/music.

Thank you for the help!

I ran into quite a lot of problems attempting to build a static rust binary from NixOS (with segfault after manually changing the interpreter)

The best solution I found is to build into this Docker image : GitHub - emk/rust-musl-builder: Docker images for compiling static Rust binaries using musl-libc and musl-gcc, with static versions of useful C libraries. Supports openssl and diesel crates.

Just in case someone finds this useful ! :wink: