How do I build a binary on NixOS that I can run on other distros?

Help
1

Hello!

So, I wrote a trivial program and built it with clang++ to get a trivial executable. Unfortunately I cannot run this binary on other distros because all the dynamic libraries and the elf interpreter point to nix-store locations.

I was wondering what is necessary to be able to run this binary on other systems (e.g. on a typical linux system like Ubuntu).

Any idea would be appreciated. Thank you!

The trivial program in question:

/tmp $ cat a.cc
int main() {
  return 0;
}
/tmp $ clang++ a.cc -o a.out
/tmp $ ldd a.out
        linux-vdso.so.1 (0x00007fff295fc000)
        libm.so.6 => /nix/store/1yvpgm763b3hvg8q4fzpzmflr5674x4j-glibc-2.32-10/lib/libm.so.6 (0x00007f758748f000)
        libgcc_s.so.1 => /nix/store/1yvpgm763b3hvg8q4fzpzmflr5674x4j-glibc-2.32-10/lib/libgcc_s.so.1 (0x00007f7587475000)
        libc.so.6 => /nix/store/1yvpgm763b3hvg8q4fzpzmflr5674x4j-glibc-2.32-10/lib/libc.so.6 (0x00007f75872b5000)
        /nix/store/1yvpgm763b3hvg8q4fzpzmflr5674x4j-glibc-2.32-10/lib/ld-linux-x86-64.so.2 => /nix/store/m0xa5bz7vw7p43wi0jppvvi3c9vgqvp7-glibc-2.32-25/lib64/ld-linux-x86-64.so.2 (0x00007f75875d4000)
2 Likes
2

You could build a statically linked binary I think with --static.

3

The experimental nix command can do this:

Usage: nix bundle FLAGS... INSTALLABLE?

Summary: bundle an application so that it works outside of the Nix store.

Examples:

  To bundle Hello:
  $ nix bundle hello
2 Likes
5

Thanks! nix bundle does not seem to work at all for me. But I found https://github.com/matthewbauer/nix-bundle which works nicely for binaries that are already in the nix-store! Unfortuantely so far my binaries are just locally built binaries without any packaging; but perhaps creating a dummy package for the binary would work!

6

Thanks! Unfortunately I don’t think making static binaries are always feasible (for example, the actual binary I’m trying to port depends on SDL2, which does not seem to have a static library in the store at all).

7

nix bundle is an experimental feature, so I’m not sure how it failed for you, but you can enter a shell with the proper nix here:

# from project root

nix-shell -I nixpkgs=channel:nixos-20.09 --packages nixFlakes
alias nix="nix --experimental-features 'nix-command flakes'"
nix flake init

# edit flake.nix and import your existing default.nix as a package

nix bundle ".#nameOfYourPackage"
8

I guess that nix-bundle involves some sort of self extraction… Isn’t it possible to simply compile using the usual loader in /lib? Otherwise I guess it should be possible to patch the elf file but it sounds quite dirty…

9

It’s one of those fancy self-extracting zips that act as binaries: nix bundle

I’m not certain how it’s done, it uses some very intetesting namespace stuff I think, but it’s similar to an appimage and doesn’t require extraction.

10

I see, but in simpler cases one may want to produce a standard simple file. For instance to keep the file small and simple, to avoid unnecessary bugs potentially coming from the namespace (like I don’t think you can run setuid apps in namespaces and nesting namespaces can be a problem), to have a simpler debugging procedure (would gdb interact nicely with nix bundle?) and more.

11

Oh, I don’t disagree, but it’s the best possible generic solution. Nix fundamentally links things to /nix/store, which isn’t present on other distros.

That, or static linking. Maybe you can do something with buildFhsUserEnv. And patchelf, of course.

12

also dynamic linking is needed to build plugins

13

sorry, i don’t have many nix, but on guix building in fhs doesn’t build for fhs

$ guix shell --container gcc-toolchain --emulate-fhs -- gcc hello.c -o hello
$ ./hello 
hello
$ ldd ./hello
	linux-vdso.so.1 (0x00007fff8b3e9000)
	libgcc_s.so.1 => /gnu/store/4zvswpr2h3b7dvqpvjcdam8vfhyjrmgl-gcc-12.2.0-lib/lib/libgcc_s.so.1 (0x00007fb58adba000)
	libc.so.6 => /gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/libc.so.6 (0x00007fb58abf8000)
	/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fb58addd000)

and i cannot run it in a sandbox

$ LANG=C.utf8 flatpak run --filesystem="$HOME" --cwd=. org.gnome.Platform/x86_64/43 -c ./hello
/bin/sh: line 1: ./hello: No such file or directory
14

How about generating an OCI container and then run that with podman? Any relatively recent distro should be able to run that.

15

I think the best option is patchelf --set-interpreter /usr/lib/ld-linux.so, or whatever the interpreter path is on traditional distros.

That doesn’t mean it will work, of course, there’s no guarantee the dependencies will be the same. But if you don’t want to create bundles, not much else you can do.