32 bit binaries env

1

I want to run and debug a 32-bit binary on a 64-bit machine.

I can successfully run it with steam-run ./binary.

But I want to run it directly like ./binary. (this is needed because I need to debug it with a library that doesn’t do follow-fork-mode child so it debugs steam-run)

Nix-ld

I have nix-ld enabled. I saw here that I need to patch nixpkgs to have nix-ld support other systems, but I’m not sure how to do it. Maybe nixpkgs.overlay, but I tried searching how the contributer does it in its nixos-config, but he just uses the nix-ld.systems option without the patch it seems to me.

Nix shell

Apart from nix-ld I checked out steam-run and it seems to use bwrap and fhsEnv, so I was thinking:

how do I create a nix shell where I specify the 32 bit loader and the 32 bit libraries I need and then enter it and just run the binary?

i386 nixpkgs

My binary depends on /lib/ld-linux.so.2 and libm.so.6.

I tried downloading the loader manually and using it

> ./libc6-i386/usr/lib32/ld-linux.so.2 ./binary
./binary: error while loading shared libraries: libm.so.6: cannot open shared object file: No such file or directory

but then I couldn’t find the 32 bit version of libm with nix-locate:

> nix-locate libm.so.6
rocmPackages.clang.out                                0 s /nix/store/khd9h2x0h08sipn0hpdzf3d4k27zlphx-rocm-toolchain/lib/libm.so.6
glibc_multi.static                                    0 s /nix/store/gmbmw0npjz8f952pbmb9q5qy3fxvdj52-glibc-multi-2.40-66-static/lib/libm.so.6
glibc_multi.static                                    0 s /nix/store/gmbmw0npjz8f952pbmb9q5qy3fxvdj52-glibc-multi-2.40-66-static/lib64/libm.so.6
glibc_memusage.debug                                  0 s /nix/store/kh5fhigrhgsd3yazbdl2s0k6fgws1sgi-glibc-gd-2.40-66-debug/lib/debug/libm.so.6
glibc.out                                     1,029,504 x /nix/store/xx7cm72qy2c0643cm1ipngd87aqwkcdp-glibc-2.40-66/lib/libm.so.6
glibc_memusage.out                            1,029,504 x /nix/store/hmkdsnrj4zbk05y7kxc5rkwsy2jsx7l5-glibc-gd-2.40-66/lib/libm.so.6
glibc.debug                                           0 s /nix/store/5l3npiqzwglfxxvv1mdrkyf9rqr5mdqf-glibc-2.40-66-debug/lib/debug/libm.so.6
glibc_multi.out                                       0 s /nix/store/l7jr2nsnmr380cwya9mszzw3bh07rnfd-glibc-multi-2.40-66/lib/libm.so.6

This don’t work for example:

> LD_LIBRARY_PATH=/nix/store/l7jr2nsnmr380cwya9mszzw3bh07rnfd-glibc-multi-2.40-66/lib/:$LD_LIBRARY_PATH ./libc6-i386/usr/lib32/ld-linux.so.2 ./binary
./binary: error while loading shared libraries: libm.so.6: wrong ELF class: ELFCLASS64

i686pkgs

I also saw somewhere pkgs.pkgsi686Linux but on search https://search.nixos.org/ it doesn’t appear. What is going on? Are the 32 bit version packages in there?

2

My setup is confusing, because I run nix flake update –-override-input nixpkgs github:KoviRobi/nixpkgs/rob-nix-ld-32bit – I really should just update flake.nix to avoid confusion

But it’s using this branch GitHub - KoviRobi/nixpkgs at rob-nix-ld-32bit which I merge nixpkgs-unstable into occasionally (merge not rebase so previous hashes in lockfile are valid).

This is the commit you want GitHub - KoviRobi/nixpkgs at nix-ld-32bit – but let me refactor things a bit so that I just add an extra module on my nixos config side and can track upstream normally.

(I didn’t really mind the merges etc as sometimes I also just picked other commits into nixpkgs-unstable that hadn’t yet landed but fixed things that I needed. I guess this is the software equivalent of having a dodgy door that requires a knack to open, getting so used to this that I barely notice it, then realising it when someone new comes along and is confused by my weird ways.)

I really should spend some time and get that PR fixed properly, just finding the time/remembering it exists…

1 Like
3

No problem at all, I completely understand. Thanks for taking the time to explain it.

Unfortunately this doesn’t quite work for me as-is. Could it be you forked a too much older nixpkgs?

Anyway I just did:

nixpkgs.follows = "kovirobi";
kovirobi.url = "github:KoviRobi/nixpkgs?branch=nix-ld-32bit";

Then nix flake update and nixos-rebuild switch and I got:

error:
       … while calling the 'seq' builtin
         at /nix/store/k38k0x7adns7glmv4z6iw7gshrg1xxc4-source/lib/modules.nix:333:18:
          332|         options = checked options;
          333|         config = checked (removeAttrs config [ "_module" ]);
             |                  ^
          334|         _module = checked (config._module);

       … while evaluating a branch condition
         at /nix/store/k38k0x7adns7glmv4z6iw7gshrg1xxc4-source/lib/modules.nix:287:9:
          286|       checkUnmatched =
          287|         if config._module.check && config._module.freeformType == null && merged.unmatchedDefns != [] then
             |         ^
          288|           let

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: function 'submoduleWith' called with unexpected argument 'class'
       at /nix/store/k38k0x7adns7glmv4z6iw7gshrg1xxc4-source/lib/types.nix:695:7:
          694|     submoduleWith =
          695|       { modules
             |       ^
          696|       , specialArgs ? {}

Even without setting programs.nix-ld.systems.

I’ll try to maybe merge nixpkgs to your branch. Or if I can figure out how to overlay nixpkgs modules I’ll do that with your PR.

4

I don’t know what the error is, I wonder if it’s because I’m using unstable and you aren’t. But I got around to making it a standalone module until it gets merged properly, you want to do something like

1 Like
5

Yeah ok thanks that will work for sure. I don’t know why it didn’t but I was using unstable btw.

update 2:
Yes now I can access the new module thanks.

Almost there! It now errors out:

./binary
[nix-ld] FATAL: panicked at src/main.rs:187:55:
called `Result::unwrap()` on an `Err` value: Posix(2)
Aborted                    (core dumped) ./binary

it’s almost comical. (RUST_BACKTRACE=full does nothing different)

update 3:
The source code shows:

update 4:
Ok obviously I did something wrong because it doesn’t find the nix-ld loader:

strace ./binary
execve("./binary", ["./binary"], 0x7ffe59441110 /* 87 vars */) = 0
[ Process PID=190310 runs in 32 bit mode. ]
mmap2(NULL, 124, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xf7a76000
openat(AT_FDCWD, "/run/current-system/sw/share/nix-ld/lib/ld.so", O_RDONLY) = -1 ENOENT (No such file or directory)

update 5:
My config is now the same as @KoviRobi:

So I think it is right to have the following:

ls /run/current-system/sw/share/nix-ld*
/run/current-system/sw/share/nix-ld-i686-linux:
lib

/run/current-system/sw/share/nix-ld-x86_64-linux:
lib

But I don’t know why the 32 bit package doesn’t start. I mean I know why, I don’t know how to tell nix-ld to use nix-ld-${system} path depending of the binary architecture