/bin/sh while using nix package manager (not on nixos)

Help
1

Hello – Is there somehow a way to short-circuit #!/bin/sh from with a nix environment?

I have a project that builds on NixOS. However i am also trying to get it to build on Nix the package manager (upon an ubuntu base).

The build process creates scripts which execute #!/bin/sh, which does exists on the base OS, and that ends up loading a glibc which is incompatible with the rest of the project and i end up getting the following error
/bin/sh: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_ABI_DT_RELR’ not found (required by /nix/store/ylaxayp4xlq28x6iv3l05gnl3s5dxmr9-glibc-2.37-8/lib/libdl.so.2)`

The solution i have found, is that once the build fails. i go back into the generated script and change the #!/bin/sh to something like #!/nix/store/....-bash-5.2-p15/bin/sh

Is there a better solution here for the workflow, like short-circuit the call to /bin/sh or blacklist /bin/sh within a nix env?

thanks!

2

Not clear what you mean by “a nix environment”.

Is this part of a nix build, or in a nix-shell/nix shell?

3

Is there some LD_PRELOAD involved in your derivation? That’s the only reason I can immediately think of for glibc version incompatibilities when running the non-Nix /bin/sh.

IMO LD_PRELOAD should be avoided as much as possible and replaced with proper rpath editing (via e.g. patchelf, or -Wl,-rpath at link time).

5

I am not using LD_PRELOAD, at least not directly, its not in my flake.nix file.

Just to clarify, this builds fine directly on NixOS, but not using Nix on top of Ubuntu. So i believe when #!/bin/sh is invoked, it loads glibc, then when other dependencies that are invoked that were built as part of the flake, they are expecting the glibc version from Nix, but instead fine the one from ubuntu. When i replace #!/bin/sh with the path for sh as installed by nix, it will all compile. I was just wondering if there was a more proper way to avoid #!/bin/sh.

6

How are you using the flake when you see the problem? What nix command(s) do you run? How do you do the below?

replace #!/bin/sh with the path for sh as installed by nix

7

Yes, you are correct replacing the path works. This is a generated script during the build process, so i’d need to let the build process fail, and then adjust the paths. This is a work-around that works. I was wondering if there was a better way to solve the problem.

8

We don’t know what you’re doing, here, nor how you are running it, and these details affect the best answer. I’m asking for specifics so that we can help.

What nix commands are you running? Is the flake public (and if so can we have a link)? What exact code are you using to replace the /bin/sh shebang?

9

Sorry, my apologies, i wrote this up to replay to your originally post, but accidentally deleted it. I will try to address all your questions here.

Not clear what you mean by “a nix environment”.

To answer your original question about the environment. (Pardon incorrect terminology). I have a base OS of Ubuntu 22.04. I have multi-user nix installed on top of Ubuntu. I enter the source directory and source/prep the build environment using the following command: nix flake --experimental-features 'nix-command flakes'. This project is based on buildroot, and if you’re nor familiar with buildroot, the build process involves many scripts and checkouts from other repos during the build process, so it is difficult to patch files once the build process has started (at least to my knowledge).

Is this part of a nix build, or in a nix-shell/nix shell?

I presume the latter, nix-shell/shell since it uses flakes, but not 100% sure on the terminology. I am running nix flake not nix shell.

Is the flake public (and if so can we have a link)?

The flake is not public, i might be able to trim it down by using a trivial buildroot project.

What exact code are you using to replace the /bin/sh shebang?

The process i currently run to get around this shebang issue: I let the process build as normal. At some point it fails with

/bin/sh: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_ABI_DT_RELR’ not found (required by /nix/store/ylaxayp4xlq28x6iv3l05gnl3s5dxmr9-glibc-2.37-8/lib/libdl.so.2)

I believe this happens b/c fs/common.mk:75 in buildroot generates a script to run fakeroot, and that script calls #!/bin/sh directly which then casues issues. The workaround is to find the sh installed by nix, so i run the following while in the nix flake env:

» which sh
/nix/store/rhvbjmcfnkg8i2dxpzr114cp1ws7f667-bash-5.2-p15/bin/sh

I edit the fs/common.mk file and replace #!/bin/sh with #!/nix/store/rhvbjmcfnkg8i2dxpzr114cp1ws7f667-bash-5.2-p15/bin/sh

I re-run the build process, and everything builds, links and it happy.

Other Notes: This process runs fine without the workaround if the native OS is NixOS since you can symlink /run/current-system/sw/bin/sh → /bin/sh.

I hope this gives more insight into the environment and process. apologies for not be more clear about it previously.

thanks!

10

Can you confirm that this is the exact and complete command you’re running? I’m pretty sure any invocation of nix flake requires a subcommand and there isn’t one in this invocation.

11

you are correct. i am using nix --experimental-features 'nix-command flakes' develop. I forgot about that subcommand as i am using direnv to evaluate .envrc which must hide that detail for i forgot about it.

for your reference:

# .envrc contents
use flake . --experimental-features 'nix-command flakes'
12

This is very relevant; buildroot will set up a very complex build environment, and probably does some cross compilation in there somewhere too. It’s designed to produce an environment with its specific libc and everything, just as nix; it’s no wonder they’re getting into each others’ hairs.

This will be pretty much impossible to debug by proxy without direct access to the codebase. Even with, a consultancy would probably charge you a lot of money for it.

That’s not to say that nix and buildroot are incompatible, but you’ll need a careful hand from someone experienced with your project to carve out a clean build environment in it.

Or at the very least a lot more detail about what you’re actually trying to do, besides running sh, nix and buildroot in some indeterminate order.

Amusingly, this is exactly why that is a bit controversial; It can make builds non-reproducible :slight_smile:

It’s pretty much impossible to know what’s wrong without looking at this script you’re executing. I wouldn’t be surprised if the problem is indeed an LD_PRELOAD set by buildroot somewhere, and ultimately has almost nothing to do with nix.

13

Yes it is quiet the beast! It does do cross compiles. However with just the modification i described above (which i have no condenced into NIX_BIN_SH=which sh sed -i s@#\!/bin/sh@#\!$NIX_BIN_SH@ ./fs/common.mk). It builds successfully and the packages function.

If LD_PRELOAD is getting set, would i be able to catch that using env directly after the invocation of #!/bin/sh? Grepping through the code i don’t see much se of LD_PRELOAD, but it is a large and complex build env, dragons could be anywhere.

In general, there doesn’t appear to be any straightforward way to address the problem which is probably a good enough answer. I doesn’t appear that i’m missing anything obvious which is really what i was after since i’m still learning nix. i do have a workaround in place, so i will continue to use that. (But i got so close to getting it to build as is!).

thanks for everyone’s comments!