/bin/cp cannot be found when using nix-build

gluonix · June 18, 2021, 2:50am

I am working with a codebase which uses /bin/cp in their Makefile and when using nix-build /bin/cp cannot be found whereas when using nix-shell or a nix docker image it’s not a problem.

My current workaround is to modify the Makefile so that it simply uses cp instead of /bin/cp but let’s say I would not be able to do so, is there a better way to handle this?

I assume the environment in which nix-build operates is somewhat different than the one of nix-shell and of the docker image. I am guessing some environment variables or profile related configuration are missing …

I thought perhaps of trying to symlink /bin/cp to the result of which cp but it seems to me somewhat convoluted.

jonringer · June 18, 2021, 4:49am

honestly, /bin/cp should not be used.

However, NixOS doesn’t have anything in /bin except sh to make scripts not super painful. Inside of the sandbox, not even /bin/sh exists.

gluonix · June 18, 2021, 5:46am

Ok, thanks. That’s what I was suspecting.

abathur · June 18, 2021, 6:22am

If it’s just the one command, or a small number, you can usually just use one of the substitute* functions to swap in either the absolute cp path or a bare cp from prePatch? If it’s complicated (say, there’s no good way to substitute without over-matching) you can also write a small patch for it and add it to patches = [];.

I’ve been working on GitHub - abathur/resholve: a shell resolver? :) to address the same general problem (i.e., fixing executable paths) for bash/shell scripts. If this was a Big Problem for you it might be possible to torture a Makefile into something shell-like-enough that the shell-specific parser resholve piggybacks on wouldn’t barf (more of a first-step…)

gluonix · June 18, 2021, 9:03pm

Thanks @abathur. It’s good to be aware of these different solutions. For now, I think I’ll just try to get the Makefiles modified to that they just use cp instead of /bin/cp, but I’ll keep your suggestions in mind.

abathur · June 18, 2021, 10:17pm

It’s a good idea to upstream things when there’s no downside for the project. But it’s also very common to need little tweaks like this and upstream isn’t always receptive/responsive (or it may cause them problems).

I grepped around Nixpkgs and see hundreds of them. In fact, I stumbled on one patching /bin/cp out:

github.com

NixOS/nixpkgs/blob/847fa4931bd699480ab7b901f92fb3a0696339d1/pkgs/development/compilers/mlton/20130715.nix#L49

    
      
          
          
buildInputs = [ gmp ];
          nativeBuildInputs = lib.optional stdenv.isLinux patchelf;
          
          
makeFlags = [ "all-no-docs" ];
          
          
configurePhase = ''
            # Fix paths in the source.
            find . -type f | grep -v -e '\.tgz''$' | xargs sed -i "s@/usr/bin/env bash@$(type -p bash)@"
          
          
  substituteInPlace $(pwd)/Makefile --replace '/bin/cp' $(type -p cp)
            substituteInPlace bin/mlton-script --replace gcc cc
            substituteInPlace bin/regression --replace gcc cc
            substituteInPlace lib/mlnlffi-lib/Makefile --replace gcc cc
            substituteInPlace mlnlffigen/gen-cppcmd --replace gcc cc
            substituteInPlace runtime/Makefile --replace gcc cc
            substituteInPlace ../${usr_prefix}/bin/mlton --replace gcc cc
          
          
  # Fix paths in the binary distribution.
            BIN_DIST_DIR="$(pwd)/../${usr_prefix}"
            for f in "bin/mlton" "lib/mlton/platform" "lib/mlton/static-library" ; do

(Though you would probably do this in prePatch, instead of configurePhase.)

gluonix · June 18, 2021, 10:41pm

Ah thanks! That is very helpful! I am now running into a more substantial issue:

/usr/bin/env: bad interpreter: No such file or directory

The upstream code I am writing a nix derivation for uses the shebang /usr/bin/env bash multiple times. I am currently searching on how to handle this. Just like with /bin/cp this problem only happens when using nix-build on my host machine. When using a docker image based on Docker Hub, it works fine.

I’d like to better understand what causes this different behavior, is this solely the sandbox configuration?

I see that the docker image has the following:

github.com

nix-community/docker-nixpkgs/blob/50b4d08da770e0f7924053003fef1778a87c242f/images/nix/default.nix#L38-L48

    
      
          extraCommands = ''
            # for /usr/bin/env
            mkdir usr
            ln -s ../bin usr/bin
          
          
  # make sure /tmp exists
            mkdir -m 1777 tmp
          
          
  # need a HOME
            mkdir -vp root
          '';

Using the same trick as for /bin/cp appears to work, e.g.:

substituteInPlace foo.sh --replace '/usr/bin/env bash' $(type -p bash)

gluonix · June 19, 2021, 12:27am

Using patchShebangs resolves the issue with:

/usr/bin/env: bad interpreter: No such file or directory

abathur · June 19, 2021, 12:45am

It’s the same basic issue, but there’s a shebang-specific function: patchShebangs.

You should be able to use patchShebangs <filename|dirname> depending on how broadly you want it to run.

I’m not sure about the docker image specifically, but nix-build will generally be more strict. It’ll ultimately come down to what is available in the sandbox.

When you use nix-shell, the existing environment is generally allowed in unless you also use the --pure flag. nix-shell is more focused on creating an environment for humans to iterate on packaging something, and it’s a pain to have to constantly exit and modify the shell to add packages for utilities the human needs to run but which aren’t otherwise package dependencies. Unfortunately, that extra wiggle room means you can have unspecified dependencies that are only working because outside paths are leaking in.

abathur · June 19, 2021, 12:46am

Ah. Beat me to the punch.

gluonix · June 19, 2021, 3:06am

Thanks for the explanations; that is useful. I am learning to use nix so it’s useful to read such explanations.