Packaging a bash script issues

I am (desperately) trying to package a very simple bash script in a given flake. To do so, I first tried following this tutorial, which didn’t work, saying it couldn’t find a command that I marked as a dependency.

Then, I asked in Nix chat and someone adviced me to use resholve, which seemed quite cool as it gets rid of any wrapping. However, it seems that my script is a bit too complex for resholve, because it has stuff like that

a_variable='a_var=$(acommand ...)'
bash -c "${a_variable}; ..."

and resholve doesn’t seem to be able to substitute properly acommand in that snippet (and yes, that kind of weird indirection is required; otherwise I would have simplified my life).

So I concluded I still had to do it the “hard” way, but this time with simpler nix functions that I understood. So I tried

runCommandLocal name {
  nativeBuildInputs = [ pkgs.makeWrapper ] ++ dependencies;
} ''
install -m755 ${./script.sh} -D "$out/bin/${name}"
patchShebangs "$out/bin/${name}"
wrapProgram "$out/bin/${name}" --prefix PATH : ${lib.makesBinPath dependencies}
''

where name is the target name of that script (ie. the command that you should type in a shell to launch it), say exscript, dependencies is the list of the dependencies of my bash script, which is currently [ bash nix findutils gnused ], and ./script.sh points to the actual script.
When I put that derivation in my flake, and build it, it produces a result/bin/exscript wrapper file and a result/bin/.exscript-wrapped file which contains the original script, except with its shebang modified.
Here is the content of the wrapper script, where I elided the hashes for readability sake.

result/bin/exscript
#! /nix/store/hash-bash-5.1-p16/bin/bash -e
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/hash-gnused-4.8/bin'':'/':'}
PATH='/nix/store/hash-gnused-4.8/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/hash-findutils-4.9.0/bin'':'/':'}
PATH='/nix/store/hash-findutils-4.9.0/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/hash-nix-2.9.1/bin'':'/':'}
PATH='/nix/store/hash-nix-2.9.1/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
PATH=${PATH:+':'$PATH':'}
PATH=${PATH/':''/nix/store/hash-bash-5.1-p16/bin'':'/':'}
PATH='/nix/store/hash-bash-5.1-p16/bin'$PATH
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
exec -a "$0" "/nix/store/hash-exscript/bin/.exscript-wrapped"  "$@" 

Now, my issue is that running my script simply with result/bin/exscript will result in a /nix/store/hash-exscript/bin/.exscript-wrapped: line 73: nix: command not found, even though I have listed nix as a dependency, and even though nix is added to the PATH in the wrapper script.
I have also tried simply copy-pasting that wrapper script somewhere else, making up a new script that simply runs nix --help and change the wrapper script to execute it. And that works. Now, I don’t know if that works because the wrapper script does its job, or simply because, well, nix is installed on my system (of course…), but in any case I have no more ideas on how to debug this situation.

Does .exscript-wrapped fiddle with the PATH?

1 Like

tl;dr: Just use pkgs.writeShellApplication. I sketch a full tutorial for packaging more complex shell scripts that you don’t write in-line, too, though.

Hrm, I have to say I’m not a big fan of that tutorial. It doesn’t explain much, and then ends up with a solution that is more complex than it needs to be. Let me give it a try.

The tutorial is right - simply running an arbitrary shell script in NixOS you will run into two problems:

  • Shebang will often point to /usr/bin or similar
  • Dependencies will not be available on $PATH, unless they’re already installed on the system and you run it from a shell that has $PATH set up correctly

This means we need to do two things:

  1. Make sure we patch the shebang of the script at build time
  2. Have some mechanism for setting up $PATH when the script runs

The former is trivial (but I’ll show you the idiomatic way of doing that too), but the latter isn’t.

The way setting up paths is generally done in NixOS is to “wrap” the script (as you attempt), such that we first run a script that basically does export PATH=${all the bin directories of our dependencies} and then executes the script in question.

There is an alternative too, though, and that is simply adding a header looking like this to the script:

#!${runtimeShell}
export PATH=${all the bin directories of our dependencies}

# rest of script here

This is how pkgs.writeShellApplication is implemented. pkgs.writeShellApplication unfortunately also adds set -euo pipefail, which may not be what you want, but if you did want the very simple way of doing this, here’s the documentation for it.

Personally I think it’s a bit unclean, and it’s better suited for simple scripts that you write in nix files, or that are very close to things you are implementing in a nix config.

Actually packaging a shell script is better through the normal packaging primitive, stdenv.mkDerivation. This is where we can use all those fancy makeWrapper and patchShebangs things.

This looks something like this:

# This derivation expects to be built with `pkgs.callPackage`.
#
# If you build it any other way, just refer to these via `pkgs.stdenv`
# and such.
{
  lib,
  stdenv,
  makeWrapper,
  curl,
}: let
  # Import from lib.
  inherit (lib) makeBinPath;
in
  # We use `rec` here so that we can reuse the `buildInputs` variable.
  #
  # While `rec` is considered an anti-pattern by some, in this case it
  # means that we can override the variable and everything will still
  # work.
  stdenv.mkDerivation rec {
    ## Metadata
    name = "test";
    version = "0.1";

    ## Source
    # This adds our shell script directly to the nix store. I'd advise
    # against doing this in practice, but rather using the tarball or
    # source directory that you're packaging. If we didn't import the
    # file directly, we could completely skip the `unpackCmd` below.
    #
    # If your script is tiny enough that you don't need all this
    # packaging, just use `writeShellApplication` instead, but this
    # derivation is written for educational purposes!
    src = ./test.sh;
    ##

    ## Dependencies
    #
    # See also https://nixos.org/manual/nixpkgs/stable/#variables-specifying-dependencies
    # Fair warning, this is all about cross-compilation, which you
    # hardly care about for shell scripts.
    #
    # `nativeBuildInputs` are the build inputs that will run on the
    # build host natively, and are expected to create files that will
    # run on the target host.
    #
    # The distinction is practically pointless for `makeWrapper`,
    # because it produces shell scripts that are target-independent, but
    # for completeness' sake I split it out.
    nativeBuildInputs = [makeWrapper];

    # These are the build inputs that are actually expected to run on
    # the target host.
    #
    # *Normally* nix will inspect the output, and make sure that any of
    # these whose paths end up in the output will also be installed as
    # runtime dependencies. Since we're packaging a shell script,
    # however, this doesn't work, because we rely on $PATH, so we need
    # to use `makeWrapper`.
    #
    # I still like putting `buildInputs` in these kinds of packages
    # because it's very explicit, and who knows, maybe one day nix will
    # be smart enough to propagate these kinds of dependencies.
    buildInputs = [curl];
    #
    # I get these wrong all the time, so if @Nobbz comes and scolds me,
    # sorry!
    ##

    ## Source unpacking
    # Nix will assume we are using a tarball of some sort by default,
    # and try to unpack the shell script, which obviously fails, so we
    # need to handle this ourselves.
    #
    # A real shell script we want to package probably comes with a
    # directory, likely bundled in a tarball, so this wouldn't normally
    # be necessary.
    #
    # See also https://nixos.org/manual/nixpkgs/stable/#ssec-unpack-phase
    unpackCmd = ''
      # $curSrc is the variable that contains the path to our source.
      mkdir test-src

      # We rename the file here, because when nix adds files to the
      # store it adds a hash, which obviously we don't want for our
      # shell script.
      cp $curSrc test-src/test.sh
    '';
    ##

    # Note that we don't have a build phase. You'd think that we would
    # need one to patch our shebang, but actually,
    # `stdenv.mkDerivation` patches all shebangs it can find by
    # default in the `fixupPhase` (which comes after the build phase),
    # so we don't have to worry about patching shebangs.
    #
    # See also https://nixos.org/manual/nixpkgs/stable/#ssec-fixup-phase

    ## Installation
    installPhase = ''
      # Before we wrap it, we need to actually install our script.
      #
      # For reference, -D creates leading directories, and m755 makes
      # it root-read-write-exec, all other users read-exec. This is
      # pretty standard for binaries, because this allows root to
      # easily delete the file if need-be.
      #
      # In practice, on NixOS, the file will be read-only for all
      # users, but it's the thought that counts.
      #
      # See also `man install`
      #
      install -Dm755 test.sh $out/bin/test.sh

      # This is where we create the wrapping script that sets PATH.
      #
      # Note the single quotes around our makeBinPath - after all, this
      # is just an argument to a binary executed by bash. While nix
      # paths can technically never result in splitting, this makes it
      # very explicit that we're putting something in bash args that may
      # need escaping otherwise.
      #
      # See also https://nixos.org/manual/nixpkgs/stable/#fun-wrapProgram
      #
      wrapProgram $out/bin/test.sh --prefix PATH : '${makeBinPath buildInputs}'
    '';
    ##
  }

Thorough explanation in comments :slight_smile:

You can create a test.sh script that requires curl, build that file with something like nix build --impure --expr 'with import (builtins.getFlake "nixpkgs") {}; pkgs.callPackage ./test.nix {}', and then run the script as ./result/bin/test.sh - or just import it, callPackage it, and add the derivation to your environment.systemPackages or whatever.

3 Likes

Not sure how much this will help if they have an existing script that is ignoring/discarding/fiddling the PATH.

1 Like

Ah, it is? I thought that was just the wrapper doing that. My bad!

1 Like

Hehe. We don’t know yet! Just making sure OP knows not to dive in to that before looking at the script. :heart:

1 Like

No it doesn’t :stuck_out_tongue: I don’t feel confident enough to do those things on NixOS. Basically my script is just a wrapper of nix flake new which adds more features to the templating mechanism of flakes, nothing crazy. It’s more an experiment to fiddle with Nix’ packaging.

Ahh, I like solutions that start with “just use [something]” and have nothing more :slight_smile:.

I just have a question about your derivation: during the unpack phase, you “unpack” the source to a directory called test-src, but you never mention that directory ever again. Is it the standard directory where you should unpack your stuff?

Besides that, thanks a lot for the thorough explanation, it’s very much appreciated, as there isn’t plenty of tutorials out there for this kind of things.


Also, I would be interested in the reason of the failure of my attempt. What did I do wrong? After all, it seems that basically the steps in your derivation are pretty much the same that my three lines of code (putting aside the patchShebangs that is apparently automatically done by mkDerivation), besides the unpack phase.

My end goal would be to make my own small repository where I could store all that stuff, but I’m clearly not there yet…

I’m just basing it on the fact that you said the script was too complex for resholve (and complex shell scripts do frequently fiddle with the PATH), and cite dynamic behavior that you can’t simplify:

a_variable='a_var=$(acommand ...)'
bash -c "${a_variable}; ..."

This latter part is a big factor in why I wondered if something is fiddling (or setting/resetting) the PATH. From what we can see, at least:

  • it does look like you’re holding wrapProgram correctly
  • your hand-test of the exports and then running Nix seems to confirm this as well

I’m not familiar enough with the flake commands to know if this could be happening internal to that process? Maybe using set -x in the script would help answer that narrow question.

1 Like

No, the documentation I link goes over this if you want more detail - and this is pretty easy to experiment with if you want to get a feel for it. It uses whatever directory it finds, and if there are multiple it will scream at you and you’ll have to explicitly tell it which one to use.

I concur though, your use of these things looks correct, I mostly jumped on the following tutorial that didn’t work bit, and wanted to show some useful ways of doing it that aren’t what I consider a flawed tutorial.

One note, your snippet says lib.makesBinPath, which is clearly a typo. Did you copy verbatim or is our assessment of “looks right” subject to copy-paste gremlins?

1 Like

Yeah–I just guessed this must be a typo since it actually generated a correct-looking wrapper.

1 Like

I wondered if it was my script that was flawed, so I removed almost everything and left just the nix call, and it worked.
Then I realized: I am a colossal idiot. I did not mean to fiddle with the PATH, but I ended up generating a variable whose name is only PATH without any prefix or suffix, and setting that to just the path of a file I am creating… :man_facepalming:


I also tested with writeShellApplication, but now shellcheck is preventing it to build, as it sees stuff like a_variable='a_var=$(acommand ...)' and warns me about the fact that, if I use single quotes, it won’t be expanded (which I know, and I want). Is there a way to nix to build anyways, or do I have to turn off shellcheck entirely (by overwriting the checkPhase argument)?

EDIT: never mind, I found it in the shellcheck documentation.

1 Like

And so the mystery is solved. Good call, @abathur :slight_smile: