Can I package a shell script without rewriting it?

I have just installed NixOS and have a bunch of shell scripts that I would like to install. I’m wondering how best to do it without having to edit them to adapt the paths to various executables in them, or if there is an automatic way to patch them while building the derivation.

First of all, I suppose I can still use my scripts without having to package them with nix. I used to just have my scripts in my home directory and set my PATH to point to them. I think I can keep it that way in nix.

But I guess the nix approach would be to create a derivation such that my scripts are installed into the nix store. I’m wondering which way I should chose and why. The nix way gives the advantage of easy reproducibility: once packaged, I’ll be sure that my scripts will always be installed along with the appropriate dependencies.

However, this requires changing the paths in my scripts to the various executables and other files that are used. For example, if a script uses the program hello, I must make sure that the packaged script will use the right hello, i.e. put something like $out/bin/hello, and let nix set the environment variable $out to refer to the right path in the nix-store.

I’m reluctant to do this for two reasons. First, I’m lazy. Second, my scripts would become dependent on nix. I’d loose the possibility to easily send my script to someone who doesn’t use nix: if that person tries to execute it, the path $out/bin/hello in the script will mean nothing.

I think I haven’t grabbed enough on nix yet to find a satisfying solution to this. What would be a good approach there? Is trying to package my personal scripts with nix going too far and should I use my good old $PATH and just use nix to install the dependencies?

2 Likes

What I am doing is, implementing them via pkgs.writeScriptBin helper (please search github for usage examples, I am also lazy :grinning:). Also you don’t need to use full path with this method; if the applications you use inside the script end up in your PATH, that means your environment already knows where to find them, so I just copy/pasted whatever I have since I don’t have many scripts (with only exception of headers, no #!/bin/bash for example, #!/usr/bin/env bash instead).

Maybe you can use that with an import mechanism so it’ll be even cleaner.

You’re right, as long as I take care of installing the other applications I use inside my scripts with nix, they are available. But then does packaging my scripts this way have an advantage over just keeping them in my home?

For people still finding this thread via search, some updates from the future on the project described below:


I started work early this year on a project, resholved, to handle this use-case (i.e., nail down dependencies in shell scripts during a build-phase without having to rewrite the scripts and render them useless for non-Nix use).

Since you’re just getting started, I hesitate to recommend it for a few reasons:

  • It isn’t in nixpkgs yet; you’ll need to build it directly in your Nix files or add it to a system/user overlay.
  • I haven’t quite gotten to a release yet, in part because I feel like the names used in the arg/ENV/nix APIs are a hot mess and would like to do a QC pass on them first.
  • It’s still early; documentation is largely self-serve by examples in the test/demo suites.

That said, it does work for a living (and has been for a few months now). I use it to package a few modules into my bash profile, for example. There’s a toy example of how to compose packages with the Nix API in the CI build:

Something you can do is wrap them using makeWrapper. That page is slightly wrong in that it’s not actually available in the standard shell environment but is instead exposed via a setup hook on the makeWrapper package. Anyway, you can use it to ensure PATH contains all of the dependencies the script needs, so if the script runs commands by looking them up in the path they’ll just work and won’t be reliant on your user environment.

The annoying part is you can’t automatically determine what the dependencies are or that you haven’t missed any (well, without using something like the aforementioned resholved that parses the whole script). But if you’re wiling to figure out what dependencies to declare, you can do something like

runCommandLocal "my-script.sh" {
  script = ./my-script.sh;
  nativeBuildInputs = [ makeWrapper ];
} ''
  makeWrapper $script $out/bin/my-script.sh \
    --prefix PATH : ${lib.makeBinPath [ bash jq otherDeps ]}
'';

This will produce a package with $out/bin/my-script.sh that’s a wrapper that sets up PATH before calling your real script. Your real script should be using /usr/bin/env in its shebang (e.g. #!/usr/bin/env bash).

This approach means you can use your scripts unmodified (as long as they’re using /usr/bin/env in their shebang) and therefore the scripts can be run as-is without Nix too.


Alternatively if your goal here is “provide dependencies via Nix” rather than specifically caring about getting the scripts themselves in the nix store, you could just use nix-shell as your script shebang in order to declare the dependencies inline, and then share your scripts however you want. This does tie the scripts to Nix though so they can’t be used elsewhere.


A third option is to modify your scripts to use @foo@ tokens for all dependencies and use substituteAll to replace them with the actual dependencies at build time.

12 Likes

Thank you, I’m glad to learn about the runCommandLocal and the wakeWrapper script. It seems it can be widely useful.

By the way, is there any difference here in using runCommandLocal instead of runCommand? I don’t really understand what is the “nework roundtrip” the manual refers to. Isn’t the derivation build locally even with runCommand?

1 Like

runCommandLocal name env buildCommand is identical to

runCommand name ({
    preferLocalBuild = true;
    allowSubstitutes = false;
} // env) buildCommand

preferLocalBuild = true instructs Nix not to forward the build command on to remote builders but to build it locally. This is a good idea if it’s faster to just build it than it would be to ask the remote builder to do so. Trivial commands (which makeWrapper is, it’s just some shell argument parsing and writing out of a single file) should be done locally.

allowSubstitutes = false says don’t bother checking binary caches for this, just build it anyway. This is a good idea if it’s faster to build the derivation than to ask a binary cache if it has the derivation cached.

So basically, if you’re doing something trivial, use runCommandLocal instead of runCommand. The manual page you linked has a reasonable heuristic, which is if you think the command will take <1s to run, it’s probably better if done locally.

3 Likes

Also, wrapProgram is convenient when the script is needed to be run as part of systemd service (since even #! /usr/bin/env bash not allowed). So, it’s something like:

let
  my-script-deps = with pkgs; [ curl ffmpeg inotify-tools ];
  my-script = pkgs.runCommandLocal "my-script.sh"
    { nativeBuildInputs = [ pkgs.makeWrapper ]; }
    ''
      install -m755 ${./my-script.sh} -D $out/bin/my-script.sh
      patchShebangs $out/bin/my-script.sh
      wrapProgram "$out/bin/my-script.sh" `
      `--prefix PATH : ${pkgs.lib.makeBinPath my-script-deps}
    '';
in
{
  systemd.services.my-script = {
    description = "my-script";
    wantedBy = [ "multi-user.target" ];
    after = [ "network.target" ];
    serviceConfig = {
      Type = "notify"; # my-script notifies `--ready` with `systemd-notify`
      StandardOutput = "append:/var/log/my-script.log";
      ExecStart = "${my-script}/bin/my-script.sh";
    };
  };
}

It seems without wrapProgram two derivations would have to be written manually.

1 Like

I haven’t tried this, but on unstable (or after the next nixos release) you may be able to use something like:

  ExecStart = resholveScript "my-script.sh" {
    inputs = [...];
    interpreter = "${bash}/bin/bash";
  } ./my-script.sh;
3 Likes