Move script from flake into its own file

I am attempting to move a bash script to its own file using the setup hook makeWrapper along with the readFile, simlinkJoin and writeShellScriptBin functions. but for some reason it doesn’t seem to find any files. What I’ve tried so far is to read the script, writing it to the store and making it executable. Here is the full flake and here are the relevant bits:

nanolibs-script = rec {
  name = "nanolibs-path";
  source = builtins.readFile ./nanolibs-script.sh;
  script = (pkgs.writeShellScriptBin name source).overrideAttrs(old: {
    buildCommand = "${old.buildCommand}\n patchShebangs $out";
  });
  buildInputs = [
    riscv-toolchain.newlib-nano
  ];
};

I then create a new derivation with symlinkJoin by adding symlinks to each of the paths, that is, the script wrapper and the libraries, then making it executable withmakeWrapper:

packages = {
  nanolibsPath = pkgs.symlinkJoin {
    name = "nanolibs-path";
    paths = [ nanolibs-script.script ] ++ nanolibs-script.buildInputs;
    buildInputs = with pkgs; [
      makeWrapper
    ];
    postBuild = ''
      wrapProgram $out/bin/${nanolibs-script.name}  --prefix PATH : $out/bin
    '';
};

Finally, I create a devShell with a shellHook that runs the executable:

devShells = {
  fe310Shell = pkgs.mkShell {
    buildInputs = with pkgs; [
      riscv-toolchain.buildPackages.gcc
      openocd
   ];
   shellHook = ''
      nix run .#nanolibsPath
   '';
};

The script itself is located in the same directory as the flake, it’s contents being:

#!/usr/bin/env bash

rm -fr nanolibs/*.a
mkdir -p nanolibs
for file in "${riscv-toolchain.newlib-nano}"/riscv32-none-elf/lib/*.a; do
   ln -s "$file" nanolibs
done
for file in nanolibs/*.a; do
   mv "$file" "${file%%.a}_nano.a"
done

As explained at the beginning, instead of all archive files in ${riscv-toolchain.newlib-nano}"/riscv32-none-elf/lib/, the above creates the nanolibs directory and renames a ghost file *_nano.a. It does not seem to find the files. Not an issue with the bash script since having this script inline, in the flake itself, does what is expected of it, such as in the following snippet:

nanolibs-script = rec {
  name = "nanolibs-path";
  source = ''
    rm -fr nanolibs/*.a
    mkdir -p nanolibs
    for file in "${riscv-toolchain.newlib-nano}"/riscv32-none-elf/lib/*.a; do
      ln -s "$file" nanolibs
    done
    for file in nanolibs/*.a; do
      mv "$file" "''${file%%.a}_nano.a"                
    done          
  '';
  buildInputs = [
    riscv-toolchain.newlib-nano
  ];
};

Then on my devShell, I replace the nix run command with shellHook = nanolibs-script.script; and it does what is expected, it renames all archive files in the ${riscv-toolchain.newlib-nano}"/riscv32-none-elf/lib directory and symlinks them into the nanolibs directory.

This is a string antiquote, and won’t work from an external file through builtins.readFile

2 Likes

You will also need to add the dependencies (buildInputs) to this wrapProgram call, so that they are in your $PATH and you can call them without the string interpolation.

Alternatively, just use writeShellApplication, since it does all of this for you.

1 Like

I was kind of intuitively suspicious about this bit. Also, it doesn’t make much sense to have this script in its own file if it’ll be useless in other systems. The question now is whether there’s a way for this directory to be found by the script. If not, I may as well forget about having it outside the flake.

it’s not flakes but easy to adapt, it adds bin/myscript file in the $PATH of the shell

{ pkgs ? import ./nix/nixpkgs { } }:

let
  myscript = pkgs.writeShellScriptBin "myscript" (builtins.readFile ./bin/myscript);
in
with pkgs;

mkShell {
  packages = [ myscript ];
}
1 Like

Yes, but that won’t work in this case because of the string interpolation. Hence I suggest pkgs.writeShellApplication :slight_smile:

You could also simply rely on the dependencies already being in $PATH with that method, but that’s not exactly reproducible.

I’m not sure to understand the difference :sweat_smile:

If I’m understanding, your two main options are either to substitute (see shell functions at/after Nixpkgs Reference Manual) the path of those files into the script, or use an environment variable and declare the variable in the wrapper.

1 Like

I simply need to copy or symlink all files in a specific store and rename them. The thing is, I’m not sure that the substitute shell function can operate on the file names and if so, whether or not it can do batch jobs (I could do a for loop ). substituteAll doesn’t seem to apply here too as seems to operate on the contents of single files.


UPDATE

I obviously didn’t understand @abathur suggestion initially. Exporting the variable and replacing the path with $NANOLIBS_PATH in the shell script did the trick:

shellHook = ''
  export NANOLIBS_PATH="${riscv-toolchain.newlib-nano}"/riscv32-none-elf/lib/*.a
  nix run .#nanolibsPath
'';

Alternatively, with substituteInPlace (replacing path in the shell script with @NANOLIBS_PATH@):

shellHook = ''
  substituteInPlace ./nanolibs-script.sh --subst-var-by NANOLIBS_PATH "${riscv-toolchain.newlib-nano}/riscv32-none-elf/lib/*.a"
  nix run .#nanolibsPath
'';

Or substitute if the original sript is to be preserved:

shellHook = ''
  substitute ./nanolibs-script.sh ./nanolibs-script-with-path.sh --subst-var-by NANOLIBS_PATH "${riscv-toolchain.newlib-nano}/riscv32-none-elf/lib/*.a"
  nix run .#nanolibsPath
'';

In which case the source attribute will have to point to the new file, along these lines: source = builtins.readFile ./nanolibs-script-with-path.sh;)

Although a combination of makeWraper with functions readFile , simlinkJoin and writeShellScriptBin work perfectly well with a separate shell script insofar as the path is set through an environmental variable or substitute function (see above), I finally settled with writeShellApplication as suggested by @TLATER as it much simpler, requiring only the folowing package declaration (in addition to an environmental variable or substitute function as well):

nanolibsPath = pkgs.writeShellApplication {
  name = "nanolibs-path";
  runtimeInputs = with pkgs; [
    riscv-toolchain.newlib-nano
  ];
  text = builtins.readFile ./nanolibs-script.sh;
};
2 Likes

This is clean and reads easily. I wonder where this could be documented. The wiki maybe?

1 Like

I wondered the same thing. The wiki seems a good place, thanks for the suggestion @Solene! I’ve another issue that needs documeting, will start working on both on the next few days.

I didn’t mention these before since they don’t really help with this problem/script, but if you have a fair number of scripts and want to keep them in separate files, you may eventually have some use for resholve (repo, nix API doc), which is more focused on discovering and substituting external executables in shell scripts.

I also wrote a little blog post this summer that reviews the main approaches to replacing supplying dependencies for shell scripts: no-look, no-leap Shell script dependencies

3 Likes

@abathur 's post goes over this as well a little implicitly, but the problem with writeShellScriptBin is this line:

It uses the ${varname) syntax to refer to variables set in nix, not the bash shell itself. It’s a bit confusing because it’s valid bash syntax too, but in fact the intention here is for nix to substitute a string into that variable, in this case the nix store directory of that package.

readFile will not cause nix to evaluate the file, so nix won’t parse it, let alone substitute that variable. Hence you can’t use writeShellScriptBin for this script, at least not without makeWrapper and some modifications.

Using writeShellApplication you can basically do what writeShellScriptBin + makeWrapper do together, even with a bit less overhead. But @abathur explains this better than I care to given the link is in this thread already :wink:

2 Likes

@Solene I wonder if this would fit a wiki nix cookbook or did you have something else in mind?

1 Like

Nothing in mind, maybe @fricklerhandwerk will have an idea? :slight_smile:

@nrdsp Yes, in its current form it would best fit the cookbook (or how-to guide) form. Please consult How-to guides - Diátaxis for details how to best set it up (it’s a fairly short read).

If you just want to get it over with, add it to the wiki. If you would like to polish it and get it reviewed, make a PR to nix.dev - there your work will also probably have more exposure. Note that nix.dev is still not clear if it wants to be a collection of tutorials or how-to guides, but we’ll fix that on the way.

2 Likes

@fricklerhandwerk I’m about to push a PR to nix.dev, but I don’t seem to have permission to do so.

1 Like