How to convert a script that dynamically calls other (dropped in scripts) to Nix

Help
1

I’m just starting with Nix so probably a bit lost.
I’m trying to convert the following into a nix config:

Have a cron driven loop script that reads a config file and then does some wrapping stuff.
After that it looks for other scripts in a certain directory (defined in the config) and executes them.

The whole “thing” currently is a poor mans “plugin system” managed with ansible. Any unrelated role may add it’s own sub-script into a location that then gets executed daily via the main script.

I have converted the script and config part with pkgs.writeTextFile and pkgs.writeShellApplication.

Now I need a starting point / example how to deploy additional scripts for usage in that main script.

2

You’ll need to give more details, scripts calling other scripts is trivial, so I’m guessing there’s more to this. But if your question is really just “how do I run a directory’s worth of scripts in bash”:

for file in "$dir"/*; do
    bash "$file"
done

… assuming you also need it to be sequential.

But there’s probably many layers of XY problem here.

3

Question is not about bash syntax, it’s about how to put that plugin idea into nix derivations.
I currently have my script, its config in the nix store, also all dependent binaries get resolved, very well.

Now lets say I put other shell script in the nix store in different places / derivation, also easy.

Now, how do I make the first script dynamically “detect/resolve” all those added scripts and then call them?

Nix style, not how I do that via vanilla shell :wink:

Edit: In a normal System I would just define that they all have to go to /some/dir and I’m done.

4

With vanilla shell. If you have all the scripts available at build time, it’d just be wasteful to create multiple derivations, so just make a derivation out of all of them and wrap your main script so it has all the runtime deps in scope.

Something like:

let 
  path = lib.makeBinPath [
    <runtime-deps>
  ];
in 
stdenv.mkDerivation {
  name = "scriptstuff";
  src = ./scripts;

  nativeBuildInputs = [ makeWrapper ];

  buildPhase = ''
    mkdir -p $out/{share,bin}
    sed s/<scriptdir>/$out/ config > $out/share/config
    install -m 0755 * $out/bin/
    wrapProgram $out/bin/mainscript --prefix PATH : ${path}
  '';
}

This is asuming that there isn’t more to your scripts. You’ve still not elaborated much on what they do or why you need this. There likely are better approaches to what you actually are trying to do.

5

While I try to digest that snippet: Its basically a backup system:

Main script sets up env, mounts backup host/fs, ensures dirs, does housekeeping on the backups.

But it does not know anything about the backups itself of how they are done.

I can then for various other system components write scripts that take a path argument (where to copy/rsync) the current data to and then they do their stuff.

I don’t actually want to change that approach, just convert it into something that lives in the nix store.

6

Hmm, I think I’m beginning to see where you’re stuck.

This boils down to “how do I give my main script a path to my stuff?”. Well, your configuration, but you’re statically shoving your config into /nix/store.

There are many ways to do that, it depends on the design of your main script. One approach would be something like this:

let
  config = pkgs.writeText "config" ''
    scriptdir = ${./scripts}
  '';
in {
  systemd.services.backups = {
    script = "${./scripts/mainscript} --config ${config}";
  };
}

You could even skip writing the main script outside and just have it in the nix file, makes substituting paths easier. Hell, you could do away with the config indirection then.

You’re already using writeText anyway, so maybe all you needed was to learn about path substitution. Nix will automatically put the whole directory in the nix store.

The derivation I sketched also does something like this, substituting the $out directory (i.e., the package location in /nix/store/) with sed.

Without seeing your script or configuration it’s hard to give you direct instructions, though.

FWIW, the NixOS approach is to use e.g. services.restic.backups, which is also split into multiple backups, and has options for defining per-backup prep scripts.

Long-term I’d suggest either flat-out adopting that or writing a similar module of your own, where the configuration is driven by NixOS options instead of less integrated bash scripts.

7

Even if I do not fully understand it, I think that’s for building a derivation from a local source tree.

But that’s not what I have in mind.

Assume I have 10 nix files that Include/define various services. Some of them also define a “subscript” into the store.
And then I have the 11th include, call it backupLoop.nix that defines a cron job and the loop script.

How does the 11th include get references the the files the other 10 includes have defined / put into the store without having to hardcode that, thats actually what my question boils down to.

8

Ok, that’s clearer. Are these NixOS modules or something custom?

9

I’d say custom as far as I understand It now.

I know that there is file substitution. But is there something like fileset definition that can danymically “grow”?

When you do that you already know you need to plumb config into mainscript, that’s easy, I already have that.

But what I need (basically):

  1. Have pkgs.writeTextFile all over the place and not hold a reference to it
  2. Somehow define that this file belongs to a fileset (this is where I’m lost)
  3. In a completely differnt place (the mainscript.nix) grab a refernce to that fileset (as a virtual directory or whatever) and then use that (from there on i’m pretty sure I can manage)

Not a big fan of Restic for various reasons.

10

Ok, so what you have as far as I can tell is a handful of nix derivations, i.e., just packages. These are inert by themselves.

How do you currently turn this into actually running software?

FWIW, NixOS is the answer to “how do I build a distro out of a bunch of nix packages”, which sounds like where you’re going.

11

WDYM by currently?

  1. have any Linux Distro
  2. have an ansible git
  3. have a role in ansible for doing backups (including mainscript and cron job)
  4. have tens of other roles that just drop a shell script into /more or whatever
  5. win cronjob calls mainscript, mainscript looks in /more and calls all script

With my NixOS conversion I currently have 3 and 4 but not 5
That’s because 4 puts files into predicable locations and thus the content of 3 is straigthforward
I don’t have to know what 4 does when defining 3 cause It gets resolved at runtime.

So for this to work I either need some compile time (nix build) resoltution but where 3 cannot rely on 4 cause then it would have to be hardcoded.
Or something else so that 3 can find those files at runtime cause they are symlinked to a well known location or something.

12

Yeah, you’re a lil’ confused :slight_smile: You’re asking how to do deployment work with a build tool.

To get a handle on mainscript, do you currently just find the nix store path by hand and put the literal path in the cronjob?

Whatever tooling you’re using to deploy has to still do that work. If it involves nix-built binaries, you will need to write the intrgration yourself. This might mean writing an ansible role that invokes nix to get a handle on the scripts you’re building with it, and then symlinking them into /more. Also make sure to create garbage collection roots to paths you’re using.

Nix itself is just a simple build tool, you can produce files from other files, but that’s about it. Nix won’t ever do anything outside of the nix store (except for writing result symlinks when you use nix-build), so it’s fundamentally impossible to create a superset of things defined in a scattered set of directories.

NixOS could be used instead of ansible, or you could write your own deployment tooling in nix from scratch, or you can invent a way to deploy nix packages with ansible (that might already exist out there, to be fair), but what you’re asking for here isn’t possible. Nix isn’t the right tool.

13

No I’m not confused :wink:

Ok Just forget everything for a moment.

Nix is a build tool, right I has built me something that has put a symlink in /etc/static/sudoers that points to something in the store.

How do I make It do the same thing for my pkgs.writeTextFile

That’s all I need to know and then you can tell me how confused I am :wink:

14

Wait, what commands did you invoke to get nix to edit the host’s sudoers file?

15

I sense a trap here…
Nix did not “edit that file” it just built me a new one when I did
security.sudo.extraRules=…
And then symlinked that file to a static path on the host system
How that came to be is currently voodoo to me
And I’m asking how I can get Nix to build me a symlink to a file in the store (one I still have a reference to)
That would for now solve my problem. Might not be elegant but a start.

16

Wait, so you are using NixOS? That’s a NixOS option, it has no meaning to nix itself.

Then how is ansible involved?

17

Ansible is where I came from, Old System, now I want to build a new one with NixOS
You asked what I currently have that’s why I asked wdym “currently”
Need a break, but If you could hint me at the definition I would have to put in a nix file that symlinks the result of pkgs.writeTextFile to a static location that would be great

That would then be the runtime resolution strategy.
I also would like to know if there is an alternative build time startegy.

18

environment.etc."foo/bar".source = expr; will symlink the result of the expr derivation into /etc/foo/bar.

1 Like
19

Yeah, you royally got me when you said you’re building something custom that is not NixOS and are trying to run backups with an ansible role :smiley:

This is trivial with NixOS, see @waffle8946 answer. You could dress it up a little with a custom option and symlinkJoin, but putting stuff in /etc is probably fine while you figure out how this all works.

2 Likes
20

Can you link my answer? I haven’t seen anything while reading the thread…