Variables isn't expanded in SystemD units

[a12l]

I’m trying to use Restic to automatically make backups to my Rsync.net storage over SSH. But when I look at the logs, it seems that the environmental variable REPOSITORY isn’t expanded to its value defined in services.restic.backups."rsync.net".environmentFile.

$ systemctl status restic-backups-rsync-dot-net.service
× restic-backups-rsync-dot-net.service
     Loaded: loaded (/etc/systemd/system/restic-backups-rsync-dot-net.service; linked; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2022-03-20 21:00:23 CET; 53min ago
TriggeredBy: ● restic-backups-rsync-dot-net.timer
    Process: 161796 ExecStartPre=/nix/store/5ckmg1ph08vb12b1q34122qk4y2wznnh-unit-script-restic-backups-rsync-dot-net-pre-start/bin/restic-backups-rsync-dot-net-pre-start (code=exited, status=1/FAILURE)
         IP: 0B in, 0B out
        CPU: 42ms

Mar 20 21:00:23 p-desktop1 systemd[1]: Starting restic-backups-rsync-dot-net.service...
Mar 20 21:00:23 p-desktop1 restic-backups-rsync-dot-net-pre-start[161798]: Fatal: unable to open config file: stat @REPOSITORY@/config: no such file or directory
Mar 20 21:00:23 p-desktop1 restic-backups-rsync-dot-net-pre-start[161798]: Is there a repository at the following location?
Mar 20 21:00:23 p-desktop1 restic-backups-rsync-dot-net-pre-start[161798]: @REPOSITORY@
Mar 20 21:00:23 p-desktop1 restic-backups-rsync-dot-net-pre-start[161808]: Fatal: create repository at @REPOSITORY@ failed: mkdir @REPOSITORY@: permission denied
Mar 20 21:00:23 p-desktop1 systemd[1]: restic-backups-rsync-dot-net.service: Control process exited, code=exited, status=1/FAILURE
Mar 20 21:00:23 p-desktop1 systemd[1]: restic-backups-rsync-dot-net.service: Failed with result 'exit-code'.
Mar 20 21:00:23 p-desktop1 systemd[1]: Failed to start restic-backups-rsync-dot-net.service.

I don’t know if it’s universal that the strings enclosed by @s are treated as environmental variables. It’s how it’s done with networking.wireless.networks.<name>.* when you’ve set the option networking.wireless.environmentFile, but

• there’s no documentation for theservices.restic.backups.<name>.environmentFile option;
• I can’t see any meaningful difference is the source code between the modules, but I don’t know what happens behind the curtains;
• I couldn’t find any general documentation in the Nixpkgs manual when I searched for “environmentFile” and “environment file”; and finally
• I’ve tested to refer to the variables by enclosing them in@s, but it didn’t work. I don’t know if it doesn’t work because it’s the wrong way to refer to variables; or if it’s because I’ve done something else wrong.

Here’s the module where I configure the agenix secrets and the restic module.

  age.secrets = {
    "users/a12l/backups/rsync.net/variables.env" = {
      file = "${self}/secrets/users/a12l/backups/rsync.net/variables.env.age";
      owner = "a12l";
      group = "users";
    };
    "users/a12l/backups/rsync.net/password" = {
      file = "${self}/secrets/users/a12l/backups/rsync.net/password.age";
      owner = "a12l";
      group = "users";
    };
  };
  services.restic.backups = {
    "rsync.net" = {
      environmentFile = config.age.secrets."users/a12l/backups/rsync.net/variables.env".path;
      user = "a12l";
      passwordFile = config.age.secrets."users/a12l/backups/rsync.net/password".path;
      initialize = true;
      repository = "@REPOSITORY@";

      paths = [
        "@PATH_1@"
        "@PATH_2@"
        "@PATH_3@"
      ];

      pruneOpts = [
        "@PRUNE_HOURLY@"
        "@PRUNE_DAILY@"
        "@PRUNE_WEEKLY@"
        "@PRUNE_MONTHLY@"
        "@PRUNE_YEARLY@"
      ];

      timerConfig = {
        OnCalendar = "@WHEN_TO_BACKUP@";
      };
    };
  };

[TLATER]

networking.wireless.networks.<name>.* is the odd case. That module uses some bespoke scripts to replace variables with ones from environment variables it gets from a file.

Most modules in NixOS are backed by systemd services. The scripts running inside of them are typically pure bash scripts, however, and don’t support systemd’s % variables or its environment variables.

Instead, you can use bash environment variables, which are specified like $VARNAME - this is pretty usual for almost every other distribution, framework, you name it, by the way. Just not odd cases like the wireless module

Now, as for your example, not all of those properties can be replaced with arbitrary variable names either. Unless it’s documented that they can be, or the property is called something with “script” in the name, it’s probably best to assume you can’t use environment substitution at all (because that might be removed in the future or otherwise be fragile).

So, assuming we can’t use substitution, we should look at another solution. What are you trying to accomplish here?

I see you are trying to use values from age for your options. I assume you’re trying to set up a remote without allowing anything to see at what time, which paths and where the backups are being sent to?

Here are some things to consider:

• With the exception of the contents of the passwordFile, all of those will be world-readable on your computer, so using agenix is essentially pointless for those values. If you are only concerned about putting this on a public repository, git-crypt is an option (except for the password).
• Even if it did work, other users on the computer could read the systemd service file and get all of that information anyway, unless you put a lot of effort into prohibiting that.
• Even if you do all the above, someone with physical access to your machine could more or less trivially capture network packets and get all of that information anyway.
• I find it hard to imagine a threat model in which you need to hide your backup times and paths as well. You should probably not be using Linux if you have such a requirement, or really any off-the-shelf OS.

Are you really sure you need to hide all of those settings behind agenix? Is there any way you could just put them in there literally?

[a12l]

@TLATER

networking.wireless.networks.<name>.* is the odd case. That module uses some bespoke scripts to replace variables with ones from environment variables it gets from a file.

Most modules in NixOS are backed by systemd services. The scripts running inside of them are typically pure bash scripts, however, and don’t support systemd’s % variables or its environment variables.

Thanks for the explanation!

Instead, you can use bash environment variables, which are specified like $VARNAME - this is pretty usual for almost every other distribution, framework, you name it, by the way. Just not odd cases like the wireless module

Now, as for your example, not all of those properties can be replaced with arbitrary variable names either. Unless it’s documented that they can be, or the property is called something with “script” in the name, it’s probably best to assume you can’t use environment substitution at all (because that might be removed in the future or otherwise be fragile).

The important attribute to hide is the value of services.restic.backups."rsync.net".repository. I don’t want scrapers to find my username and specific server I’m using for my backups. A little security by obscurity.

How can I use bash environmental names for the repository attribute? I naively tested to replace "@REPOSITORY@" with `“$REPOSITORY”, but that doesn’t work.

$ systemctl status restic-backups-rsync.net.service
× restic-backups-rsync.net.service
     Loaded: loaded (/etc/systemd/system/restic-backups-rsync.net.service; linked; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2022-03-21 12:00:00 CET; 57min ago
TriggeredBy: ● restic-backups-rsync.net.timer
    Process: 18162 ExecStartPre=/nix/store/2ansxby9mpsb46hyb47dy5xqa3skfzk8-unit-script-restic-backups-rsync.net-pre-start/bin/restic-backups-rsync.net-pre-start (code=exited, status=1/FAILURE)
         IP: 0B in, 0B out
        CPU: 41ms

Mar 21 12:00:00 p-desktop1 systemd[1]: Starting restic-backups-rsync.net.service...
Mar 21 12:00:00 p-desktop1 restic-backups-rsync.net-pre-start[18165]: Fatal: unable to open config file: stat $REPOSITORY/config: no such file or directory
Mar 21 12:00:00 p-desktop1 restic-backups-rsync.net-pre-start[18165]: Is there a repository at the following location?
Mar 21 12:00:00 p-desktop1 restic-backups-rsync.net-pre-start[18165]: $REPOSITORY
Mar 21 12:00:00 p-desktop1 restic-backups-rsync.net-pre-start[18179]: Fatal: create repository at $REPOSITORY failed: mkdir $REPOSITORY: permission denied
Mar 21 12:00:00 p-desktop1 systemd[1]: restic-backups-rsync.net.service: Control process exited, code=exited, status=1/FAILURE
Mar 21 12:00:00 p-desktop1 systemd[1]: restic-backups-rsync.net.service: Failed with result 'exit-code'.
Mar 21 12:00:00 p-desktop1 systemd[1]: Failed to start restic-backups-rsync.net.service.

So, assuming we can’t use substitution, we should look at another solution. What are you trying to accomplish here?

I see you are trying to use values from age for your options. I assume you’re trying to set up a remote without allowing anything to see at what time, which paths and where the backups are being sent to?

It’s primarily a little security by obscurity, to hamper potential ransomware gangs to gauge where to attack my backup system. I don’t think I’ll be targeted directly, but I can’t rule out some potential future automated attack where the gangs’ malware can disable backup systems if they’ve enough information about it.

• With the exception of the contents of the passwordFile , all of those will be world-readable on your computer, so using agenix is essentially pointless for those values. If you are only concerned about putting this on a public repository, git-crypt is an option (except for the password).

I’m using Pijul for version control of my configuration. It’s still in beta, so I don’t think anyone has developed a similar tool for it. Couldn’t find anything in their chat either about encryption.

• Even if it did work, other users on the computer could read the systemd service file and get all of that information anyway, unless you put a lot of effort into prohibiting that.
• Even if you do all the above, someone with physical access to your machine could more or less trivially capture network packets and get all of that information anyway.

These attack vectors doesn’t concern me.

• I find it hard to imagine a threat model in which you need to hide your backup times and paths as well. You should probably not be using Linux if you have such a requirement, or really any off-the-shelf OS.

As I stated above, it’s primarily ransomware I’m concerned with. And it’s not so uncommon to want to hide when backups are made, at least in bigger computer systems. It’s one big reason to setup a pull-based backup system, and not a push-based one as I’m currently trying to setup. I’m planning to use a pull-based one in the unspecified future, but that’s a bit more work that I currently don’t have time for.

Thanks again for your explanation and help!

[TLATER]

Right, so the only problem at this point is the repository option? It looks like internally that’s sent to restic via an environment variable: https://github.com/NixOS/nixpkgs/blob/2c66a7a6e036971c4847cca424125f55b9eb0b0b/nixos/modules/services/backup/restic.nix#L248

Since it’s set as an environment variable, it won’t be evaluated by bash, but by restic directly, which probably doesn’t attempt to resolve it against other environment variables, so there’s no way to get that option to substitute from an environment file.

You can try to add -r $REPOSITORY to backup.extraBackupArgs. That looks like it should work, since that string is currently not escaped. You can also try setting the RUSTIC_REPOSITORY variable instead; I’m not sure which environment setting will take precedence over the other. In either case, you’ll need to set the repository variable to an empty string, or some other bogus value, because the module is designed to require that option to be set.

The long-term fix is using something like git-crypt, however, or to host your config files on a non-public repository. If you are really concerned about potential scanning attacks just finding URLs to infrastructure you use, that’s probably a more reasonable approach.

[a12l]

@TLATER

You can try to add -r $REPOSITORY to backup.extraBackupArgs . That looks like it should work, since that string is currently not escaped. You can also try setting the RUSTIC_REPOSITORY variable instead; I’m not sure which environment setting will take precedence over the other. In either case, you’ll need to set the repository variable to an empty string, or some other bogus value, because the module is designed to require that option to be set.

I renamed the environmental variable from REPOSITORY to RESTIC_REPOSITORY (you’d a small typo), and set services.restic.backups."rsync.net".repository = "";, and now it works. Thanks!

The long-term fix is using something like git-crypt, however, or to host your config files on a non-public repository. If you are really concerned about potential scanning attacks just finding URLs to infrastructure you use, that’s probably a more reasonable approach.

I just saw that restic has the option --repository-file="", similar to its --password-file="". So it shouldn’t be hard to add the option services.restic.backups.<name>.repositoryFile, as an alternative to services.restic.backups.<name>.repository?

Thanks for your help!

[TLATER]

@a12l

RESTIC_REPOSITORY (you’d a small typo),

Good spot, shows what my mind is on these days

@a12l

I just saw that restic has the option --repository-file="" , similar to its --password-file="" .

Yep, but again, you’ll be relying on it working that way long-term. If the repository option being empty eventually stops working, or something similar, you’ll need a new fix. The most long-term fix is definitely to find a way to hide your option values from the public without relying on bespoke hacks around how the modules work.

You can also try to add an option for that to the NixOS module. Since it’s supported upstream, there might be enough appetite for an alternative way of specifying the URL.