Borgbackup can't mount/unlock LUKS device in preHook

Hi!

I’m trying to run a scheduled borgbackup job on a LUKS encrypted removable HDD. The HDD is a dedicated backup drive, so I want to mount it in the preHook and unmount it in the postHook.

However, when the service runs, it fails with the error message:

borgbackup-job-testBackup.service: Failed to set up mount namespacing: /data/backup/testBackup: No such file or directory
borgbackup-job-testBackup.service: Failed at step NAMESPACE spawning /nix/store/ra2sr1y6fy6gjr63bglql0r594g8qxca-unit-script-borgbackup-job-tes>

Everything runs fine when I manually mount the HDD, but I was hoping that the prehook would do that for me (as it is actually stated in the documentation: that’s what it’s for).

Here’s the testConfig I’m using (the repository is pre-initialized, since the creation is buggy with nix):

services.borgbackup = {
  jobs.testBackup = {
    privateTmp = false;
    preHook = with pkgs; /*bash*/ ''
      echo "Starting the prehook"

      ${cryptsetup}/bin/cryptsetup open /dev/sde1 backupDrive --key-file=/etc/secrets/backupKeyfile.key
      ${mount}/bin/mount --mkdir /dev/mapper/backupDrive /data/backup
    '';
    postHook = with pkgs; /*bash*/ ''
      ${umount}/bin/umount /data/backup
      ${cryptsetup}/bin/cryptsetup close /dev/mapper/backupDrive
    '';
   startAt = "*-*-* *:*:*5";
    prune.keep = {
      within = "1d";
    };

    removableDevice = true;
    doInit = true;
    compression = "auto,lzma";
    encryption.mode = "none";
    inhibitsSleep = true;
    repo = "/data/backup/testBackup";
    paths = [
      "/tmp/testbackup"
    ];
    readWritePaths = [
      "/tmp/testbackup"
    ];
  };
};

Yeah, the whole service is run with systemd “hardening”, it’s in a cgroup with as few permissions as possible.

preHook is also part of the backup script instead of using the systemd feature, so there’s no way to give this better permissions with just in-module options.

The idiomatic way of doing this is probably to use a systemd mount file for your drive and to then make the systemd service that runs this backup depend on it. You clearly also want to stop having the encrypted drive available after the unit exits, though, and systemd doesn’t provide any relationships that allow you to describe that, so manual scripting is indeed necessary.

In this case it’s probably best to modify the underlying systemd unit, and use preStart and preStop with + to gain root privileges for just those processes, e.g.:

{ pkgs, ... }: {
  # insert borg config here

  systemd.services."borgbackup-job-testBakup" = {
    # Wish we could just:
    # upholds = [ "data-backup.mount" ];

    serviceConfig = {
      ExecStartPre = [
         "+${pkgs.cryptsetup}/bin/cryptsetup open /dev/sde1 backupDrive --key-file=/etc/secrets/backupKeyfile.key"
         "+${pkgs.mount}/bin/mount /data/backup"
      ];
      
      ExecStopPost = [
        "+${pkgs.unmount}/bin/umount /data/backup"
        "+${pkgs.cryptsetup}/bin/cryptsetup close /dev/mapper/backupDrive"
      ];
    };
  };
}
1 Like

Ok, that makes sense. Thanks for the explanation. :slight_smile:

Can the preHook actually mount drives, then or is the documentation lying when it claims that it “can […] be used to mount file systems”?

I’m also wondering where you got the information about the systemd “hardening”. Just curious where I can find that info in the future for better understanding on my part.

Thank you, too for your example, I’ll adapt that. Do you reckon that the Wiki find this information useful, too? And if yes, would you be ok if your example would be adapted there?

Guess it’s lying. It’s probably worth raising an issue about this to the NixOS repo (or checking if one already exists), obviously the prehook could be implemented to actually expose the privilege-granting feature instead to enable this.

Presumably someone added the hardening stuff and neglected this bit of documentation, and didn’t think of this use case. You could dig through the blame for more context, I suppose.

From the source code: nixpkgs/nixos/modules/services/backup/borgbackup.nix at 23cbb250f3bf4f516a2d0bf03c51a30900848075 · NixOS/nixpkgs · GitHub

Also the error message is a dead giveaway if you have a lot of experience with NixOS, systemd and cgroups.

You can learn more about these systemd features here: systemd.exec

NixOS modules use those features extensively to help mitigate against malicious or broken services. This is part of why I tend to be quite overbearing whenever I see someone messing with cross-user stuff, as a side note (e.g., trying to add sudo configured with NOPASSWD to a script like yours to circumvent the namespace issue, rather than using the proper built-in features to do things, because everything’s a permission problem if all you have is sudo), but you’re doing everything properly here.

Sure, if you’ve tested it and it actually works. I’m not certain this is the issue, there’s a chance this is internal to borg.

Giving a nicer API upstream would be best, of course, but this is an alright solution in the meantime.

Yeah, but make sure you add it to https://wiki.nixos.org, rather than nixos.wiki - the latter is unofficial and hosted by a third party, but still has better search ranking because it has existed for a very long time.

I tried it in the meantime and unfortunately, your solution wtih ExecStartPre and ExecStopPost throws the same error. Additionally, it says that

The process /nix/store/<...>-cryptsetup-2.7.3-bin/bin/cryptsetup could not be executed and failed
borgbackup-job-testBackup.service: Control process exited, code=exited, status=226/NAMESPACE

So, I guess I’ll have to declare another service? For the unmounting, I could either declare another timer which triggers when I’m assuming the backup is done. Or I create a file on mount that acts as a flag that the backup is in progress, which in turn gets deleted by postHook of the backup job. Both are a bit hack-y solutions, I guess. :confused:

Hmm, this might be more fundamental configuration on your end unrelated to this service. Anything more in journalctl --boot -xe?

Not really seeing any dead giveaways, tbh:

Sep 24 22:09:58 muffinman systemd[1]: Starting BorgBackup job testBackup...
░░ Subject: A start job for unit borgbackup-job-testBackup.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit borgbackup-job-testBackup.service has begun execution.
░░ 
░░ The job identifier is 155757.
Sep 24 22:09:58 muffinman (yptsetup)[2997543]: borgbackup-job-testBackup.service: Failed to set up mount namespacing: /data/backup/testBackup: No such file or directory
Sep 24 22:09:58 muffinman (yptsetup)[2997543]: borgbackup-job-testBackup.service: Failed at step NAMESPACE spawning /nix/store/ldfgf3gg85083fhl3m17lmz31d7l5rmn-cryptsetup-2.7.3-bin/bin/cryptsetup: No such file >
░░ Subject: Process /nix/store/ldfgf3gg85083fhl3m17lmz31d7l5rmn-cryptsetup-2.7.3-bin/bin/cryptsetup could not be executed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The process /nix/store/ldfgf3gg85083fhl3m17lmz31d7l5rmn-cryptsetup-2.7.3-bin/bin/cryptsetup could not be executed and failed.
░░ 
░░ The error number returned by this process is ERRNO.
Sep 24 22:09:58 muffinman systemd[1]: borgbackup-job-testBackup.service: Control process exited, code=exited, status=226/NAMESPACE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStartPre= process belonging to unit borgbackup-job-testBackup.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 226.
Sep 24 22:09:58 muffinman (umount)[2997544]: borgbackup-job-testBackup.service: Failed to set up mount namespacing: /data/backup/testBackup: No such file or directory
Sep 24 22:09:58 muffinman (umount)[2997544]: borgbackup-job-testBackup.service: Failed at step NAMESPACE spawning /nix/store/icvzgs36w80l0nhh7ixg7ah8x569bb20-umount-util-linux-2.39.4/bin/umount: No such file or>
░░ Subject: Process /nix/store/icvzgs36w80l0nhh7ixg7ah8x569bb20-umount-util-linux-2.39.4/bin/umount could not be executed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The process /nix/store/icvzgs36w80l0nhh7ixg7ah8x569bb20-umount-util-linux-2.39.4/bin/umount could not be executed and failed.
░░ 
░░ The error number returned by this process is ERRNO.
Sep 24 22:09:58 muffinman systemd[1]: borgbackup-job-testBackup.service: Control process exited, code=exited, status=226/NAMESPACE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStopPost= process belonging to unit borgbackup-job-testBackup.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 226.
Sep 24 22:09:58 muffinman systemd[1]: borgbackup-job-testBackup.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit borgbackup-job-testBackup.service has entered the 'failed' state with result 'exit-code'.
Sep 24 22:09:58 muffinman systemd[1]: Failed to start BorgBackup job testBackup.
░░ Subject: A start job for unit borgbackup-job-testBackup.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit borgbackup-job-testBackup.service has finished with a failure.
░░ 
░░ The job identifier is 155757 and the job result is failed.
Sep 24 22:09:58 muffinman sudo[2997537]: pam_unix(sudo:session): session closed for user root

Could you share your config? Something seems inherently broken on your system.

Cursory searches say this kind of stuff can happen if you symlink /var/tmp to /tmp (which makes sense, because systemd namespaces go in /var/tmp, so the service would fail to start cryptsetup, and presumably borg also sets up its namespace in there when it uses FUSE to mount its repository), but I doubt you did that.

Edit: Can you run journalctl -f while you start the unit and see if anything is printed?

Sharing my config would take a while, since I haven’t split my private info from the public info yet (IP address, public ssh port to VPS, etc.) and therefore, I’m not (yet) comfortable to go public with my config.

But here’s the output of journalctl -f:

Sep 24 22:38:36 muffinman sudo[3133067]: prunebutt : TTY=pts/3 ; PWD=/home/prunebutt/nixos ; USER=root ; COMMAND=/run/current-system/sw/bin/systemctl start borgbackup-job-testBackup.service
Sep 24 22:38:36 muffinman sudo[3133067]: pam_unix(sudo:session): session opened for user root(uid=0) by prunebutt(uid=1000)
Sep 24 22:38:36 muffinman systemd[1]: Starting BorgBackup job testBackup...
Sep 24 22:38:36 muffinman (yptsetup)[3133072]: borgbackup-job-testBackup.service: Failed to set up mount namespacing: /data/backup/testBackup: No such file or directory
Sep 24 22:38:36 muffinman (yptsetup)[3133072]: borgbackup-job-testBackup.service: Failed at step NAMESPACE spawning /nix/store/ldfgf3gg85083fhl3m17lmz31d7l5rmn-cryptsetup-2.7.3-bin/bin/cryptsetup: No such file or directory
Sep 24 22:38:36 muffinman systemd[1]: borgbackup-job-testBackup.service: Control process exited, code=exited, status=226/NAMESPACE
Sep 24 22:38:36 muffinman (umount)[3133073]: borgbackup-job-testBackup.service: Failed to set up mount namespacing: /data/backup/testBackup: No such file or directory
Sep 24 22:38:36 muffinman (umount)[3133073]: borgbackup-job-testBackup.service: Failed at step NAMESPACE spawning /nix/store/icvzgs36w80l0nhh7ixg7ah8x569bb20-umount-util-linux-2.39.4/bin/umount: No such file or directory
Sep 24 22:38:36 muffinman systemd[1]: borgbackup-job-testBackup.service: Control process exited, code=exited, status=226/NAMESPACE
Sep 24 22:38:36 muffinman systemd[1]: borgbackup-job-testBackup.service: Failed with result 'exit-code'.
Sep 24 22:38:36 muffinman systemd[1]: Failed to start BorgBackup job testBackup.
Sep 24 22:38:36 muffinman sudo[3133067]: pam_unix(sudo:session): session closed for user root

Edit: This is the full config of the testBackup:

{pkgs, ...}:let
  backupName = "testBackup";
in {
  services.borgbackup = {
    jobs."${backupName}" = {
      startAt = "*-*-* *:*:*5";
      prune.keep = {
        within = "1d";
      };

      removableDevice = true;
      doInit = true;
      compression = "auto,lzma";
      encryption.mode = "none";
      inhibitsSleep = true;
      repo = "/data/backup/testBackup";
      paths = [
        "/home/prunebutt/testbackup"
      ];
    };
  };

  systemd.services."borgbackup-job-${backupName}".serviceConfig = {
    ExecStartPre = with pkgs; [
      "+${cryptsetup}/bin/cryptsetup open /dev/sde1 backupDrive --key-file=/etc/secrets/backupKeyfile.key"
      "+${mount}/bin/mount --mkdir /dev/mapper/backupDrive /data/backup"
    ];
    ExecStopPost = with pkgs; [
      "+${umount}/bin/umount /data/backup"
      "+${cryptsetup}/bin/cryptsetup close /dev/mapper/backupDrive"
    ];
  };
}

I just faced the same issue while setting up a backup job to a USB HDD. I found a working solution that’s fairly simple but maybe not that pretty. I’ll show the reason first, so that the workaround can make more sense.

This was my starting point (only the relevant part, anonymized):

repo = "/mnt/backup/computername";
preHook = ''
${pkgs.util-linux}/bin/mount "/dev/disk/by-uuid/00000000-0000-0000-0000-000000000000" /mnt
'';

The drive is mounted to /mnt but the Borg repo is actually two directories deep into the disk partition. When the NixOS module generates the service for the backup job, it adds the path defined by repo to ReadWritePaths property of the systemd service as a part of the hardening/sandboxing. The problem is that the full path does not exist yet when the service is started, causing it to fail with the “Failed to set up mount namespacing” error.

This means the solution is to make sure the repo is available at a location that exists already when the service starts. This is what I ended up doing:

repo = "/mnt";
preHook = ''
${pkgs.util-linux}/bin/mount "/dev/disk/by-uuid/00000000-0000-0000-0000-000000000000" /mnt
${pkgs.util-linux}/bin/mount --bind /mnt/backup/computername /mnt
'';

The path /mnt already exists, so the ReadWritePaths will be valid at the service startup. Just mounting the drive causes the path /mnt/backup/computername to exist (I have created the directory beforehand on the HDD). That directory is then bind mounted to /mnt, bringing the subdirectory up to the mount point, which is where Borg will then make the backup. It’s a little weird, but as far as I know, entirely valid to mount something somewhere and then shadow it with another mount on top of it.

Notice that you’ll need to use umount twice to unmount first the bind mount and then the actual partition mount in postHook.

There might be cleaner way to do this, maybe by somehow replacing the repo in the ReadWritePaths with the mount point, but I’m not experienced enough with Nix to know how to do that or if it’s even possible, so opted for a more common Linux tools way of doing this.


Finally, as a completely optional step, I also added this to my configuration (it obviously depends on the name of the backup job):

systemd.services.borgbackup-job-hdd.serviceConfig.PrivateMounts = true;

It causes the mounts done by the preHook to be visible only to the backup job itself, so while the service is running, the globally visible /mnt directory isn’t disturbed. Also everything mounted with PrivateMounts should be automatically unmounted when the service stops running. I don’t know how the use of cryptsetup interacts with the option.

1 Like

Awesome! That did it for me as well. Since I already had the backup at /data/backup/hostName, I needed to add this as well:

services.borgbackup.jobs.backupJob.environment.BORG_RELOCATED_REPO_ACCESS_IS_OK = "yes";
1 Like