Trouble setting up `borgbackup`

Hello, all!

I am having some trouble getting borgbackup to work. I don’t know what is causing it, to be honest. I’ve tried several different setups to now avail.


Setup and results

First setup:

{ config, lib, pkgs, myVars, ... }:

{
  services.borgbackup = {
    jobs = {
      "homeserver" = {
        paths = myVars.general.testPath;
        environment.BORG_RSH = "ssh -i ${myVars.general.borgRsh}";
        repo = myVars.general.borgRepo;
        compression = "zstd,8";
        startAt = "hourly";

        encryption = {
          mode = "repokey-blake2";
          passCommand = "cat ${myVars.general.borgPassPath}";
        };
      };
    };
  };

  environment.systemPackages = with pkgs; [ borgbackup ];
}

Result:

~$ systemctl status borgbackup-job-homeserver.service

× borgbackup-job-homeserver.service - BorgBackup job homeserver
     Loaded: loaded (/etc/systemd/system/borgbackup-job-homeserver.service; linked; preset: ignored)
     Active: failed (Result: exit-code) since Wed 2024-11-13 14:26:44 CET; 2s ago
   Duration: 910ms
 Invocation: c30b96e96d314c218b2940b400c33fa9
TriggeredBy: ● borgbackup-job-homeserver.timer
    Process: 36373 ExecStart=/nix/store/sx3cbbp2qv0h7nk08wiz3537bxr9awis-unit-script-borgbackup-job-homeserver-start/bin/borgbackup-job-homeserver-start >
   Main PID: 36373 (code=exited, status=2)
         IP: 3.8K in, 4.2K out
         IO: 0B read, 600K written
   Mem peak: 66.8M
        CPU: 544ms

nov. 13 14:26:43 homeserver systemd[1]: Started BorgBackup job homeserver.
nov. 13 14:26:44 homeserver borgbackup-job-homeserver-start[36388]: Remote: Host key verification failed.
nov. 13 14:26:44 homeserver borgbackup-job-homeserver-start[36388]: Connection closed by remote host. Is borg working on the server?
nov. 13 14:26:44 homeserver borgbackup-job-homeserver-start[36393]: Connection closed by remote host. Is borg working on the server?
nov. 13 14:26:44 homeserver systemd[1]: borgbackup-job-homeserver.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
nov. 13 14:26:44 homeserver systemd[1]: borgbackup-job-homeserver.service: Failed with result 'exit-code'.
nov. 13 14:26:44 homeserver systemd[1]: borgbackup-job-homeserver.service: Consumed 544ms CPU time, 66.8M memory peak, 600K written to disk, 3.8K incomin>

Second setup:

{ config, lib, pkgs, myVars, ... }:

{
  services.borgbackup = {
    jobs = {
      "homeserver" = {
        paths = myVars.general.testPath;
        environment.BORG_RSH = "ssh -i ${myVars.general.borgRsh}";
        repo = myVars.general.borgRepo;
        compression = "zstd,8";
        startAt = "hourly";
        user = myVars.mainUsers.server.user; # Added this line

        encryption = {
          mode = "repokey-blake2";
          passCommand = "cat ${myVars.general.borgPassPath}";
        };
      };
    };
  };

  environment.systemPackages = with pkgs; [ borgbackup ];
}

Result:

~$ systemctl status borgbackup-job-homeserver.service
× borgbackup-job-homeserver.service - BorgBackup job homeserver
     Loaded: loaded (/etc/systemd/system/borgbackup-job-homeserver.service; linked; preset: ignored)
     Active: failed (Result: exit-code) since Wed 2024-11-13 14:55:29 CET; 1s ago
   Duration: 1.717s
 Invocation: 8323daccdec14d2da56a92c3235a373d
TriggeredBy: ● borgbackup-job-homeserver.timer
    Process: 92545 ExecStart=/nix/store/sx3cbbp2qv0h7nk08wiz3537bxr9awis-unit-script-borgbackup-job-homeserver-start/bin/borgbackup-job-homeserver-start (code=exited, status=2)
   Main PID: 92545 (code=exited, status=2)
         IP: 5.7K in, 6.2K out
         IO: 0B read, 600K written
   Mem peak: 66.7M
        CPU: 555ms
                                                                                                                                                                                                                                                                                                                          
nov. 13 14:55:28 homeserver borgbackup-job-homeserver-start[92559]: Remote: Permission denied, please try again.
nov. 13 14:55:28 homeserver borgbackup-job-homeserver-start[92559]: Remote: [REDACTED (REPO)]: Permission denied (publickey,password).
nov. 13 14:55:28 homeserver borgbackup-job-homeserver-start[92559]: Connection closed by remote host. Is borg working on the server?
nov. 13 14:55:28 homeserver borgbackup-job-homeserver-start[92570]: Remote: Permission denied, please try again.
nov. 13 14:55:28 homeserver borgbackup-job-homeserver-start[92570]: Remote: Permission denied, please try again.
nov. 13 14:55:28 homeserver borgbackup-job-homeserver-start[92570]: Remote: [REDACTED (REPO)]: Permission denied (publickey,password).
nov. 13 14:55:28 homeserver borgbackup-job-homeserver-start[92570]: Connection closed by remote host. Is borg working on the server?
nov. 13 14:55:29 homeserver systemd[1]: borgbackup-job-homeserver.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
nov. 13 14:55:29 homeserver systemd[1]: borgbackup-job-homeserver.service: Failed with result 'exit-code'.
nov. 13 14:55:29 homeserver systemd[1]: borgbackup-job-homeserver.service: Consumed 555ms CPU time, 66.7M memory peak, 600K written to disk, 5.6K incoming IP traffic, 6.2K outgoing IP traffic.

And many more; removing stuff, adding stuff. Still nothing.

The interesting part is that manually trying to run it works just fine. Executing below commands works a charm:

~$ export BORG_REPO='[REDACTED (REPO)]'
~$ export BORG_PASSCOMMAND="cat /path/to/passcommand-file"
~$ export BORG_RSH="ssh -i /path/to/private-ssh-key"

~$ borg info ::

Repository ID: [REDACTED (ID)]
Location: [REDACTED (REPO]
Encrypted: Yes (repokey BLAKE2b)
Cache: /home/myuser/.cache/borg/[REDACTED (ID)]
Security dir: /home/myuser/.config/borg/security/[REDACTED (ID)]
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:                    0 B                  0 B                  0 B
                                                                                                                                                                                                                                                                                                                          
                       Unique chunks         Total chunks
Chunk index:                       0                    0

Addendum

At first I thought that it couldn’t read the BORG_PASSCOMMAND path, due to it being only readable with privilege escalation, but changing it to a user-readable dir did not work, either.

FWIW I suspect it might have to do with SSH, GPG, its agents, or a combination of those. I suspect this because trying to do ssh-add -L lists only my GPG key’s SSH subkey. Trying to add an SSH key using ssh-add .ssh/private-ssh-key results in Could not add identity "/path/to/private-ssh-key": agent refused operation.

Doing echo UPDATESTARTUPTTY | gpg-connect-agent allows me to add said SSH key, but that doesn’t remedy the issue.


Relevant files

# services.nix (home-manager)
{ config, lib, pkgs, myVars, ... }:

{
  services = {
    gpg-agent = {
      enable = true;

      enableSshSupport = true;
      enableExtraSocket = true;
      enableBashIntegration = true;
      enableZshIntegration = true;
      enableScDaemon = false; # Smartcard

      defaultCacheTtl = 34560000;
      defaultCacheTtlSsh = 34560000;
      maxCacheTtl = 34560000;
      maxCacheTtlSsh = 34560000;

      sshKeys = [ "[REDACTED (SSH-SUBKEY KEYGRIP)]" ];

      # extraConfig = ''
      #   allow-preset-passphrase
      # '';
    };

    ssh-agent = { enable = true; };
  };
}
# [PROGRAMS] defaults.nix (non-home-manager)
{
  programs = {
    gnupg = {
      agent = {
        enable = true;

        enableSSHSupport = true;

        settings = {
          default-cache-ttl = 34560000;
          max-cache-ttl = 34560000;
        };
      };
    };

    # ssh = { startAgent = true; }; # Conflicts with gnupg's enableSSHSupport

    zsh = { enable = true; };
  };

  environment.systemPackages = with pkgs; [
    gnupg
    zsh
  ];
}
# Excerpt from programs.nix (home-manager)
...
programs = {
...
    gpg = {
      enable = true;
    };

    ssh = {
      enable = true;

      addKeysToAgent = "yes";
      compression = true;
      forwardAgent = true;
    };
...
# [SERVICES] defaults.nix (non-home-manager)
{ config, lib, pkgs, ... }:

{
  services.openssh = {
    enable = true;

    settings = {
      X11Forwarding = true;
      PermitRootLogin = "no";
      PasswordAuthentication = false;
    };

    openFirewall = true;
  };
}

EDIT

It might also be worth mentioning that this is a headless box.

You may need to add your host’s SSH key to the backup server’s SSH authorized_keys file. See users.users.<name>.openssh.authorizedKeys.keys, for instance. Note that this would only add it to the authorized_keys file of the user that borgbackup is using for authentication(which is the correct approach, security-wise, in this case you’d add it to the user with the name corresponding to your myVars.mainUsers.server.user user.

Thanks for the suggestion :slight_smile: I am trying to set it up for borgbase; I have added the key to the repo :man_shrugging:

1 Like