Nix store seems out of sync with my configuration

Help
1

I’m having some odd problems with my configuration.

After having an issue where a stuck systemd.automount unit was preventing me from activating my configuration, I sucessfully rolled back to a previous generation and sorted the problem out.

However, now my restic module is not working (it was working fine previously). The restic binary is not in my path, even though I can find various restic-related objects in the store. The module sets up mounts for the restic backup repos (one is a cifs share on a NAS, the other a local USB drive), and those mount at boot with no issue, even though the rest of the module they are in seems to be completely ignored.

I have tried removing the module then rebuilding, then running nix-garbage-collect (via nh clean all --keep n which I think should do the trick.

However none of this seems to work: restic is still not found, my backup drives are still mounting, and re-adding and enabling the restic module does not result in a new generation, with the output indicating that nothing has changed and the configuration is already built.

I’m a bit stumped. How can I fix this, either to relink the restic binary, or to remove the old ‘orphaned’ store objects?

If it helps, I am running nixos unstable on x86_64-linux platform, and my configuration uses Denix to manage the modules.

Thanks!

2

How exactly have you setup restic? IIRC the module does only set up an on schedule task to backup your system, but doesn’t provide a binary for ad-hoc backups or recovery tasks. You have to pull the binary by other means if you want to do that

3

There should be a wrapped binary available by default but not the unwrapped one: NixOS Search

1 Like
4

@eblechschmidt is right.

For each <name> there should indeed be a restic-<name>, but not a singular restic.

5

Thanks for the responses everyone.

This was my restic module:

# -- SERVICE: restic backups
# Lots on inspiration drawn from the following:
# https://www.arthurkoziel.com/restic-backups-b2-nixos/
# https://imranmustafa.net/simple-restic-backup-on-nixos/
{
  delib,
  config,
  inputs,
  pkgs,
  lies,
  ...
}:
delib.module {
  name = "services.restic";

  options = delib.singleEnableOption false;

  nixos.always.imports = [ inputs.sops-nix.nixosModules.sops ];

  nixos.ifEnabled =
    { myconfig, ... }:
    {
      environment.systemPackages = with pkgs; [
        restic
        libnotify # for failure notifications on restic services
        runitor # monitor systemd service and integrate with healthchecks.io
      ];

      sops = {
        secrets."restic-password" = {
          owner = myconfig.users.primary.username;
        };
      };

      services.restic = {
        backups =
          # shared options for backups
          let
            initialize = true;
            inhibitsSleep = true;
            passwordFile = config.sops.secrets.restic-password.path;
            exclude = [
              "/home/*/.cache"
              ".cargo"
              ".compose-cache"
              ".mozilla"
              ".npm"
              ".rustup"
              ".var"
              "/home/*/.local/share/containers"
              "/home/*/.local/share/docker"
              "/home/*/.local/share/flatpak"
              "/home/*/.local/share/gnome-boxes"
              "/home/*/.local/share/libvirt"
              "/home/*/.local/share/Trash"
              "/home/*/.config/chromium"
              "/home/*/.config/libvirt"
              "/home/*/git"
              "/home/*/pCloudDrive"
              "/home/*/VMs"
              "/home/*/Music"
              "/home/*/.local/share/syncthing"
              ".cache"
              "cache"
              ".tmp"
              ".temp"
              ".git"
              "tmp"
              "temp"
              ".log"
              "log"
              ".Trash"
              ".partial"
              ".config/**/blob_storage"
              ".config/**/Application Cache"
              ".config/**/Cache"
              ".config/**/CachedData"
              ".config/**/CachedStorage" # fastmail
              ".config/**/Code Cache"
              ".config/**/GPUCache"
              ".var/app/**/blob_storage"
              ".var/app/**/Application Cache"
              ".var/app/**/Cache"
              ".var/app/**/CachedData"
              ".var/app/**/GPUCache"
            ];

          in
          {
            # daily backup of home to NAS mounted cifs share
            nas = {
              inherit
                initialize
                inhibitsSleep
                passwordFile
                exclude
                ;

              paths = [
                "/home"
                "/var/lib/nixos"
              ];
              repository = "/mnt/thursday/Backups/restic-${config.networking.hostName}-home";
              timerConfig = {
                OnCalendar = "0/18:00:00";
                Persistent = true;
              };
              pruneOpts = [
                "--keep-daily 7"
                "--keep-weekly 5"
                "--keep-monthly 12"
              ];
            };

            # daily backup of home to local external USB SSD drive
            local = {
              inherit
                initialize
                inhibitsSleep
                passwordFile
                exclude
                ;

              paths = [
                "/home"
                "/var/lib/nixos"
              ];

              repository = "/mnt/media/restic-${config.networking.hostName}-home";
              timerConfig = {
                OnCalendar = "0/17:00:00";
                Persistent = true;
              };
              pruneOpts = [
                "--keep-daily 7"
                "--keep-weekly 5"
                "--keep-monthly 12"
              ];
            };

          };
      };

      # these should enable monitoring with healthchecks.io
      systemd.services = {
        restic-backups-nas = {
          preStart = "${pkgs.curl}/bin/curl -sS -m 10 --retry 5 https://hc-ping.com/${lies.hcheckPing}/morse-restic-backups-nas/start";
          postStop = "${pkgs.curl}/bin/curl -sS -m 10 --retry 5 https://hc-ping.com/${lies.hcheckPing}/morse-restic-backups-nas/\${EXIT_STATUS}";
        };

        restic-backups-local = {
          preStart = "${pkgs.curl}/bin/curl -sS -m 10 --retry 5 https://hc-ping.com/${lies.hcheckPing}/morse-restic-backups-local/start";
          postStop = ''
            ${pkgs.curl}/bin/curl -sS -m 10 --retry 5 https://hc-ping.com/${lies.hcheckPing}/morse-restic-backups-local/''${EXIT_STATUS}
          '';
        };
      };

      # Make sure that the cifs share is mounted automatically
      fileSystems."/mnt/thursday/Backups" = {
        device = "//192.168.178.67/external_backups";
        fsType = "cifs";
        options = [
          "credentials=${config.sops.templates."sambaNas".path}"
          "auto"
          "nofail"
          "x-systemd.idle-timeout=60"
          "x-systemd.device-timeout=5s"
          "x-systemd.mount-timeout=5s"
          "rw"
          "uid=1000"
          "gid=100"
        ];
      };

      # Also ensure that the external SSD for local backup is mounted at boot
      fileSystems."/mnt/media" = {
        device = "/dev/disk/by-uuid/93fb617f-9bed-420c-bbb0-9638d0df3f68";
        fsType = "btrfs";
        options = [
          "auto"
          "nofail"
          "x-systemd.idle-timeout=60"
          "x-systemd.device-timeout=5s"
          "x-systemd.mount-timeout=5s"
          "rw"
          "uid=1000"
          "gid=100"
        ];
      };

    };
}

Day to day, I used the wrapped binary to check on snapshots etc., e.g. sudo restic-nas snapshots, but I had the unwrapped binary so that I could also do an ad-hoc restore of files from a mirrored repo in a Hetzner box.

At the moment, I have moved this module file outside of my config directory, and commented out the code that enabled this module for the host. When I run any of the wrapped commands, or check the status of the systemd services (e.g. restic-backups-nas.service) I get command not found. However, the mounts for the repos you see here are still currently mounted. which restic also returns not found (for the unwrapped binary).

This of course is to be expected when you remove the relevant code in the config. However, when I had it enabled I got exactly the same result: none of the commands above were found, but the mounts were mounted.

Any idea what might be going on? Yesterday after removing this module, I changed come config for kitty and switched, and that worked fine, activating the new config and creating a new generation, so it isn’t a global problem with NixOS.

6

Tbh. this will be really hard to debug from just looking at this one file. If I were you I would start up a repl and inspect the config options that are set and go from there.

7

I’ll give that a try, thanks.

8

I think I’ve solved it, so I’m leaving this here for future me and anyone else facing similar issues.

  • Used the nix repl as suggested. Before making any more changes, restic service shows just the default/empty config and enabled is false, as expected (because there is no configuration).

  • Restored the old restic module, enabled in backups.nix then checked repl again. Full config shown, and the module is marked as enabled.

  • Switched to the new config. All the services and wrapped binaries are now accessible, but activation failed because mounting the /mnt/media (USB) mount exited. In actual fact, it is still mounted, and I can back up to it using systemctl start. The error mentioned option ‘uid’ unknown, so I deleted that and the group option.

  • Tried switching again, but now it said unit file not found for the mount. This is a problem that I have faced several times when trying to sort out automounting of shares in particular, and which I haven’t entirely fixed.

  • Commented out the relevant fileSystem call for /mnt/media, then followed the process here: How can I remove a mount unit? - Help - NixOS Discourse (rebuild switch [which failed to activate the config again or create a new generation, but I pressed on]), reboot, uncomment and fix code, switch again).

  • That worked!

So I still don’t know quite how it got into this state, but I seem to have resolved it. Thanks to all for prompting me to use nix repl. I haven’t investigated that before, so I have now learned how to use if for basic debugging, which I am sure will be very useful in the future.