NixOS Server: What to backup?

1

I have (finally…) gotten around to setting up my first NixOS based server, to soon replace my two-decade-old Debian server.

Of course, a server needs a backup. Most of the server I can quickly recreate from the NixOS configuration, but of course not all the stateful parts.

So I set up borgbackup, which works fine, and it’s backing up some selected directories specific to my server (/data/something, /var/backup/mysql). But I wonder if there are futher files or directories worth backing up that are in non-obvious places.

For example, it seems that /etc/ssh/ssh_host_ed25519_key is rather valuable, and was created when installing the server. So should I back up /etc? But that seems rather odd for a declarative system, where I’d expect to re-create /etc from my NixOS configuration.

Is the ssh key just an odd exception, or is there more valuable state in /etc? Should I just back up the full directory?
And are there other surprising locations for state?

2 Likes
2

I would use nixos configuration to relocate the ssh keys somewhere else. (services.openssh.hostKeys)

There isn’t much in /etc that’s important. Other than /etc/nixos, the ssh host keys are pretty much it.

There is at least one semi-important thing in /var, though. /var/lib/nixos contains a record of the association between user/group names and ids, which could be important to making sure the uids and gids from your backups continue to be meaningful after a restoration.

I run an ephemeral root, so to an extent, I’m essentially restoring from a backup on each boot. You can see what I go to the trouble of saving in my config here.

1 Like
3

You might find joy from

https://grahamc.com/blog/erase-your-darlings

on the other hand you may not.

Identify state on the system is important.

Security is a ‘key management problem’, and encryption keys are state.

If i can ‘talk a good config’ for a moment, use zfs , and you can create continuous snapshots and rollback state too . Maybe one day nixos can integrate that more closely with the system, so i don’t have to roll my own scripts for doing this.

nixos-rebuild switch --zfs_snap_shot_state

nixos-rebuild --rollback --zfs_snap_shot_state

these command do not exists today, it’s just a bit imagineering on my part.

What ever you do, you need to

image
image557×694 98.4 KB

1 Like
4

That’s both pretty neat! But I think I’ll stick close to the default nixos experience for now, because I for now I want to “own” as little complexity as possible, and hope that maybe in the future NixOS provides such an experience out of the box.

1 Like
5

I agree it’s a little extreme but it’s a classic bit of research.

Configuration and state management is a hot topic right now.

Managing state is not something nix does out of the box, but maybe one day it will. You never know, you might write it and commit it Nix.

Nix is not finished, unlike other technologies which may appear to be complete finished. (at dead end).

6

I use the following list in restic:

        paths = [
          "/etc/group"
          "/etc/machine-id"
          "/etc/NetworkManager/system-connections"
          "/etc/passwd"
          "/etc/subgid"
          "/home"
          "/root"
          "/var/backup"
          "/var/lib"
        ];
4 Likes