Dealting with configurations that are set in a database

Help
1

For me the main appeal of nix was the fact that I can deploy a new machine with a single command(using something like nixosanywhere) and everything will be set exactly like I want it to be. and at least when it comes to my desktop this is the case but when it comes to my homelab things get more complicated.

To be more specific some services use databases, Now I am more then fine with the fact that calibre-web uses calibre.db for all the books , and same can be said for jellyfin. It makes a lot of sense to use database for runtime data and I don"t see any reason to define it as state. But the problem is some services use database to store configuration as well.

My homelab running more then 10 services that use a database(some communicate with each other), and I “dread” the day when I will need to migrate to a new machine.

So far I came with a few options.

  1. I can(to some extent) point my services to use databases that I will store inside of a syncthing
    directory. I already do that for taskwarrior as I need to share the database between multiple machines and taskwarrior is not a service its a cli tool.

  2. I can use

 system.activationScripts.<some name>

which is a script that gets activated after a build. This way I will be able to copy the .db file
once on migration(and put a flag file, or parse some output form the database). I already use this to set my samba username(just once). I can use a subset and inject desired values directly into the databse as well.

But what do you think? is there a nix idiomatic way to address this?

2

Unless the service has an api for asserting the contents of the configuration as contained in the database or the project explicitly considers the database to be a public api it’s asking to get burnt IMO. At minimum I’d want the activation script to take a dump of whatever tables it touches before touching them just in case.

3
  1. Don’t sync databases with Syncthing unless you only are using the data on one machine at a time. And only sqlite…I don’t think syncing a MySql/Postres db would ever work right.
  2. If your NixOS config matches the new setup, you can use MySql/Postgres database dumps to restore data, but it’s going to be a manual restore. I use these modules to periodically grab atomic (I think that’s the right word) snapshots of my MySql/Postgres databases in case I need to restore data manually: (full backup config for my home server)
# Postgres backup                                                                                                                                                                          
  services.postgresqlBackup = {                                                                                                                                                              
    enable = true;                                                                                                                                                                           
    location = "/var/backups";                                                                                                                                                               
    backupAll = true;                                                                                                                                                                        
  };                                                                                                                                                                                         
  # Mysql/maridb backup                                                                                                                                                                      
  services.mysqlBackup = {                                                                                                                                                                   
    enable = true;                                                                                                                                                                           
    location = "/var/backups";                                                                                                                                                               
    databases = [                                                                                                                                                                            
      "nextcloud"                                                                                                                                          
    ];                                                                                                                                                           
    singleTransaction = true;                                                                                                                                     
    user = "root";                                                                                                                                                                           
  };
  1. Using something like ZFS snapshots, you might get lucky and be able to use a snapshot of the database directory with a low-traffic server, but I wouldn’t exclusively rely on that because the snapshot could easily catch the database mid-transaction and you end up with potentially lost/corrupted data.
4

Definitely the way to go.

If you don’t want to manually create services, you can use e.g. the restic module with its backupPrepareCommand: NixOS Search

Just throw a script in there that creates a database dump with the DB API for that and you’re golden.

If you’d actually like to have the configuration be declarative, you’ll need to run scripts on every launch that use whatever APIs your services expose. E.g. for nextcloud there is the nextcloud-occ command, other services may need to be configured through the web API with a special user.

You should create regular database dumps for data backup in either case.

5

Just to add some more context the emphasis is to seed the database with configurations only, with the actual data that is being managed being only a bonus.

As a conceptual example, When it comes to syncthing I defined everything that is
required for a new deployment with the actual data synced as soon as synchting starts at the first time.

This is a good point, I am not sure if all of my services have API but I will give it a look.

Yes, I am not looking to create a race conditions. I am using syncthing only for databases that are accessed from a single machine at a time by me, such as taskwarrior. And thanks about the remark about MySql/Postres db I wasn’t aware that they “work” differently form sqlite.

Thanks, I am actually already using restic to backup some personal data so I think I can go with adding the databases as well. And then to use the activation script to import it once if some if statement fails. This is still problematic(fully automatic deployment) as restic is not available on first install.

6

Also want to add, you really want to avoid creating activation scripts for any purpose if you can help it. If they fail, you can’t update or even boot your system.

7

That’s true. It makes a lot more sense to probably just attach this stuff as requisite systemd dependencies to the service itself. I have a real tendency to forget to apply hard-won lessons in production (systemd-unit all the things) to people’s homelab configurations :upside_down_face:

8

It does makes sense to move this to one shot systemd service but I am not sure why a failure in activation script is catastrophic.

I assume that using something like nixosanywhere on first boot will still work(I remember I had problems with sops activation scripts failing and system still booted.

And If we are talking about a running system then you will fail to do the switch to a new config but should still have the current one.

9

It’s catastrophic because activation runs at boot? I don’t understand your question.

10

I was under the impression it only runs when I apply the config with nixos-rebuild switch and that is all. meaning if it doesn’t fail on first boot then you are good.

But if activation scripts run on every boot then this is very bad. everything will be working until it doesn’t and no easy way to fix this(is there a way to fix this?). And now that I think about it it makes sense they are running on every boot(sops needs to open everything each boot is just one example that comes to mind).

Isn’t there a way to silence errors form custom activation scripts? like using
set +e or something like “ || true’”?

but regardless I see your point now and will be updating my configurations to use systemd one shot units.

11

Which reminds me, I need to ask about upstreaming all our sops modifications, because we moved it all to purely systemd out of activation scripts because sometime aws kms would fail in stupid ways and systemd just handles this and activation scripts don’t.

12

Not really, the concept of NixOS is inherently that the system gets fully redeployed on boot. This is ultimately not all that problematic though, you can always boot a previous generation.

Avoiding activation scripts where possible is still best, since that’s the time when your deployment is most vulnerable, and you’re subject to the most possible deployment variance. The activation script should be as atomic as feasible.

sops-nix deliberately doesn’t use a systemd service because it wants to support using secrets during init. Systemd-in-init isn’t yet the default on NixOS.

vars is where the ecosystem is headed anyway, and seems more suited to the kind of extension you describe.

13

Fair, now I can ignore doing that guilt-free! Thanks :slight_smile:

14

If you’re willing to share, I’d definitely make use of those! I’ve been hoping for such a design upstream :slight_smile:

15

I’ll see what I can dig out