NixOS allows rolling back to previous configurations, but this only affects files managed by nix, not all system files.
This might create problems with some services:
Upgrade NixOS, which upgrades a service to a new version
Service migrates database schema to newer version
Need to roll back NixOS config due to issues
Old service version now incompatible with new database schema
Is there a way to integrate NixOS generations with btrfs/zfs snapshots? Ideally when running nixos-rebuild switch, it would create a filesystem snapshot, and rolling back to a previous generation would also revert to the corresponding filesystem snapshot.
I’ve heard the “opt-in state” approach by using e.g. Impermanence, but it doesn’t solve the database schema problem since databases are in the persistent storage for many user services.
Another example of such a problem (edited files being problematic even after rollback) might be this Reddit post.
Are there existing solutions for coordinating NixOS configuration rollbacks with filesystem data rollbacks?
Ideally one should have to option to choose between rolling back config or config and data (especially from the boot menu)
I don’t think there’s anything that would prevent this. The easiest solution would probably be to add this to a wrapper around nixos-rebuild. I haven’t seen any projects attempt this, though.
The reason why there are no pre-made projects is probably that it isn’t quite that trivial. btrfs snapshots are really cool, but not magic. You still need to put your database in backup mode if you don’t want to accidentally snapshot it mid-transaction. Since there is no generic way to ensure this happens, you can’t build a generic snapshot tool that would be able to actually do rollbacks like this.
So you’d have to hand-write the most important part to preserve data integrity for your specific system, and the snapshot commands themselves would then be rather trivial.
That said, a concerted effort to add appropriate backup preparation scripts to all NixOS modules, and a nice interface to use it, would be very cool, and could be used as the basis of tooling that provides data-rollback.
To add on this, there’s ongoing research project with some implementations that aims to be production ready around to deliver exactly the framework enabling this (with code for some hand picked examples, e.g. PostgreSQL), unfortunately, the project is just in early phases and there’s nothing to use yet.
I actually started my adventure in the nix world using disnixos, disnix, and dysnomia! I just checked and I have issues going back to 2022 haha.
Anyway, I had completely forgotten about state management in dysnomia. In summary, each service defines their own specific snapshot and rollback command with dysnomia. Then, disnix provides global disnix-snapshot <service> and disnix-restore <service> commands that provide a uniform interface over the concept of backup.
I arrived at the exact some solution when introducing contracts. One difference is in the user-facing interface. Disnix expects you to annotate a service by its type, say mysql-database which makes it harder to extend IIRC while with the contracts implementation, we use structural type (if it quacks, it’s a duck) which allows an easier extension story.
Seems to me that this is already very close to the perfect solution. After all, you wouldn’t want snapshots only during changes to nix configuration, right? So it makes sense to just add snapshots at nix-appropriate times to whatever snapshot mechanism you already have for your important mutable data.
i think a proper solution here would include tracking state deemed relevant to back up. this info could then be made use of for backups more generally, as well as for the likes of Impermanence.
Would you be interested in a file backup contract?
I came to the same realization as you just made, that this info - which state should be saved - is valuable for backups but also impermanence and filesystem snapshots.
