Data conversion

[frank87]

Nixos is very good at using different versions of software. But especially in services like databases new versions mean conversions.

Wouldn’t it be nice to have some kind of standard in Nixos to record the current versions in the filesystem, so the activationscript can handle the conversions?

It could be very simple: some directory with files, and a makefile filled with things like:

postgresql_22    : postgresql_21
        do_postgresql_update_22
        rm $^

(and vise versa...)

Or some file with a list of updates, you can diff to detect changes.

[mightyiam]

Does the NixOS Postgres module have no migration story?

[frank87]

Don’t think so. The default package for postgresql is decided by system.stateVersion…

[bitbloxhub]

Upgrading - NixOS Manual

This script should help you.

[waffle8946]

Third time it’s been discussed in the last month, I think we can look at the existing discussions

Incrementally salvaging stateVersion

system.stateVersion is widely acknowledged to be an imperfect thing. I’d like to propose incrementally augmenting it with a less imperfect thing. Problems I want to address: Users, new and old, perennially want to update system.stateVersion. We can tell them not to until we’re blue in the face, but there’s a certain kind of OCD computer toucher who simply cannot allow anything that looks like a date in their system config to be more than a year old; and the NixOS community has, I would wager,…

Unpopular opinion: Maybe `stateVersion` shouldn't exist at all

Hello there. First, I must say that I am just an ordinary NixOS user, not a module maintainer, and certainly not an experienced expert user, so my perspective may therefore be highly one-sided. And second, I must clarify that the title is not intended to disparage the hard work of the module maintainers. For the reasons behind this statement, please read on. Today, I spent several hours investigating exactly what purpose stateVersion serves. I even attempted to use hands-on experimentation to c…

[frank87]

I know how to upgrade databases. It’s just “nixos upgrades cleanly” breaks down with services that need migration of some kind.

Your configuration expects some layout of it’s working data. This layout could be dependent on the software version. Some Disnomia-like construction could be used to track migrations of the mutable state.

[frank87]

I was in that discussion. And someone proposed to start a discussion on possible implementations for a replacement.

Creating the script framework could be relatively easy (as proposed: make could find upgrade paths from a wide range of old versions. But once I made something just with diff that worked)

But then the problems start: what if your upgrade fails… you have to build upgrade and downgrade-script. You don’t know the required upgrades until the system is activated…

[frank87]

A lot of the gory details could be copied from dysnomia: backing up before migration, standard in the updating process.

The only part missing is de data conversion itself.

[TLATER]

@frank87

I was in that discussion. And someone proposed to start a discussion on possible implementations for a replacement.

Currently you’re asking “what could a script look like?”, not answering any of the questions about upgrade failures or rollbacks that made people skeptical of this approach, so this is just a repeat of the other 3 threads.

Rather than simply “discuss” approaches, someone needs to sit down and design a version policy and process for data migration that never results in broken systems.

“We could write a script” without saying what the script does is not that.

@frank87

The only part missing is de data conversion itself.

The script @bitbloxhub linked is how the data conversion is done, that’s the “easy” part.

If you can define a robust process for the rest, that’d be an excellent proposal, but what you seem to be proposing so far (just do what dysnomia does?) would fail under many edge cases in the wild.

[frank87]

Sorry for not following the discussion close enough…