Dynamic state and backups on ephemeral rootfs

1

Hi,

I reciently switched to having an ephemeral rootfs. For context, an ephemeral
rootfs uses a throwaway rootfs (usually a tmpfs) on every boot; this forces
state to be explicit and helps manage both configuration drift and
non-reproducible states.

One of the main challenges is with this is preserving state.
Oftentimes, people use ‘state’ directories and either symlink stateful
directories around the filesystem or configure their applications to use
directories under this path.

I’d rather avoid this approach because ‘what data goes where’ becomes
part of the configuration and will also need to be maintained.
Moreover, if the same derivation is booted in a different system,
it will will yield errors, which are not indicative of a problem, but rather
a configuration bug where the model of the machine where the configuration is
being applied and the actual state of the machine have diverged.

In my particular use-case, I use btrfs subvolumes to capture ‘big projects’.
When I’m done working with some project and after verifying the backup, I
simply drop the subvolume to save space.
I don’t want to change my config every time volume comes in or out, and if I
boot the same derivation on a filesystem that does not contain that subvolume,
I’d rather not see an error.

I came out with a prototype systemd.generator(7) that generates
systemd.mount(5) units at activation-time based on the physical layout of
the filesystem:

A simple layout tree structure allows me to encode what subvolumes go where.
The advantage is that since the subvolumes MUST be there in order to represent
they are needed (and where they should be mounted), there is no state
duplication nor configuration drift.
Further, if a subvolume ‘disappears’, its representation also does and thus
the subvolume is no-longer needed.
You can find more details about this in the README of my prototype.

I’ve been using this for the past couple of days, and I’m very happy with it.
As a nice side-effect, it allows me to decide to persist ‘something’ very
easily, but just creating a subvolume for it in the right location under a
layout tree.

The main downside with my approach is that, since the mountpoints are no-longer
known at system evaluation-time, it becomes really hard to make any guarantees
about backups or generate any configuration files for backup tools.
So, I’ve been scratching my head for a little while and came up with another
activation-time generator.

I’ve described my approach for backups in this branch:

This is not implemented and is only at planning stages, but before going
forward, I would like to have a second pair of eyes and maybe hear what
you think.

Thank your for reading this far!

Roos

References

Addendum

Here’s a foresight of what I’m working my way up to:

I want to be able to generate a ISO I can simply flash in a USB key
and have my system boot in any laptop, so I can stop transporting my
laptop around.
Secrets would become ‘optional’ system components and can be applied
(if needed) without a rebuild.
Data would also become optional and I could simply btrfs receive it
from one of my storage servers and have backups working regularly.
How to pull updates from backups I have not yet solved,
but I’m not there yet :slight_smile:

4 Likes
2

I’m not sure I understood what you are trying to achieve, but:

boot the same derivation on a filesystem that does not contain that subvolume,
I’d rather not see an error.

you can create submodules at boot with systemd.tmpfiles.rules = [ "v /foo" ];