I’m running nixops on GCE, and for months it was fine. Lately, I noticed that for some reason the website started going down. What I noticed is that for some reason, every few days, all the services are gone!
The services exist in /nix/store but for some reason they disappear from /etc/systemd/system.
I don’t get how that happens. I tried recreating it by rebooting, or by force rebooting, but nothing I do causes this.
My only computer which controls this deployment was off when this happened, so it couldn’t be a client-side issue.
I switched to morph, mostly because the Hetzner support is broken. While it doesn’t do the initial provisioning and they are weird about not describing a straight - forward install method, I don’t look back.
I implemented Auto-update in morph but it is a little non-straight forward.
Basically, something triggers auto - update on your system already, now you have to make sure that you have the right configuration in /etc/nixos/configuration.nix after every update AND it is idem potent, i.e. Updating from that config results in a similar setup. I ended up with a config that just imports the config from the nix store. Symlink didn’t work because the relative paths in the file were not resolved correctly then.