My nixops deploy periodically disappears

I’m running nixops on GCE, and for months it was fine. Lately, I noticed that for some reason the website started going down. What I noticed is that for some reason, every few days, all the services are gone!

The services exist in /nix/store but for some reason they disappear from /etc/systemd/system.

I don’t get how that happens. I tried recreating it by rebooting, or by force rebooting, but nothing I do causes this.

My only computer which controls this deployment was off when this happened, so it couldn’t be a client-side issue.

What could be causing this?

I once had a similar effect by auto - updating being on. It used a more or less empty nixos config that I had used for installation.

Maybe you have a similar effect? Check your /etc /nixos directory for older configs. Check for auto update timer units in systemd.

1 Like

Thanks, that seems to solve the problem.

But is there a way to properly auto-update?

I switched to morph, mostly because the Hetzner support is broken. While it doesn’t do the initial provisioning and they are weird about not describing a straight - forward install method, I don’t look back.

I implemented Auto-update in morph but it is a little non-straight forward.

Basically, something triggers auto - update on your system already, now you have to make sure that you have the right configuration in /etc/nixos/configuration.nix after every update AND it is idem potent, i.e. Updating from that config results in a similar setup. I ended up with a config that just imports the config from the nix store. Symlink didn’t work because the relative paths in the file were not resolved correctly then.

If you use krops that it should just work because the nixos configuration is uploaded via rsync before every deploy. This is my krops server: dotfiles/default.nix at a5330b93fb0e390a1e396fc58847a4653933acbf · Mic92/dotfiles · GitHub