Best practices for auto-updating remotely deployed systems

Help
1

I’ve recently set up a server with Colmena, but before I decided to do that my original plan was “clone the git repo with all my other configs, push/pull whenever i make changes on my laptop, and have system.autoUpgrade on”.

With Colmena however, the configuration doesn’t make it to the server, so system.autoUpgrade isn’t an option as I see it (since it basically runs nixos-rebuild)…so I’m wondering if there’s an alternative that combines being able to “push” updated configs to remote targets as well as said targets auto-updating

2

I’ve wondered this as well. I think the way this is typically handled in an organizational setting is to set up CI/CD that can do a deploy when the repo changes, and then use something like dependabot to automatically update dependencies. You can of course do manual updates, or use a little cronjob to automate that process on the laptop.

Sorry, that’s kind of a non-answer, but the best I’ve got for you.

3

I believe system.autoUpgrade.flake could be used to automate this.

4

Yes, that is correct. However then you have no control when manually pushing changes. I am currently experimenting with autoupgrades here: nixfiles/modules/nixos/autoupgrade.nix at 41072019798f14014bd25dc3bab8603312c7febf · LorenzBischof/nixfiles · GitHub

This requires the configs to be available (they can be private) and the flake lock to regularly be upgraded (can also be manually, instead of via CICD). I also build my systems in CICD to check for build failures and push the results to a Nix cache.

The autoupgrades are:

  • disabled when the manually deployed Git repository was dirty (uncomitted changes)
  • skipped if the deployed commit is not already in the default branch (not pushed or merged yet)

This ensures that manually deployed changes are not automatically downgraded. Open to suggestions and improvements

5

This is more or less what I do, too…

  • disabled when the manually deployed Git repository was dirty (uncomitted changes)
  system.autoUpgrade = {
    enable = ((inputs.self.rev or "dirty") != "dirty");
    # …
  };

I generally pull in a nix flake update --commit-lock-file on my desktop, and build that. I review the changes and may switch to the new revision for more testing.

Then I build the system closures for the rest of my systems (nu syntax):

nix flake show --json | from json | get nixosConfigurations | columns
 | each { |host| 
    pueue add $"nix build --out-link result-($host) .#nixosConfigurations.($host).config.system.build.toplevel"
   }

Then if those builds succeed, I push all the system closures to attic cache, and push the updated git revision to local soft-serve git repo. All the other hosts update when they next wake from sleep or overnight.

  • skipped if the deployed commit is not already in the default branch (not pushed or merged yet)

A neat additional idea, but not one I have needed or can implement quite as simply.

I do occasionally want to hold back a system when an upgrade is known to break things, like currently I have some hosts held back from systemd 257.8 because of a bridged vlan bug in networkd. Most of the time, I just stop the nixos-upgrade.timer, or build a dirty revision on that host. I haven’t, but I could also add a condition clause on the upgrade service that something like /etc/noautoupgrade doesn’t exist, and touch that file as another manual signal.

6

That is no coincidence. Your post was what inspired my solution: How are you handling automatic updates? - #4 by uep - Help - Aux

1 Like