Dead man switch for nixos-rebuild switch / boot

aou · May 8, 2025, 12:42am

tldr; switch-fix.nix lets you set an automatic rollback to current generation / profile on nixos-rebuild [switch | boot] unless you cancel with cancel-rollback from a terminal within a set amount of time

github.com

femtodata/nix-utils/blob/d18e28bd23f7b6686565ea96a8786fccde12ec13/modules/switch-fix.nix

{ pkgs, ... }:
let
  switch-fix = pkgs.writeShellScriptBin "switch-fix" ''
    sudo nix-env --profile /nix/var/nix/profiles/system --set /run/current-system
    sudo /run/current-system/bin/switch-to-configuration switch
  '';
  boot-fix = pkgs.writeShellScriptBin "boot-fix" ''
    sudo nix-env --profile /nix/var/nix/profiles/system --set $1
    sudo $1/bin/switch-to-configuration boot
  '';
  rollback-profile-path = "/var/lib/nix-autorollback/profile";
  rollback-delay = "90";
  set-rollback = pkgs.writeShellScriptBin "set-rollback" ''
    sudo mkdir -p $(dirname ${rollback-profile-path})
    sudo ln -sf $(readlink /run/current-system) ${rollback-profile-path}
  '';
  cancel-rollback = pkgs.writeShellScriptBin "cancel-rollback" ''
    sudo systemctl stop nix-autorollback.service
    sudo rm ${rollback-profile-path}
  '';

This file has been truncated. show original

TLATER · May 8, 2025, 4:34am

Ooh, nice.

GitHub - serokell/deploy-rs: A simple multi-profile Nix-flake deploy tool. does similar things for anyone interested, I still hope that it eventually gets local deployment as a first class citizen, until then this project seems like a cool alternative

aou · May 8, 2025, 5:02am

Yup, I started with deploy-rs, liked it! The two issues I had were, switch would sometimes not “stick”, in that on reboot, a previous generation would be default, and the auto rollback was sometimes iffy, either because of internet connection issues, or not possible (at least at the time) with boot (instead of switch).

This bit of bits was originally to address that first issue, and eventually the second.

murad · June 12, 2025, 3:25pm

Sorry maybe I’m misunderstanding but what does this do that nixos-rebuild test doesn’t?

aou · June 12, 2025, 4:28pm

Two things I can think of:

nixos-rebuild test won’t reboot into a previous generation if you’ve done something technically correct, nix config-wise, but functionally terrible (like, make your remote box inaccessible through bad network config). This is not too much of a problem in a cloud instance, you usually have access to some kind of KVM console where you can force reboot, but I’ve been in situations where I have a bare-metal server somewhere without technical people around to help me reboot.
nixos-rebuilt test by definition, as far as I know, can’t test the full reboot process. That is, there are config changes that require a reboot to fully test. This module allows you to do nixos-rebuild boot, reboot, and if you leave it alone without ssh’ing in and cancel-reboot, it will roll back to the last (presumably) known good generation.

TLATER · June 12, 2025, 4:51pm

Or imagine you’ve deployed NixOS to a satellite and there somehow was a detail different in staging

Some providers also charge for rescue shell access in some configurations.

aou · June 13, 2025, 12:35am

Ok that’s a better example. I have to imagine they had all sort of failsafes for that situation, like a/b boot configs similar to phones. Hmmmmm… nixos-a-b-boot…