How To: really atomic updates for NixOS - avoiding breakages

By default, manuals reference sudo nixos-rebuild switch as the way to rebuild the system everywhere.

This way swaps out and restarts various components, and can cause a lot of breakages. While in theory it may work, this mechanism has so much entropy (so many moving parts) to it, that atomic updates make more sense.

These imply updates that are staged to the next boot. The current system stays unchanged, and the rebuilt system is staged as the next boot target (NixOS generation).

You do a reboot and select it, the normal and well tested boot process is less likely to fail, and if it does, there is no running system that could break.

sudo nixos-rebuild boot && reboot
5 Likes

Not to mention, if systemd or the kernel gets an update, you must do a reboot anyway.

1 Like

It depends what kind of changes you deploy. When I do a simple tweak of some config, switch is OK (and even better if I don’t want to lose the desktop state). But on larger updates I certainly do the boot approach.

Nit: IIRC systemd can self-update during runtime (and some distros do even kernel live patching), but… overall the state transition is too complex to work well for arbitrary upgrades. That’s not just about Nix(OS) itself, but also about upstreams supporting this well.

3 Likes

It’d honestly be pretty cool if we had some kind of metadata or at least a heuristic that could show users when a switch is enough to fully deploy the new system and when not.

There’s restartIfChanged and reloadTriggers, which can be used to figure that out for systemd units, and kernel or systemd versions in general are probably pretty easy to identify. Are there any other missing pieces?

6 Likes

There is GitHub - thefossguy/nixos-needsreboot: Determine if you need to reboot your NixOS machine… I’ve been meaning to try this out for ages.

4 Likes

Author here, I got busy with other things and someone else created a fork of it. I’ve been meaning to get at this to make it less worse. At the moment, it doesn’t do anything other than lookup for hash changes and then do a version compare in nix store paths that are in /nix/var/nix/profiles/system-${generation}-link/sw/bin/${binaryName}. That too, is for only some packages that I use.

Ideally, it should check for the version by asking Nix, not jump in the Nix store like an AI crawler.

2 Likes

Cool! This only makes sense if you always use nixos-rebuild boot, right?

Meant to reply to this thread ages ago (oops!)

When I saw the topic come up I remembered a blog I (18 months in!) a somewhat new person stumbled over on fedi and wished I’d known sooner… As the OP is right - switch is mentioned everywhere

Hope it’s useful to others finding this thread in future :slightly_smiling_face:

3 Likes

While I appreciate the desire for safe updates and appreciate this guide, this sort of represents a bigger problem. That is, from a user perspective, rebooting after every update is undesirable and cumbersome. There should be better confidence with in-place updates for an end-user OS, I think.

Is there some way I could submit this as something to request dev time on for the next release?

FYI on the post the nixos-rebuild --dry-activate should be nixos-rebuild dry-activate

1 Like

That’s unnecessary. You do need to reboot after every update to the kernel and init system though (at least with systemd) and likely some other specific cases. That’s overall not a NixOS-specific problem, that’s a general Linux problem.

switch is fine for simple changes that don’t require such updates.

2 Likes

Systemd can and is updated without a reboot by using systemctl daemon-reexec https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#daemon-reexec

Kernel modules can also be a reason to update as they are loaded from /run/booted-system.

4 Likes