Deployment tools: evaluating NixOps, deploy-rs, and vanilla nix-rebuild

[uep]

Yeah, in this case the service doesn’t actually fail, and furthermore it seems to actually do a switch even when there are no changes… so a bit more work might be needed generally.

Edit: it’s worse than that… even with --refresh, nixos-rebuild runs, fails to fetch a new revision from git, and then builds and switches to the old revision. Which can result in a rollback if the current system was built from a newer revision in /etc/nixos than what it has cached from the last time the autoupgrade ran.

https://github.com/NixOS/nixpkgs/issues/274146

[uep]

I masked the problem for now with a preStart command that checks ssh connectivity to the git server; when that fails it then follows these Restart settings.

That’s the right solution for waiting for the wifi to connect, but not for the ignored errors during the rebuild run itself.

[Solene]

If this can help, I made a video about NixOS deployment tools

A journey into the world of NixOS deployment tools

[ttamttam1]

Currently I use deploy-rs for push based operations. It plays nice with a vanilla nix flake configs and supports home manager without a backing nixos home-manager module. The login/sudo prompt is a problem. I currently just have a dedicated ssh key I use for deploying new systems but I personally don’t like adding more entrypoints to my systems. I’m looking into using system.autoUpgrade for my desktop computers as the pull based option.
I think missing right now is choice for pull based systems, Bento looks great but I have my reasons for not wanting to use it. nixos-rebuild is also pull based, but has its own issues, in particular it still performs evaluation on the client instead of build-host, which would kill my rpi. One day I might write Yet Another NixOs Deployer, but for now I just want to make sure I can push to machines and don’t miss security patches.

[prescientmoon]

I’m also having issues with the sudo prompt… Wish there was a tool which got this right (I’ve heard nixinate handles it nicely, but last I tried I was running into different issues)

[p3t3]

@hugosenari

Basically what it does is constantly asking your system to be reinstalled, pointing to a repository.
So I’m not expecting it to update your inputs. And you have to add --refresh or nix will use flake cache (not a fresh pull)

Thanks for the info I made a little tweak for my use and a systemd unit(and timer were) generated.

{ config, ... }:
{
    system.autoUpgrade.enable = true;
    system.autoUpgrade.dates  = "Fri *-*-1..7,15..21 01:00:00";
    system.autoUpgrade.flake  = "github:${config.userDefinedGlobalVariables.githubFlakeRepositoryName}#${config.userDefinedGlobalVariables.hostTag}";
    system.autoUpgrade.randomizedDelaySec = "5m";

}

From the documentation for system.autoUpgrade.enable " Whether to periodically upgrade NixOS to the latest version. If enabled, a systemd timer will run nixos-rebuild switch --upgrade once a day."

looking at the file that systemd will execute I can see a single line

nix/store/as1snmyxhr9633n30pbcy2fcbbggii4p-nixos-rebuild/bin/nixos-rebuild switch --flake github:p3t33/nixos_flake#homelab --upgrade

The --upgrade have nothing to do with a flake(and is used with channels) which is updated to the best of my knowledge with

nix flake update

which means that the command has some junk in it.

I also not sure why would you need --refresh when using a flake which is responsible to make sure to keep the state of your packages in check. If I remember correctly --refresh is a channel option.

[hugosenari]

Flake has cache of repositories, --refresh is used to bypass this.
But that can be wrong

Maybe since you will run only on Friday, flake git cache may be invalidated as “too old”.
In my case, I settled it to once every 20min, may be a problem.

https://github.com/NixOS/nix/issues/4007#issuecomment-692025931

[nrbray]

@delroth

I auto-generate a wireguard p2p mesh by having each machine look at the wireguard config of other machines in my network’s Nix config.

This sounds great. May I ask if there is a good place to start with an example you have shared or would recommend?

[jpentland]

I am also using “nixos-rebuild --target-host”

The issue is in order to apply the system config, you either have to log in as root, or use remote sudo. However there is no support for supplying a password when using remote sudo, so then you need passwordless sudo on the target. Not ideal. Looking for a solution that supports better authentication.

[bowmanjd]

Agreed. How do you feel about public key only ssh authentication as root on the target? That is one way to get around the sudo problem. But then you are logging in as root… But then I guess that is what sudo does anyway… Curious what you think.

[steinuil]

I use nixos-rebuild --target-host and disable all authentication methods except for pubkey on all my machines, and log in as root on the remote host.

@jpentland

However there is no support for supplying a password when using remote sudo, so then you need passwordless sudo on the target.

This page on the wiki mentions that you can ser NIX_SSHOPTS="-o RequestTTY=force" to use a password when using remote sudo, but I’m not sure how up to date that information is.