Remote nixos-rebuild: sudo askpass problem

I’m running into the following problem when trying to deploy remotely with nixos-rebuild:

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

This problem is discussed in all sorts of places, but all those I have found (except those that require connecting as root) run into a dead end on recent versions of Nix.

Do you know a solution that works on 23.05?

You’ll want to use NIX_SSHOPTS="-tt" when executing nixos-rebuild.

Check out how I use it here → nix-config/default.nix at e2cb23f73a67407d098b8d45002bcfee7e19ec47 · Kranzes/nix-config · GitHub

1 Like

I can’t spot any significant difference between this and one of the variations I have tried before. Nevertheless, I copied it and massaged it to be usable directly in the shell (adding --verbose, otherwise there is no indication when the password should be typed) to get this

(export HOST=calvin NIX_SSHOPTS="-tt"; nixos-rebuild test -s --use-remote-sudo --fast --flake ~/my-config-files/nixos/#${HOST} --target-host ${HOST} --build-host ${HOST} --verbose)

Executing gives the output

building the system configuration...
Building in flake mode.
$ nix --extra-experimental-features nix-command flakes eval --raw /home/me/my-config-files/nixos/#nixosConfigurations."calvin" --verbose
Running nix with these NIX_SSHOPTS: -tt -o ControlMaster=auto -o ControlPath=/tmp/nixos-rebuild.4qlGce/ssh-%n -o ControlPersist=60
$ nix --extra-experimental-features nix-command flakes copy --derivation --to ssh://calvin /nix/store/zf2x3kan87m31gl0spkvi4z3jr5ikxk7-nixos-system-calvin-23.05.20230604.e7603eb.drv
$ ssh -tt -o ControlMaster=auto -o ControlPath=/tmp/nixos-rebuild.4qlGce/ssh-%n -o ControlPersist=60 calvin sudo --preserve-env=NIXOS_INSTALL_BOOTLOADER -- nix-store -r /nix/store/zf2x3kan87m31gl0spkvi4z3jr5ikxk7-nixos-system-calvin-23.05.20230604.e7603eb.drv --verbose
Shared connection to calvin closed.
$ ssh -tt -o ControlMaster=auto -o ControlPath=/tmp/nixos-rebuild.4qlGce/ssh-%n -o ControlPersist=60 calvin sudo --preserve-env=NIXOS_INSTALL_BOOTLOADER -- [sudo] password for me: 
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/bin/switch-to-configuration testjw3p0kwpld-nixos-system-calvin-23.05.20230604.e7603eb
zsh:1: no matches found: [sudo]
zsh:2: bad pattern: ^[[35
Shared connection to calvin closed.
warning: error(s) occurred while switching to the new configuration

Is the problem being caused by my remote shell being zsh?

I don’t know what -tt does, in my script I just use a single -t:

I’m (even more) thoroughly confused now!

The core of this version is pretty close to what I started with, and gives me the same error that I also got at the beginning:

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

But I see that you explicitly specify an ssh identity file. (I was disappointed to have to provide a password, as I’m trying this on hosts for which I have password-less ssh configured.) So here is my attempt at feeding my ssh identity (which is one of the documented ssh defaults, anyway) into your version:

(export NIX_SSHOPTS="-t -i ${HOME}/.ssh/id_rsa"; nixos-rebuild switch -j auto --use-remote-sudo --build-host localhost --target-host calvin --flake ~/my-config-files/nixos#calvin --verbose) 

This worked … just ONCE. I cannot get it to work again. All subsequent attempts have failed with one of

  • the error shown at the top of this post
  • something containing error: getting status of '/home/me/xxxxx': No such file or directory, where the xxxxx is the password, if I try to type it too early
  • something that shows a [sudo] password for me: prompt, but still fails in some way or other

In other words, what happens depends on (at least) when exactly I try to type the password at the invisible prompt.

This is insane.

That’s interesting, I switched to full passwordless sudo a few months back because I got tired of typing the password all the time so it was working at one point.

Maybe the answers from this thread might help as well:

1 Like

I have already visited that thread on this quest.

I find it unreasonable that there should be no sensible way of doing this without NOPASSWD.

At this point I’m tempted to give up on this approach and try something like morph, colmena, Cachix Deploy, deploy-rs, etc. … but there are sooooooo many of them that I dread the time sink of getting to understand which are best/sufficiently suited to my needs. Even though I really don’t want to fall prey to NIH and wheel-reinventing syndromes, I wonder whether trying to roll my own simple scripting solution based on nix-copy-closure might be the way to go. As I’ve never used it beyond toy explorations, I’d rather not go down this path, but from the outside it seems like it might be the cheapest option.

From the ssh manpage:

Multiple -t options force tty allocation, even if ssh has no local tty.

So you actually want -tt, as suggested in the original answer, to force the tty allocation.

Unfortunately, I’ve been there, done that, got the T-shirt, and it doesn’t work. Others on the internet have observed it failing too. Sorry, I don’t have the energy to search for these again, but the summary is: it used to work but no longer does.

Nevertheless, I wanted to check again in case I missed something. First I tried to check what error I get when using a single t … and it worked! Just the once. Doesn’t work with the double t.

I’m now starting to suspect that the working-only-the-first-time phenomenon might be related to some cache invalidation or ttl, which extends the window of time in which the password can be typed on the first attempt after having given it a rest.

In any case, this whole business is not good for my mental health.

I haven’t tried the last suggestion in that thread yet but it looked promising from what I’ve seen.
It just haven’t gotten around to try it.

As for the problem in this thread I don’t have another machine I can test against because 23.05 doesn’t work yet on Raspberry Pi 4s.
I feel with you however by not wanting to use one of the other tools, if one doesn’t have a lot of systems to manage a simple shell script works really quite well and would be easy to maintain when the underlying tools work well.

I got frustrated with this as well. Maybe it’s caused by the local shell being zsh? Anyway, ages ago I used -t as well, when that stopped working I switched to a setup using security.pam.enableSSHAgentAuth for sudo. This way you can have actual authentication (and not NOPASSWD) without password typing, which sidesteps the tty issue.

1 Like

You mean this one? All the NOPASSWDs all over the place severely tempered my interest.

I did wonder whether zsh was somehow responsible, so I did try it from local bash too, with no noticeable difference in results. I didn’t go as far as changing the remote shell.

That looks interesting. But it’s not clear to me exactly how to configure the keys.

Well it’s only three that you really require and if those are the commands required then it’s quite sensible.
At least better than full passwordless sudo.

This looks really interesting as well, thank you.
I give it a try when I look at rewriting the sudo config.


In my experience you can get that down to 2.


It simply reuses your SSH keys for authentication as well as authorization, so if you use ssh to sign into your server you already have the keys configured. If you set this up and ssh into the server, you will be able to use sudo without a password (well, assuming you also activate the sudo PAM service).

The theory is that you want to authenticate that the person trying to use sudo is in fact the person they’re claiming to be, and not someone who has managed to exploit into a privileged users’ environment somehow, so you use their SSH key to verify that instead of prompting for a password.

This has two downsides; on the one hand, you need to enable ssh agent forwarding, and while this won’t actually grant access to your private key, the remote server will be able to use the key while you are connected. So for this purpose it’s probably best to make individual keys specific to single servers, and make sure you actually check server fingerprints when prompted.

You also can’t use the password prompt as a “do you really want to do this?” prompt, though if your key is encrypted and your ssh agent isn’t set up to cache the passphrase it can kind of act like that.

Outside of this, in a vacuum, for security purposes it should in fact be better than simple password authentication due to the length of the key used. Ignoring the impact of the added complexity you get from having PAM interface with ssh, of course.