Remote nixos-rebuild: sudo askpass problem

Help
1

I’m running into the following problem when trying to deploy remotely with nixos-rebuild:

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

This problem is discussed in all sorts of places, but all those I have found (except those that require connecting as root) run into a dead end on recent versions of Nix.

Do you know a solution that works on 23.05?

2

You’ll want to use NIX_SSHOPTS="-tt" when executing nixos-rebuild.

Check out how I use it here → https://github.com/Kranzes/nix-config/blob/e2cb23f73a67407d098b8d45002bcfee7e19ec47/devel/default.nix#L22

2 Likes
3

I can’t spot any significant difference between this and one of the variations I have tried before. Nevertheless, I copied it and massaged it to be usable directly in the shell (adding --verbose, otherwise there is no indication when the password should be typed) to get this

(export HOST=calvin NIX_SSHOPTS="-tt"; nixos-rebuild test -s --use-remote-sudo --fast --flake ~/my-config-files/nixos/#${HOST} --target-host ${HOST} --build-host ${HOST} --verbose)

Executing gives the output

building the system configuration...
Building in flake mode.
$ nix --extra-experimental-features nix-command flakes eval --raw /home/me/my-config-files/nixos/#nixosConfigurations."calvin".config.system.build.toplevel.drvPath --verbose
Running nix with these NIX_SSHOPTS: -tt -o ControlMaster=auto -o ControlPath=/tmp/nixos-rebuild.4qlGce/ssh-%n -o ControlPersist=60
$ nix --extra-experimental-features nix-command flakes copy --derivation --to ssh://calvin /nix/store/zf2x3kan87m31gl0spkvi4z3jr5ikxk7-nixos-system-calvin-23.05.20230604.e7603eb.drv
$ ssh -tt -o ControlMaster=auto -o ControlPath=/tmp/nixos-rebuild.4qlGce/ssh-%n -o ControlPersist=60 calvin sudo --preserve-env=NIXOS_INSTALL_BOOTLOADER -- nix-store -r /nix/store/zf2x3kan87m31gl0spkvi4z3jr5ikxk7-nixos-system-calvin-23.05.20230604.e7603eb.drv --verbose
Shared connection to calvin closed.
$ ssh -tt -o ControlMaster=auto -o ControlPath=/tmp/nixos-rebuild.4qlGce/ssh-%n -o ControlPersist=60 calvin sudo --preserve-env=NIXOS_INSTALL_BOOTLOADER -- [sudo] password for me: 
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/bin/switch-to-configuration testjw3p0kwpld-nixos-system-calvin-23.05.20230604.e7603eb
zsh:1: no matches found: [sudo]
zsh:2: bad pattern: ^[[35
Shared connection to calvin closed.
warning: error(s) occurred while switching to the new configuration

Is the problem being caused by my remote shell being zsh?

4

I don’t know what -tt does, in my script I just use a single -t:
https://git.2li.ch/Nebucatnetzer/nixos/src/branch/master/scripts/remote_switch.sh

5

I’m (even more) thoroughly confused now!

The core of this version is pretty close to what I started with, and gives me the same error that I also got at the beginning:

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

But I see that you explicitly specify an ssh identity file. (I was disappointed to have to provide a password, as I’m trying this on hosts for which I have password-less ssh configured.) So here is my attempt at feeding my ssh identity (which is one of the documented ssh defaults, anyway) into your version:

(export NIX_SSHOPTS="-t -i ${HOME}/.ssh/id_rsa"; nixos-rebuild switch -j auto --use-remote-sudo --build-host localhost --target-host calvin --flake ~/my-config-files/nixos#calvin --verbose)

This worked … just ONCE. I cannot get it to work again. All subsequent attempts have failed with one of

  • the error shown at the top of this post
  • something containing error: getting status of '/home/me/xxxxx': No such file or directory, where the xxxxx is the password, if I try to type it too early
  • something that shows a [sudo] password for me: prompt, but still fails in some way or other

In other words, what happens depends on (at least) when exactly I try to type the password at the invisible prompt.

This is insane.

1 Like
6

That’s interesting, I switched to full passwordless sudo a few months back because I got tired of typing the password all the time so it was working at one point.

Maybe the answers from this thread might help as well:

1 Like
7

I have already visited that thread on this quest.

I find it unreasonable that there should be no sensible way of doing this without NOPASSWD.

At this point I’m tempted to give up on this approach and try something like morph, colmena, Cachix Deploy, deploy-rs, etc. … but there are sooooooo many of them that I dread the time sink of getting to understand which are best/sufficiently suited to my needs. Even though I really don’t want to fall prey to NIH and wheel-reinventing syndromes, I wonder whether trying to roll my own simple scripting solution based on nix-copy-closure might be the way to go. As I’ve never used it beyond toy explorations, I’d rather not go down this path, but from the outside it seems like it might be the cheapest option.

1 Like
8

From the ssh manpage:

Multiple -t options force tty allocation, even if ssh has no local tty.

So you actually want -tt, as suggested in the original answer, to force the tty allocation.

9

Unfortunately, I’ve been there, done that, got the T-shirt, and it doesn’t work. Others on the internet have observed it failing too. Sorry, I don’t have the energy to search for these again, but the summary is: it used to work but no longer does.

Nevertheless, I wanted to check again in case I missed something. First I tried to check what error I get when using a single t … and it worked! Just the once. Doesn’t work with the double t.

I’m now starting to suspect that the working-only-the-first-time phenomenon might be related to some cache invalidation or ttl, which extends the window of time in which the password can be typed on the first attempt after having given it a rest.

In any case, this whole business is not good for my mental health.

1 Like
10

I haven’t tried the last suggestion in that thread yet but it looked promising from what I’ve seen.
It just haven’t gotten around to try it.

As for the problem in this thread I don’t have another machine I can test against because 23.05 doesn’t work yet on Raspberry Pi 4s.
I feel with you however by not wanting to use one of the other tools, if one doesn’t have a lot of systems to manage a simple shell script works really quite well and would be easy to maintain when the underlying tools work well.

11

I got frustrated with this as well. Maybe it’s caused by the local shell being zsh? Anyway, ages ago I used -t as well, when that stopped working I switched to a setup using security.pam.enableSSHAgentAuth for sudo. This way you can have actual authentication (and not NOPASSWD) without password typing, which sidesteps the tty issue.

2 Likes
12

You mean this one? All the NOPASSWDs all over the place severely tempered my interest.

I did wonder whether zsh was somehow responsible, so I did try it from local bash too, with no noticeable difference in results. I didn’t go as far as changing the remote shell.

That looks interesting. But it’s not clear to me exactly how to configure the keys.

13

Well it’s only three that you really require and if those are the commands required then it’s quite sensible.
At least better than full passwordless sudo.

14

This looks really interesting as well, thank you.
I give it a try when I look at rewriting the sudo config.

15

Excessive…

In my experience you can get that down to 2.

2 Likes
16

It simply reuses your SSH keys for authentication as well as authorization, so if you use ssh to sign into your server you already have the keys configured. If you set this up and ssh into the server, you will be able to use sudo without a password (well, assuming you also activate the sudo PAM service).

The theory is that you want to authenticate that the person trying to use sudo is in fact the person they’re claiming to be, and not someone who has managed to exploit into a privileged users’ environment somehow, so you use their SSH key to verify that instead of prompting for a password.

This has two downsides; on the one hand, you need to enable ssh agent forwarding, and while this won’t actually grant access to your private key, the remote server will be able to use the key while you are connected. So for this purpose it’s probably best to make individual keys specific to single servers, and make sure you actually check server fingerprints when prompted.

You also can’t use the password prompt as a “do you really want to do this?” prompt, though if your key is encrypted and your ssh agent isn’t set up to cache the passphrase it can kind of act like that.

Outside of this, in a vacuum, for security purposes it should in fact be better than simple password authentication due to the length of the key used. Ignoring the impact of the added complexity you get from having PAM interface with ssh, of course.

3 Likes
18

is there a way to only allow this kind of thing for a single key?

In particular I assume enabling this for my user keys is unsafe, as machine A could ssh into B, and run a sudo nixos-rebuild switch --target-host (without knowing the password) targeting machine A, and enabling passwordless sudo. This means any malicious script on mavhine A can get root access.

This issue could be sidestepped by protecting a specific ssh key with a passphrase, although that’s not what I want most of the time.

And in the end, I might be too paranoid. Surely nobody would design such a specific attack targeting my setup in particular. I still find it a bit weird that we have to resort to weird hacks like this. For one, I don’t even want to deploy to a remote machine. I just want to build there and then deploy to the current one. Why is sudo even required here???

19

pam_rssh is a newer, more recently maintained alternative to pam_ssh_agent_auth. It doesn’t have a NixOS module (yet!) but it’s on my list to add.

I’ve recently been using a hardware security key with pam_rssh for remote sudo. It’s a little annoying to tap the key three times on each deploy (once for ssh, again for profile update, and finally for activation) but it feels a bit safer than passwordless sudo or typing passwords all the time.

Since both pam_rssh and pam_ssh_agent_auth rely on agent forwarding in practice, there are some security footguns. If you are concerned that the remote host could be compromised, you want to avoid a situation where the forwarded agent is used to get access to more of your systems. I only forward keys backed by a hardware security key for that reason. I’m sure there’s more that could be improved with this setup, like maybe forwarding a different key to each host.

2 Likes
20

If you are paranoid enough to think about this (and you should be), you should be protecting your ssh keys with passphrases in any case. Most developer-targeting malware goes and copies the contents of ~/.ssh as the first step, immediately followed by ~/.netrc. Even if you didn’t have specific security implications around leaking your keys, leaking your keys at all is already quite catastrophic.

Such malware is known to have been distributed via Minecraft mods, gnome themes, vscode plugins, java libraries, python libraries, npm projects, crypto wallets and pretty much everything else under the sun, often crafted to look very official. The package manager ones are particularly insidious, usually being typo squatters, and hooking into the install scripts, just fat-fingering one letter once when running pip install is enough to bust your unprotected keys. The package hosts usually don’t have sufficient manpower or incentive to check all the incoming packages for this type of thing, and most of them allow arbitrary code execution on installation, so it’s a bit of a wild west.

Keys should never be unprotected at rest.

Let the ssh agent cache your passphrase if typing passwords frequently is too much of a bother, and use diceware and password managers if memorability is an issue. And/or use hardware tokens, 6-digit pins with brute force protection are really convenient, and it means your keys cannot be exposed via software alone (and are hard to expose even if someone gets ahold of your hardware token).

That’s not the issue this thread is discussing; is sudo on the remote actually required for this? You’ll need it locally for the local deployment, but the remote builds should be possible without a privileged user?

You can! Looks like the NixOS module doesn’t have an option for this by default, it just reuses services.openssh.authorizedKeysFiles: https://github.com/NixOS/nixpkgs/blob/261abe8a44a7e8392598d038d2e01f7b33cf26d0/nixos/modules/security/pam.nix#L530

You could make your own pam config line that does the same thing, but only permits keys from a specific list. Would be nice to have an option for that upstream, too.

1 Like
21

yeah, I’m considering adding passphrases to my ssh keys, but it still doesn’t feel safe to be able to run a single script through a ssh session and have that script be able to run sudo whenever it pleases. A more realistic example would be a neovim plugin getting compromised and running malicious code when I want to edit a file on another machine.

And yes, I am considering buying one of those hardware key thingies — I just find them a bit too expensive for now (it’s hard to justify spending 50€ on something you realistically won’t grt any benefits from, but still want for the sake of security, while a student)

I don’t know if this is a bug, but --build-host still required sudo access. I also tried building the actual derivation using --builders, but that still downloaded a shit ton of stuff on the current machine, which filled up all the available space.

In the end I bit the bullet, deleted stuff until I had about 40gb of free space, and rebuilt on the current machine (the build took about 37). It’s not great, but it did the job.