Remote nixos-rebuild: sudo askpass problem

That’s interesting, I switched to full passwordless sudo a few months back because I got tired of typing the password all the time so it was working at one point.

Maybe the answers from this thread might help as well:

1 Like

I have already visited that thread on this quest.

I find it unreasonable that there should be no sensible way of doing this without NOPASSWD.

At this point I’m tempted to give up on this approach and try something like morph, colmena, Cachix Deploy, deploy-rs, etc. … but there are sooooooo many of them that I dread the time sink of getting to understand which are best/sufficiently suited to my needs. Even though I really don’t want to fall prey to NIH and wheel-reinventing syndromes, I wonder whether trying to roll my own simple scripting solution based on nix-copy-closure might be the way to go. As I’ve never used it beyond toy explorations, I’d rather not go down this path, but from the outside it seems like it might be the cheapest option.

1 Like

From the ssh manpage:

Multiple -t options force tty allocation, even if ssh has no local tty.

So you actually want -tt, as suggested in the original answer, to force the tty allocation.

Unfortunately, I’ve been there, done that, got the T-shirt, and it doesn’t work. Others on the internet have observed it failing too. Sorry, I don’t have the energy to search for these again, but the summary is: it used to work but no longer does.

Nevertheless, I wanted to check again in case I missed something. First I tried to check what error I get when using a single t … and it worked! Just the once. Doesn’t work with the double t.

I’m now starting to suspect that the working-only-the-first-time phenomenon might be related to some cache invalidation or ttl, which extends the window of time in which the password can be typed on the first attempt after having given it a rest.

In any case, this whole business is not good for my mental health.

1 Like

I haven’t tried the last suggestion in that thread yet but it looked promising from what I’ve seen.
It just haven’t gotten around to try it.

As for the problem in this thread I don’t have another machine I can test against because 23.05 doesn’t work yet on Raspberry Pi 4s.
I feel with you however by not wanting to use one of the other tools, if one doesn’t have a lot of systems to manage a simple shell script works really quite well and would be easy to maintain when the underlying tools work well.

I got frustrated with this as well. Maybe it’s caused by the local shell being zsh? Anyway, ages ago I used -t as well, when that stopped working I switched to a setup using security.pam.enableSSHAgentAuth for sudo. This way you can have actual authentication (and not NOPASSWD) without password typing, which sidesteps the tty issue.

2 Likes

You mean this one? All the NOPASSWDs all over the place severely tempered my interest.

I did wonder whether zsh was somehow responsible, so I did try it from local bash too, with no noticeable difference in results. I didn’t go as far as changing the remote shell.

That looks interesting. But it’s not clear to me exactly how to configure the keys.

Well it’s only three that you really require and if those are the commands required then it’s quite sensible.
At least better than full passwordless sudo.

This looks really interesting as well, thank you.
I give it a try when I look at rewriting the sudo config.

Excessive…

In my experience you can get that down to 2.

2 Likes

It simply reuses your SSH keys for authentication as well as authorization, so if you use ssh to sign into your server you already have the keys configured. If you set this up and ssh into the server, you will be able to use sudo without a password (well, assuming you also activate the sudo PAM service).

The theory is that you want to authenticate that the person trying to use sudo is in fact the person they’re claiming to be, and not someone who has managed to exploit into a privileged users’ environment somehow, so you use their SSH key to verify that instead of prompting for a password.

This has two downsides; on the one hand, you need to enable ssh agent forwarding, and while this won’t actually grant access to your private key, the remote server will be able to use the key while you are connected. So for this purpose it’s probably best to make individual keys specific to single servers, and make sure you actually check server fingerprints when prompted.

You also can’t use the password prompt as a “do you really want to do this?” prompt, though if your key is encrypted and your ssh agent isn’t set up to cache the passphrase it can kind of act like that.

Outside of this, in a vacuum, for security purposes it should in fact be better than simple password authentication due to the length of the key used. Ignoring the impact of the added complexity you get from having PAM interface with ssh, of course.

3 Likes

is there a way to only allow this kind of thing for a single key?

In particular I assume enabling this for my user keys is unsafe, as machine A could ssh into B, and run a sudo nixos-rebuild switch --target-host (without knowing the password) targeting machine A, and enabling passwordless sudo. This means any malicious script on mavhine A can get root access.

This issue could be sidestepped by protecting a specific ssh key with a passphrase, although that’s not what I want most of the time.

And in the end, I might be too paranoid. Surely nobody would design such a specific attack targeting my setup in particular. I still find it a bit weird that we have to resort to weird hacks like this. For one, I don’t even want to deploy to a remote machine. I just want to build there and then deploy to the current one. Why is sudo even required here???

pam_rssh is a newer, more recently maintained alternative to pam_ssh_agent_auth. It doesn’t have a NixOS module (yet!) but it’s on my list to add.

I’ve recently been using a hardware security key with pam_rssh for remote sudo. It’s a little annoying to tap the key three times on each deploy (once for ssh, again for profile update, and finally for activation) but it feels a bit safer than passwordless sudo or typing passwords all the time.

Since both pam_rssh and pam_ssh_agent_auth rely on agent forwarding in practice, there are some security footguns. If you are concerned that the remote host could be compromised, you want to avoid a situation where the forwarded agent is used to get access to more of your systems. I only forward keys backed by a hardware security key for that reason. I’m sure there’s more that could be improved with this setup, like maybe forwarding a different key to each host.

2 Likes

If you are paranoid enough to think about this (and you should be), you should be protecting your ssh keys with passphrases in any case. Most developer-targeting malware goes and copies the contents of ~/.ssh as the first step, immediately followed by ~/.netrc. Even if you didn’t have specific security implications around leaking your keys, leaking your keys at all is already quite catastrophic.

Such malware is known to have been distributed via Minecraft mods, gnome themes, vscode plugins, java libraries, python libraries, npm projects, crypto wallets and pretty much everything else under the sun, often crafted to look very official. The package manager ones are particularly insidious, usually being typo squatters, and hooking into the install scripts, just fat-fingering one letter once when running pip install is enough to bust your unprotected keys. The package hosts usually don’t have sufficient manpower or incentive to check all the incoming packages for this type of thing, and most of them allow arbitrary code execution on installation, so it’s a bit of a wild west.

Keys should never be unprotected at rest.

Let the ssh agent cache your passphrase if typing passwords frequently is too much of a bother, and use diceware and password managers if memorability is an issue. And/or use hardware tokens, 6-digit pins with brute force protection are really convenient, and it means your keys cannot be exposed via software alone (and are hard to expose even if someone gets ahold of your hardware token).

That’s not the issue this thread is discussing; is sudo on the remote actually required for this? You’ll need it locally for the local deployment, but the remote builds should be possible without a privileged user?

You can! Looks like the NixOS module doesn’t have an option for this by default, it just reuses services.openssh.authorizedKeysFiles: https://github.com/NixOS/nixpkgs/blob/261abe8a44a7e8392598d038d2e01f7b33cf26d0/nixos/modules/security/pam.nix#L530

You could make your own pam config line that does the same thing, but only permits keys from a specific list. Would be nice to have an option for that upstream, too.

1 Like

yeah, I’m considering adding passphrases to my ssh keys, but it still doesn’t feel safe to be able to run a single script through a ssh session and have that script be able to run sudo whenever it pleases. A more realistic example would be a neovim plugin getting compromised and running malicious code when I want to edit a file on another machine.

And yes, I am considering buying one of those hardware key thingies — I just find them a bit too expensive for now (it’s hard to justify spending 50€ on something you realistically won’t grt any benefits from, but still want for the sake of security, while a student)

I don’t know if this is a bug, but --build-host still required sudo access. I also tried building the actual derivation using --builders, but that still downloaded a shit ton of stuff on the current machine, which filled up all the available space.

In the end I bit the bullet, deleted stuff until I had about 40gb of free space, and rebuilt on the current machine (the build took about 37). It’s not great, but it did the job.

I’m using nixos-rebuild with --use-remote-sudo to deploy to remote nixos hosts, and just noticed I get the “askpass” error while inside a tmux session. It works fine outside of tmux. Weird.

That’s an interesting datapoint. All my shells are in screen, perhaps this is a bug related to multiplexer environments. What terminal emulator do you use?

I’m using Kitty. I should try others as well, though.

After some more testing, it’s not so straightforward as this. Turns out nixos-rebuild --use-remote-sudo does work in tmux, but not on “old” panes. I have a long running tmux session I attach to, and attempting the command in an existing pane triggers the “askpass” error. But if I create a fresh pane and try again, the command works beautifully.

This fits with my success bypassing tmux entirely, since I typically do not have any long running terminal sessions outside of tmux. Weird how time seems to break this command.

I’d wager you have a messy gpg-/ssh-agent config. You’re likely starting multiple instances, both with a systemd user unit and without it, and the resulting socket env variable sticks around in old shells, while newer ones get the variable from systemd and don’t try to start a new instance.