Nixos-rebuild remote deployments non root pam

security.pam = {
sshAgentAuth.enable = true;
services.sudo.sshAgentAuth = true;
};

Not working for remote deployments with PAM with --use-remote-sudo, still asks for password

@TLATER

1 Like

Original thread is here: Remote nixos-rebuild works with `build`, but not with `switch`

Can you confirm that on the server-side whether doing anything with sudo it doesn’t ask for a password?

on the remote server it asks for password when I use sudo

when i am logged into the server via ssh with my normal user

Have you configured your client to allow ssh agent forwarding? Try signing in with:

ssh -A <hostname>

ssh -A shania@ip works for login no password required

Yeah, then you need to configure your ssh client, that’s why I originally asked if this works without a password.

Just configure in your ~/.ssh/config:

Host <hostname>
  ForwardAgent yes

ahh, now it works with that set @TLATER

Could you explain how all of this works in an easy way now as well? And do I need both sshAgentAuth.enable = true;
services.sudo.sshAgentAuth = true;

Well, PAM (pluggable authentication module) is a generic service that checks a user’s permission to do stuff. It’s used for authentication all throughout your typical Linux distro.

Those two settings configure it firstly to enable the ssh-agent of a user to be used to authenticate them, and secondly it allows ssh-agent authentication to give authorization to use sudo (instead of a password). PAM is also configured (by default, via authorizedKeyFiles) to only permit sudo to users using your ssh key (which will be asserted by cryptographic means).

ssh-agent in turn is a service that runs when you use ssh - it can be used to store ssh passwords in memory, so you don’t have to type your password repeatedly, but it can also act as a proxy to run cryptographic operations with your ssh key.

ssh supports forwarding your ssh-agent’s authorization over the ssh protocol, so that your server’s user can use your client’s ssh-agent to do cryptographic operations (or retrieve a memory-saved password). Since this is not necessarily what you want (don’t want some malicious server to steal your passwords or do nefarious things with your ssh key), it has to be enabled on a per-connection basis - so we configure your client to allow this on your server. This should not be done on servers you do not trust.

Altogether, what this ultimately means is that when you use sudo on your server, the server checks if the ssh key your ssh session is using is permitted by authorizedKeyFiles, and if it is it will not ask you for a password. That then means you don’t need a TTY and you bypass the bug from the other thread.

It’s probably also technically a bit more secure than normal password auth, because you can’t guess a key. But in practice the complexity of SSH-agent-over-ssh-over-PAM authentication probably balances that out. Either way, it’s convenient, relatively standard, and fixes our issue.

1 Like

Thank you very much for the help and taking your time.

One last question, how does nixos-rebuild --use-remote-sudo know to use the -A flag for the ssh connection? Is that set by default?

It uses your ssh config, so the settings from here make it do that.

Ah because when I use ssh user@host, I still need to provid a password for the user and it does not use the ssh key for authentication

Right, then simply configure the user as well:

Host <hostname>
  ForwardAgent yes
  User <user>  

And don’t specify it on the command line. Same for ports, alternate hostnames, special keepalive config, etc.

Okay, but in the end I think what we did here is not really more secure than just using the root user for the deployments

I think it is better because only one ssh for my normal user not additionally for the root user, and I still have to prefix commands with sudo for sensitive operations, but overall not more secure

It’s no different from having a separate user with a password, you’re effectively just using a very long password that is stored encrypted somewhere on your personal computer’s home directory (or a yubikey…).

The main difference between this and permitting your user to use all commands without a password is that if there is an exploit of said user it doesn’t lead to immediate, full privilege escalation. It also avoids immediate root logins, which acts as a bit of a barrier to a variety of exploits.

The main difference between this and using root@<host> with a password/ssh key for root is that if SSH fails to verify your logins correctly, PAM can still prevent exploits.

Security is a game of layers, ultimately something needs permissions to deploy new configurations. We can’t prohibit that entirely, all we can do is put layers of authorization between us and someone else trying to do the same thing without permission. How many layers and what exactly they do is the question - and what’s best for which use case is up for debate. This setup adds more layers without adding much overhead once it’s configured, so I think it’s quite good for a general-purpose setup, but your use case may not require that.

What do you mean with

The main difference between this and using root@<host> with a password/ssh key for root is that if SSH fails to verify your logins correctly, PAM can still prevent exploits.

If I disable password login for root and put the public ssh key on the remote server, it looks for the private key on my local machine if that matches, how can it fail to verify here? Are you speaking of the ssh exploits that have been happening in the past?

Yeah. Software is never perfect, ssh almost certainly still has bugs that are unkown to the world today, and will get new ones as changes are made. Relying just on ssh as a barrier between the world and full access to your server has been shown to be suboptimal over and over again, not using root directly to log in (via ssh or otherwise) is basic security practice at this point. Most serious deployments configure root without a password so login is prevented at a kernel level.

1 Like