I’m using nixos-rebuild with --use-remote-sudo to deploy to remote nixos hosts, and just noticed I get the “askpass” error while inside a tmux session. It works fine outside of tmux. Weird.
That’s an interesting datapoint. All my shells are in screen, perhaps this is a bug related to multiplexer environments. What terminal emulator do you use?
I’m using Kitty. I should try others as well, though.
After some more testing, it’s not so straightforward as this. Turns out nixos-rebuild --use-remote-sudo does work in tmux, but not on “old” panes. I have a long running tmux session I attach to, and attempting the command in an existing pane triggers the “askpass” error. But if I create a fresh pane and try again, the command works beautifully.
This fits with my success bypassing tmux entirely, since I typically do not have any long running terminal sessions outside of tmux. Weird how time seems to break this command.
I’d wager you have a messy gpg-/ssh-agent config. You’re likely starting multiple instances, both with a systemd user unit and without it, and the resulting socket env variable sticks around in old shells, while newer ones get the variable from systemd and don’t try to start a new instance.
I think you’re right. My $SSH_AUTH_SOCK is changing and pointing to a missing tmp file in older tmux panes. Thanks for pointing me in the right direction.
I’m running into the same issue, except I can’t get any workarounds to fix it since NIX_SSHOPTS and all ssh options seems to be totally ignored no matter what I try. I created a new ~/.ssh/config:
Host *
PasswordAuthentication No
RequestTTY force
But when I try to sudo nixos-rebuild switch --fast --flake .#varda --target-host varda@ip --use-remote-sudo. I still get prompted for a password. I also tried export NIX_SSHOPTS="-i /home/genevieve/.ssh/server"; sudo nixos-rebuild ... and still get prompted.
Of course, all of these attempts are failing with sudo: a terminal is required to read the password, even when adding things like NIX_SSHOPTS="-o RequestTTY=force".
I tried this in brand new bash sessions with no configuration at all, in foot and the gnome terminal, but nothing helped. SSH works just fine (and the options I’ve set are honored) in those sessions. Am I missing something obvious?
Long shot, but have you tried running nixos-rebuild not as root? I’m not sure if when it runs as root it’ll read ~/.ssh/config or /root/.ssh/config. Either way, running this as root is pretty pointless when you’re doing remote deployments.
Oh, also, disabling password auth on the client side at best makes the remote not allow you to log in at all, so maybe try changing that on the remote, not locally.
Edit: Right, no, disabling password auth for ssh will not magically make sudo not require a password anyway. Where did you get these workarounds? They cannot work, except ReqestTTY, but as evidenced by everyone else not being able to work around it with that there’s clearly some other bug breaking this.
Oh, you’re absolutely right. I originally ran without root, got a permission denied error, forgot that I was building on the host, and just automatically prepended sudo. In fact I had to append --use-remote-sudo.
Regarding password auth, the NixOS config I’m currently trying to apply disables it on the server. I was just using it as an example to demonstrate that NIX_SSHOPTS seemed to be ignored.
This behavior is still extremely odd, though. The command NIX_SSHOPTS="-tt" nixos-rebuild switch --fast --flake .#varda --target-host varda@10.0.0.1 --build-host varda@10.0.0.1 --use-remote-sudo gives the very strange output:
building the system configuration...
warning: Git tree '/home/genevieve/dots-flake' is dirty
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/0v9kr6rmpqsfb389qb8p3q1b1lbkdpbj-nixos-system-varda-24.05.20240303.b8697e5.drv^*'
Shared connection to 10.0.0.1 closed.
[sudo] password for varda:
error: selector 'these' matches no derivations
: No such file or directorym3zacyyq9wxs1y84w6vs1jsiqbfzfs-sshd.conf-settings.drv
: No such file or directory5gz0mbqd2n54r3nx4vhjwhmc4dp8y1-sshd.conf-final.drv
: No such file or directoryvpzlal4wapnpk74y61a8jsjrisb837-check-sshd-config.drv
: No such file or directory5nhnidr63fw7jpwk90drp21zxs0hj1-sshd.pam.drv
: No such file or directoryx2sazhrrfkpjpsxmviyd8vaji4lcr3-unit-40-wlp0s20f3.network.drv
: No such file or directory5sny96lcz7d4aknii1nhal7p7j9afi-etc-sysctl.d-60-nixos.conf.drv
: No such file or directoryhiwvrbiab2w34dkzb6minv5nzf67px-X-Reload-Triggers-systemd-networkd.drv
: No such file or directory1m17baqis06hazafqw5pl4z1ywvs6l-unit-systemd-networkd.service.drv
: No such file or directoryd4iyalzp3fc80kj52is90ia33wqwz91-ipv6-privacy-extensions.rules.drv
: No such file or directorydyqsl7f4cfz81c8ndp49szimzpvz7ha-X-Restart-Triggers-systemd-udevd.drv
: No such file or directorynwwy7vhwyvhzsjfvnm7hza62q0l77ma-unit-systemd-udevd.service.drv
: No such file or directorynppg85y5sqlx0kkvim03zrndxylhmbw-X-Restart-Triggers-systemd-sysctl.drv
: No such file or directory1yqwvf6h1d89f9f2ik626mckwxss1i4-unit-systemd-sysctl.service.drv
: No such file or directorya1kxv9sl5x0dw25vwqibbamqvp75nbv-X-Restart-Triggers-sshd.drv
: No such file or directorygp4scqv6hysgr5ch5n5kb606pvl2hc1-unit-sshd.service.drv
: No such file or directoryrd34dv9qw5494n6i48hmljj51lf8nvi-system-units.drv
: No such file or directory34hyxz5hclacsdfjmr7ndw8ahxv9fyy-hwdb.bin.drv
: No such file or directory05xy7gm315w5sj5gciss0c7pmdbwpgf-udev-rules.drv
: No such file or directoryvg41f7vm6iivk4rig2jvvys8mrvmzyb-etc.drv
: No such file or directoryv9kr6rmpqsfb389qb8p3q1b1lbkdpbj-nixos-system-varda-24.05.20240303.b8697e5.drv
bash: line 22: building: command not found
: No such file or directory52l2vhhdj1xabh6dlhia9vdcbwwjbaa-nixos-system-varda-24.05.20240303.b8697e5
Shared connection to 10.0.0.1 closed.
They’re both linux machines, so build-host isn’t necessary, but removing it I just get stuck in a loop of
[sudo] password for varda:
Shared connection to 10.0.0.1 closed.
Edit: You mentioned the other workarounds cannot work besides RequestTTY: why would -tt not work?
Edit two: It seems like most deploy tools also have issues with this (deploy-rs works but discourages typing the sudo password) so I think I’ll just give in and investigate something like NixOS 23.11 manual | Nix & NixOS.
--build-host means that you’re asking the remote to build the derivations too. It has nothing to do with architecture or OS. Not that this should be causing this failure, but it also looks like you have some nix version mismatches going on, and you are using --fast to disable nix fixing that, so who knows.
That’s the mystery nobody in this thread has gotten to the bottom of. -tt used to work, and then stopped working. Presumably somewhere in the scripts it’s overridden or something, and if anyone had a reasonable range to narrow it down to you could probably bisect it.
Yep, that’s what I’ve been recommending for a while now, just a little up the thread.
1 Like
I should have been less terse: I know what --build-host does, I just originally adapted a command from some documentation assuming you were building for a different architecture and so --build-host and --fast would be necessary. That’s what I was referring to when mentioning that they have the same architecture/OS. Regardless, thanks for the help!
1 Like