SSH Error: Shell "bash" is not executable: No such file or directory

TL;DR

ssh may not be fully patched.

$ ssh -T git@github.com
Shell "bash" is not executable: No such file or directory

End TL;DR

$ git clone git@github.com:crinklywrappr/hayabusa.git
Cloning into 'hayabusa'...
Shell "bash" is not executable: No such file or directory
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Cloning only fails on my repositories, so it’s probably something to do with keyfiles.

The config and permissions seem correct, and I double-checked the public key in github, which is also correct. I don’t suspect an issue with this configuration, since I updated NixOS Friday and my home directory remained untouched on a separate partition. git was working prior to the upgrade.

$ bat -p .gitconfig 
[user]
    name = Daniel Fitzpatrick
    email = crinklywrappr@foo.bar

$ exa -la ~ | fgrep .ssh
drwx------   - crinklywrappr  9 Feb 20:41 .ssh

$ exa -la ~/.ssh
.rw-r--r--   56 crinklywrappr  9 Feb 20:41 config
.rw------- 3.4k crinklywrappr  2 Aug  2022 id_rsa
.rw-r--r--  745 crinklywrappr  8 Aug  2022 id_rsa.pub
.rw------- 2.6k crinklywrappr 12 Dec  2024 known_hosts
.rw------- 1.8k crinklywrappr  1 May  2024 known_hosts.old

$ ssh-add -l
4096 SHA256:... crinklywrappr@foo.bar (RSA)

I have an encrypted keyfile. The gnupg agent is enabled, so I expect to see it present a modal asking for a password. That seems to be where the trouble is w/ `Shell “bash” is not executable…".

I ran with strace and got this:

newfstatat(AT_FDCWD, "/run/current-system/sw/bin/ssh", {st_mode=S_IFREG|0555, st_size=1073912, ...}, 0) = 0
pipe2([7, 8], 0)                        = 0
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f4130befa10) = 7974
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
close(8)                                = 0
read(7, "", 8)                          = 0
close(7)                                = 0
close(3)                                = 0
close(6)                                = 0
read(5, Shell "bash" is not executable: No such file or directory
"", 4)                          = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=7974, si_uid=1000, si_status=255, si_utime=0, si_stime=0} ---
write(2, "fatal: Could not read from remot"..., 126fatal: Could not read from remote repository.

I’m not fluent in strace, but it looks like it executed ssh which tried to call out to bash. Which, of course, on NixOS will not work.

Let’s narrow it down…

$ ssh -T git@github.com
Shell "bash" is not executable: No such file or directory

I tried to work around this problem by creating a simpler key by following the instructions here, being sure to update ~/.ssh/config to use the new key, and I had the same issue. So, I think that effectively rules out my original idea that it had something to do with the passphrase.

I tried to find in the openssh repo where it might be shelling out but didn’t have any luck. I don’t know how to work around this problem and it effectively makes my system unusable.

I haven’t tried openssh unstable b/c the version is the same and none of the recent changes look like patches or bugfixes.

@das_j tagging you since you’re listed as the top maintainer on openssh. My apologies for the noise. Especially if this turns out to be PEBCAK.

I may have stumbled onto a workaround. somewhere a script or binary appears to be looking at the SHELL var, and using that to open a new shell

$ SHELL=/nix/store/gwgqdl0242ymlikq9s9s62gkp5cvyal3-bash-5.2p37/bin/bash ssh -T git@github.com
Hi crinklywrappr! You've successfully authenticated, but GitHub does not provide shell access.

$ SHELL=/nix/store/gwgqdl0242ymlikq9s9s62gkp5cvyal3-bash-5.2p37/bin/bash git clone git@github.com:crinklywrappr/hayabusa.git
Cloning into 'hayabusa'...
remote: Enumerating objects: 956, done.
remote: Counting objects: 100% (72/72), done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 956 (delta 36), reused 40 (delta 28), pack-reused 884 (from 1)
Receiving objects: 100% (956/956), 193.93 KiB | 1.78 MiB/s, done.
Resolving deltas: 100% (560/560), done.

What a strange issue!

Unfortunately this is only a partial solution. I’ll have to find a workaround for magit, too.

After looking in my .bashrc, I noticed that I had this line

SHELL=bash

Commenting it and rebooting seemed to resolve the issue.

I had almost certainly done that to simplify neofetch output. Crazy how it would cause such a headache.