ssh remote command not executed on openBSD - ssh

Trying to execute command via ssh, but I'm not receiving anything back. The server A is Solaris machine and Server B is SUSE linux with OpenBSD sshd. Checked
ssh remote command does not return
https://www.openbsd.org/papers/auug2002-ssh.pdf
https://superuser.com/questions/792070/commands-over-ssh-not-producing-output
and other sources, but can't resolve it.
The command I'm trying is (in bash):
ssh -v -i .ssh/id_rsa -p1234 user2#abcdefg 'ls'
(tried without quotes and with double quotes as well )
And the debug info I'm receiving is :
Sun_SSH_1.1.6, SSH protocols 1.5/2.0, OpenSSL 0x0090704f
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: ssh_connect: needpriv 0
debug1: Connecting to ABCDEFG [100.100.100.100] port 1234.
debug1: Connection established.
debug1: identity file ../../../.ssh/id_rsa type 1
debug1: Logging to host: abcdefg
debug1: Local user: user1 Remote user: user2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2
debug1: match: OpenSSH_6.2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-Sun_SSH_1.1.6
debug1: use_engine is 'yes'
debug1: pkcs11 engine initialized, now setting it as default for RSA, DSA, and symmetric ciphers
debug1: pkcs11 engine initialization complete
debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible
Unknown code 0
)
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: Peer sent proposed langtags, ctos:
debug1: Peer sent proposed langtags, stoc:
debug1: We proposed langtags, ctos: i-default
debug1: We proposed langtags, stoc: i-default
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: dh_gen_key: priv key bits set: 128/256
debug1: bits set: 1014/2048
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'abcdefg' is known and matches the RSA host key.
debug1: Found key in /home/user1/.ssh/known_hosts:1
debug1: bits set: 1019/2048
debug1: ssh_rsa_verify: signature correct
debug1: newkeys: mode 1
debug1: set_newkeys: setting new keys for 'out' mode
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: newkeys: mode 0
debug1: set_newkeys: setting new keys for 'in' mode
debug1: SSH2_MSG_NEWKEYS received
debug1: done: ssh_kex2.
debug1: send SSH2_MSG_SERVICE_REQUEST
debug1: got SSH2_MSG_SERVICE_ACCEPT
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying public key: .ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 149 lastkey 80b0320 hint 0
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey)
debug1: SSH receive window size: 198720 B
debug1: channel 0: new [client-session]
debug1: send channel open 0
debug1: Entering interactive session.
debug1: ssh_session2_setup: id 0
debug1: Sending command: ls
debug1: channel request 0: exec
debug1: channel 0: open confirm rwindow 0 rmax 32768
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: rcvd eof
debug1: channel 0: output open -> drain
debug1: channel 0: obuf empty
debug1: channel 0: close_write
debug1: channel 0: output drain -> closed
debug1: channel 0: rcvd close
debug1: channel 0: close_read
debug1: channel 0: input open -> closed
debug1: channel 0: almost dead
debug1: channel 0: gc: notify user
debug1: channel 0: gc: user detached
debug1: channel 0: send close
debug1: channel 0: is dead
debug1: channel 0: garbage collecting
debug1: channel_free: channel 0: client-session, nchannels 1
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.1 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status 1
The ssh_config file on serverB looks like:
# $OpenBSD: sshd_config,v 1.80 2008/07/02 02:24:18 djm Exp $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# Configuration file set-up to meet APG43L needs.
Protocol 2
PasswordAuthentication yes
UsePAM yes
################################################################################
# Access to sftp subsystem is DISABLED #
# Subsystem sftp internal-sftp #
################################################################################
PermitRootLogin no
Ciphers aes128-ctr,aes256-ctr,arcfour256,arcfour,aes128-cbc,aes256-cbc
PrintLastLog no
Port 1234
# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile .ssh/authorized_keys
AllowGroups group1 group2 group3
AllowUsers user2
RSAAuthentication yes
PubkeyAuthentication yes4
EDIT:
I tried to execute the commandon the remote machine and redirect its output to remote file, but this doesn't work either. No file is created ( I can write that file in the directory if I'm on remote server). I adapted the question also.
ssh -v -i .ssh/id_rsa -p1234 user2#abcdefg 'ls > output.dat'

Weird, certainly not your command -- similar with single quotes works for me. I get the same debug output to "debug1: Sending command: ls") so everything is authenticating fine, but you're just not getting the output I then get!
Some googling implies this variation on the command gets results
echo 'ls' | ssh -v -i .ssh/id_rsa -p1234 user2#abcdefg
https://superuser.com/questions/792070/commands-over-ssh-not-producing-output
I'm assuming you have executable rights to your home directory (be amazed if you didn't)... but assumption is the mother of all f ups, so do check you can actually ls on the actual client.

Related

cannot establish a robust ssh reverse tunnel with autossh: takes long time for a new tunnel to be established after failure

I am trying to establish a robust ssh tunnel with autossh. I have a home server (home), that is reachable from the internet, and a backup server (backup), that is behind a firewall and not reachable from the internet. To be able to ssh into backup from home, I want to set up a reverse ssh tunnel.
I am able to do so, set up keys etc, and this works fine. I now want to "robustify" my ssh tunnel to changes of IPs (my ISPs provide dynamic IPs), network failures etc. I am trying to use autossh for this purpose, but with no success.
What I do:
on backup, I start the following (I forward another port than the port 22 to the port 22 of my main server, to avoid clugging of port 22 logs by people trying to break on port 22):
(note: I obfuscate a bit the logs to not provide my IP :) ):
$ /usr/bin/autossh -o ServerAliveInterval=1 -o ExitOnForwardFailure=yes -NR 8085:localhost:22 ssh_forwarding_user_on_main#address_of_main -v -p SOME_PORT_OTHER_THAN_22
Once this is done, I can do on main:
$ ssh pi#localhost -p 8085 -o PreferredAuthentications=publickey
and this works just fine, as expected :) .
However, if I change the public IP address of the backup server (by jumping from my home network to my mobile phone as an access point for example), the reverse tunnel and ssh connection die, and it seems like autossh is not able to restore the ssh reverse tunnel :( .
Looking at the logs (thanks to the -v) of the autossh reverse tunnel, I see first that:
Timeout, server MYSERVER not responding.
OpenSSH_8.4p1 Raspbian-5+b1, OpenSSL 1.1.1k 25 Mar 2021
I.e., autossh has well detected that the ssh reverse tunnel was lost, but then:
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to MY_MAIN_SERVER [MY_MAIN__IP] port SOME_PORT_OTHER_THAN_22.
debug1: Connection established.
debug1: identity file /home/pi/.ssh/id_rsa type -1
debug1: identity file /home/pi/.ssh/id_rsa-cert type -1
debug1: identity file /home/pi/.ssh/id_dsa type -1
debug1: identity file /home/pi/.ssh/id_dsa-cert type -1
debug1: identity file /home/pi/.ssh/id_ecdsa type -1
debug1: identity file /home/pi/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/pi/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/pi/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/pi/.ssh/id_ed25519 type -1
debug1: identity file /home/pi/.ssh/id_ed25519-cert type -1
debug1: identity file /home/pi/.ssh/id_ed25519_sk type -1
debug1: identity file /home/pi/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/pi/.ssh/id_xmss type -1
debug1: identity file /home/pi/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Raspbian-5+b1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2+rpt1
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2+rpt1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to MAIN:OTHER_THAN_22 as 'SOME_USER'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:OTC3Mln4Z99L2HGvnClKbT3pyKdTaEdW5fOGWXo5t3A
debug1: Host '[MAIN]:NOT_22' is known and matches the ECDSA host key.
debug1: Found key in /home/pi/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: pi#raspberrypi RSA SHA256:A29ooC6kJ8xwStSziTKZJlkJ5nzabqDR12cMhcgFR+E agent
debug1: Will attempt key: /home/pi/.ssh/id_rsa
debug1: Will attempt key: /home/pi/.ssh/id_dsa
debug1: Will attempt key: /home/pi/.ssh/id_ecdsa
debug1: Will attempt key: /home/pi/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/pi/.ssh/id_ed25519
debug1: Will attempt key: /home/pi/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/pi/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: pi#raspberrypi RSA SHA256:A29ooC6kJ8xwStSziTKZJlkJ5nzabqDR12cMhcgFR+E agent
debug1: Server accepts key: pi#raspberrypi RSA SHA256:A29ooC6kJ8xwStSziTKZJlkJ5nzabqDR12cMhcgFR+E agent
debug1: Authentication succeeded (publickey).
Authenticated to MAIN ([MAIN_IP]:NOT_22).
debug1: Local connections to LOCALHOST:51547 forwarded to remote address 127.0.0.1:51547
debug1: Local forwarding listening on ::1 port 51547.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 51547.
debug1: channel 1: new [port listener]
debug1: Remote connections from LOCALHOST:51547 forwarded to local address 127.0.0.1:51548
debug1: Remote connections from LOCALHOST:8090 forwarded to local address localhost:22
debug1: ssh_init_forwarding: expecting replies for 1 forwards
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: Remote: /home/SOME_USER/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/SOME_USER/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: remote forward failure for: listen 51547, connect 127.0.0.1:51548
Warning: remote port forwarding failed for listen port 51547
debug1: forwarding_success: all expected forwarding replies received
debug1: remote forward failure for: listen 8090, connect localhost:22
Warning: remote port forwarding failed for listen port 8090
So it seems that autossh correctly detects that the tunnel went down, and correctly re-connects to the main server. However, for some reason I do not understand, the ssh tunnel is not restored.
Edit
I went off from my computer to have a cup of coffee, and when I came back, magics, the following had been added to the output:
debug1: Connection to port 51547 forwarding to 127.0.0.1 port 51547 requested.
debug1: channel 2: new [direct-tcpip]
channel 2: open failed: administratively prohibited: open failed
debug1: channel 2: free: direct-tcpip: listening port 51547 for 127.0.0.1 port 51547, connect from 127.0.0.1 port 45238 to 127.0.0.1 port 51547, nchannels 3
debug1: channel 0: free: port listener, nchannels 2
debug1: channel 1: free: port listener, nchannels 1
Transferred: sent 24480, received 9928 bytes, in 614.4 seconds
Bytes per second: sent 39.8, received 16.2
debug1: Exit status 0
OpenSSH_8.4p1 Raspbian-5+b1, OpenSSL 1.1.1k 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
[following is similar to the initial successful connection]
and now the reverse tunnel seems to be active again :) . But it took a good 5 or 10 minutes before the tunnel came back. Any idea how to make sure the tunnel comes back faster? ...

ssh between amazon EC2 instances with X11 forwarding

I have two EC2 instances in a VPC - a bastion and dev machine.
I can ssh into the bastion and run X11 apps. However when I ssh from the bastion to the dev machine, X forwarding fails:
> ssh -vX -i ~/.ssh/my_key.pem ec2-user#X.X.X.X
…
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Remote: Can't get IP address for X11 DISPLAY.
X11 forwarding request failed on channel 0
The full log is below.
The remote instance (ie. the dev machine) has the xauth package installed, and the /etc/ssh/sshd_config file (on the dev machine) has the following entries:
X11Forwarding yes
X11UseLocalhost no
Does anyone know what the problem could be?
cheers
steve
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Connecting to X.X.X.X [X.X.X.X] port 22.
debug1: Connection established.
debug1: identity file /home/ec2-user/.ssh/my_key.pem type -1
debug1: identity file /home/ec2-user/.ssh/my_key.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm#openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm#openssh.com none
debug1: kex: curve25519-sha256#libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256#libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA Y:Y:Y:Y:Y:Y:Y:Y:Y
debug1: Host 'X.X.X.X' is known and matches the ECDSA host key.
debug1: Found key in /home/ec2-user/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/ec2-user/.ssh/my_key.pem
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to X.X.X.X ([X.X.X.X]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Remote: Can't get IP address for X11 DISPLAY.
X11 forwarding request failed on channel 0
The solution was to set X11UseLocalhost to "yes".

How to correctly graphically ssh -X into docker container?

I'm trying to ssh into a running container with
ssh -X user#IP
ssh -Y user#IP
I've launched with
docker run -it image:latest /bin/bash -c 'sudo service ssh stop;sudo service ssh start;/bin/bash'
I'm able to ssh into the container, but I can't run a GUI application. When I use echo $DISPLAY, I get localhost:10.0
I've tried dozens of solutions of ForwardX11, X11Forwarding, ForwardX11Trusted, etc.
I get this warning on login:
Warning: No xauth data; using fake authentication data for X11 forwarding.
I read the because it's a warning, it can be ignored. When trying to run a GUI application, I get
Invalid MIT-MAGIC-COOKIE-1 keyCan't open display: localhost:10.0
I've tried so many things but none have worked thus far. What can I do?
Thanks.
Just in case, here's a verbose output:
j#laptop:~/Documents/tmp/Docker/docker-system> ssh -vY moos#192.168.1.100
OpenSSH_6.6, OpenSSL 0.9.8zc 15 Oct 2014
debug1: Reading configuration data /home/j/.ssh/config
debug1: /home/j/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/centrifydc/ssh/ssh_config
debug1: /etc/centrifydc/ssh/ssh_config line 49: Applying options for *
debug1: Connecting to - [-] port 22.
debug1: Connection established.
debug1: identity file /home/j/.ssh/id_rsa type 1
debug1: identity file /home/j/.ssh/id_rsa-cert type -1
debug1: identity file /home/j/.ssh/id_dsa type -1
debug1: identity file /home/j/.ssh/id_dsa-cert type -1
debug1: identity file /home/j/.ssh/id_ecdsa type -1
debug1: identity file /home/j/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/j/.ssh/id_ed25519 type -1
debug1: identity file /home/j/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH* compat 0x04000000
debug1: An invalid name was supplied
Cannot determine realm for numeric host address
debug1: An invalid name was supplied
Cannot determine realm for numeric host address
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm#openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm#openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 9b:dc:35:c2:f3:5f:86:54:58:91:fa:87:cd:58:d9:bb
debug1: Host '-' is known and matches the ECDSA host key.
debug1: Found key in /home/j/.ssh/known_hosts:11
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentication succeeded (none).
Authenticated to - ([-]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: No xauth program.
Warning: No xauth data; using fake authentication data for X11 forwarding.
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Requesting authentication agent forwarding.
Last login: Thu May 19 15:15:45 2016 from -
j#3ce2104c8e2d:~$ gedit
debug1: client_input_channel_open: ctype x11 rchan 4 win 65536 max 16384
debug1: client_request_x11: request from ::1 39148
debug1: channel 1: new [x11]
debug1: confirm x11
debug1: channel 1: free: x11, nchannels 2
Invalid MIT-MAGIC-COOKIE-1 keyCan't open display: localhost:10.0

unable to get agent forwarding on vagrant working

Despite my best efforts, I cannot seem to get agent forwarding on Vagrant. I'm running OS X with the following vagrant/virtualbox combo:
[chrisshorrock:~]$ vboxwebsrv --version
Oracle VM VirtualBox web service Version 4.2.18
(C) 2007-2013 Oracle Corporation
All rights reserved.
4.2.18r88780
[chrisshorrock:~]$ vagrant --version
Vagrant 1.3.5
Firstly - in my vagrant file I have
config.ssh.forward_agent = true
I've confirmed that my key (on my os x box) is added to my ssh agent with:
ssh-add -L
My ~/.ssh/config file contains the following:
Host 127.0.0.1
ForwardAgent yes
I can connect to outside servers, and have confirmed that something like:
ssh -T git#github.com
Works properly (both locally and remotely indicating that ssh agent forwarding functionality seems correct on the OS X box), however, when I connect to my vagrant instance, I have no such luck:
[chrisshorrock:~]$ ssh -v vagrant#127.0.0.1 -p 2222
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/chrisshorrock/.ssh/config
debug1: /Users/chrisshorrock/.ssh/config line 35: Applying options for 127.0.0.1
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 2222.
debug1: Connection established.
debug1: identity file /Users/chrisshorrock/.ssh/id_rsa type 1
debug1: identity file /Users/chrisshorrock/.ssh/id_rsa-cert type -1
debug1: identity file /Users/chrisshorrock/.ssh/id_dsa type -1
debug1: identity file /Users/chrisshorrock/.ssh/id_dsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.1 pat OpenSSH_5*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 50:db:75:ba:11:2f:43:c9:ab:14:40:6d:7f:a1:ee:e3
debug1: Host '[127.0.0.1]:2222' is known and matches the RSA host key.
debug1: Found key in /Users/chrisshorrock/.ssh/known_hosts:29
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/chrisshorrock/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: Authentication succeeded (publickey).
Authenticated to 127.0.0.1 ([127.0.0.1]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: Requesting authentication agent forwarding.
debug1: Sending environment.
debug1: Sending env LANG = en_CA.UTF-8
debug1: Sending env LC_CTYPE = en_CA.UTF-8
Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.2.0-56-generic x86_64)
* Documentation: https://help.ubuntu.com/
Welcome to your Vagrant-built virtual machine.
Last login: Fri Nov 22 09:58:24 2013 from 10.0.2.2
[vagrant:~]$ ssh -T git#github.com
Warning: Permanently added the RSA host key for IP address '192.30.252.130' to the list of known hosts.
Permission denied (publickey).
I'm at a loss of things to try.
Problem solved. The box was using oh-my-zsh, which had the plugins defined as:
plugins=(git cp command-not-found git-extras gnu-utils history pip python ruby screen ssh-agent svn)
The culprit here was the ssh-agent helper for oh-my-zsh.

jenkins - can't ssh to remote server (key - permission denied) but works from cli

I have Jenkins running on my local machine trying to figure out remote ssh problem I'm having on a server. I'm getting this permission denied error which indicates a problem with the key, yet from the same user account on the shell, I can definitely connect.
Started by user anonymous
Building in workspace /Users/jgoodwin/jenkins/workspace/app
[postprocessor] $ /bin/sh -xe /var/folders/b0/h_wtmzss6cx11p6153y9h2cr0000gn/T/hudson4163212101874527747.sh
+ echo /Users/jgoodwin
/Users/jgoodwin
+ whoami
jgoodwin
+ ssh -i /Users/jgoodwin/.ssh/id_rsa remoteuser#server 'echo success'
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Build step 'Execute shell' marked build as failure
Finished: FAILURE
This is run right on the shell:
Jasons-MacBook-Air:~ jgoodwin$ echo $HOME
/Users/jgoodwin
Jasons-MacBook-Air:~ jgoodwin$ whoami
jgoodwin
Jasons-MacBook-Air:~ jgoodwin$ ssh -i /Users/jgoodwin/.ssh/id_rsa remoteuser#server 'echo success'
success
I'm fairly stumped - I've done quite a bit of work with hudson in the past and I don't think I had any issues doing this type of work. The error indicates a problem with the keys but they are clearly fine.
Edit:
Verbose log as per request
OpenSSH_5.9p1, OpenSSL 0.9.8x 10 May 2012
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 53: Applying options for *
debug1: Connecting to hostname [ip] port 22.
debug1: Connection established.
debug1: identity file /Users/jgoodwin/.ssh/id_rsa type 1
debug1: identity file /Users/jgoodwin/.ssh/id_rsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA ed:d4:92:3f:33:bd:dd:b9:eb:d1:b2:19:4c:f1:70:e9
debug1: Host 'hostname' is known and matches the RSA host key.
debug1: Found key in /Users/jgoodwin/.ssh/known_hosts:6
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/jgoodwin/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: key_parse_private_pem: PEM_read_PrivateKey failed
debug1: read PEM private key done: type <unknown>
debug1: read_passphrase: can't open /dev/tty: Device not configured
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Build step 'Execute shell' marked build as failure
Edit: successful attempt added 8/15
OpenSSH_5.9p1, OpenSSL 0.9.8x 10 May 2012
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 53: Applying options for *
debug1: Connecting to hostname [ip] port 22.
debug1: Connection established.
debug1: identity file /Users/jgoodwin/.ssh/id_rsa type 1
debug1: identity file /Users/jgoodwin/.ssh/id_rsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 40:bf:b5:74:1c:5f:b6:93:00:4b:ca:1d:fc:0f:39:ec
debug1: Host 'hostname' is known and matches the RSA host key.
debug1: Found key in /Users/jgoodwin/.ssh/known_hosts:3
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/jgoodwin/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to hostname ([54.226.250.218]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_CA.UTF-8
Last login: Thu Aug 15 13:09:32 2013 from 66.199.39.230
Multiple reasons could cause this behavior, like key caching with agent / keychain manager, etc.
I would suggest to use the -v argument to compare the 2 outputs:
ssh -v -i /Users/jgoodwin/.ssh/id_rsa remoteuser#server
This will let you compare what is going in in a more verbose way. If you still cannot solve it, please post your verbose outputs to compare.
Note: you can add up to 3 -v arguments for increased verbosity.
Updated:
#JasonG From what I see the details of the failure are:
debug1: Offering RSA public key: /Users/jgoodwin/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: key_parse_private_pem: PEM_read_PrivateKey failed
debug1: read PEM private key done: type
debug1: read_passphrase: can't open /dev/tty: Device not configured
It looks like your key has a passphrase and the passphrase cannot be entered because we are not in an interactive shell. The command line from your standard shell may benefit from the Keycahin which would 'type the passphrase' for you.
If you could generate the same verbosity for your successful command so that we could compare...
Jenkins run shell scripts is slight different with cmd line on environment
there are some environment difference in your case, which we didn't notice. like initial script, path setting.
besides the method provided by #coffeebreaks, try below
check the system environment, like show command env
write above steps into bash scripts and run the scripts both in cmd line and jenkins jobs
use another user instead of your initial user who starts jenkins instance
Check whether the home directory is encrypted :
http://gopukrish.wordpress.com/2013/04/24/ssh-cant-connect-authentications-that-can-continue-publickeygssapi-keyexgssapi-with-micpassword/
In my situation, I was using a keychain (as #coffeebreaks suggested) which was being setup when I sourced in the jenkins user .bashrc file. Unfortunately, the jenkins backend doesn't seem to source this file like a standard shell login does.
The solution was to add the following code prior to the scp call in the Jenkins pipeline:
. ~/.bashrc
Looks like the public key is missing under /Users/jgoodwin/.ssh/id_rsa.pub instead id_rsa there.... can you pls double check and provide permission 600 to that file and re-run your jenkins job
debug1: Offering RSA public key: /Users/jgoodwin/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: key_parse_private_pem: PEM_read_PrivateKey failed