A few days ago i was rsyncing files to my server but it suddenly froze. Then i noticed that my SSH session also froze. When i tried sshing back it froze once again. Today my friend wanted me to ssh into his server so he made me a user gave it a password etc... (Note that he has succesfully ssh-ed in and sshd is running on port 22) but when i tried to connect it froze once again so i ran the command again with:
-vvv
The output is:
OpenSSH_9.1p1, OpenSSL 1.1.1q 5 Jul 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname <the ip address of the server> is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/anon/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/anon/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to ip [ip] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: connect to address ip port 22: Connection timed out
ssh: connect to host ip port 22: Connection timed out
That is the info i was able to gather but here is some general system info:
Os: arch (freshly updated)
I haven't tried much apart from updating my system since i am not an expert at SSH.
I am trying to establish a robust ssh tunnel with autossh. I have a home server (home), that is reachable from the internet, and a backup server (backup), that is behind a firewall and not reachable from the internet. To be able to ssh into backup from home, I want to set up a reverse ssh tunnel.
I am able to do so, set up keys etc, and this works fine. I now want to "robustify" my ssh tunnel to changes of IPs (my ISPs provide dynamic IPs), network failures etc. I am trying to use autossh for this purpose, but with no success.
What I do:
on backup, I start the following (I forward another port than the port 22 to the port 22 of my main server, to avoid clugging of port 22 logs by people trying to break on port 22):
(note: I obfuscate a bit the logs to not provide my IP :) ):
$ /usr/bin/autossh -o ServerAliveInterval=1 -o ExitOnForwardFailure=yes -NR 8085:localhost:22 ssh_forwarding_user_on_main#address_of_main -v -p SOME_PORT_OTHER_THAN_22
Once this is done, I can do on main:
$ ssh pi#localhost -p 8085 -o PreferredAuthentications=publickey
and this works just fine, as expected :) .
However, if I change the public IP address of the backup server (by jumping from my home network to my mobile phone as an access point for example), the reverse tunnel and ssh connection die, and it seems like autossh is not able to restore the ssh reverse tunnel :( .
Looking at the logs (thanks to the -v) of the autossh reverse tunnel, I see first that:
Timeout, server MYSERVER not responding.
OpenSSH_8.4p1 Raspbian-5+b1, OpenSSL 1.1.1k 25 Mar 2021
I.e., autossh has well detected that the ssh reverse tunnel was lost, but then:
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to MY_MAIN_SERVER [MY_MAIN__IP] port SOME_PORT_OTHER_THAN_22.
debug1: Connection established.
debug1: identity file /home/pi/.ssh/id_rsa type -1
debug1: identity file /home/pi/.ssh/id_rsa-cert type -1
debug1: identity file /home/pi/.ssh/id_dsa type -1
debug1: identity file /home/pi/.ssh/id_dsa-cert type -1
debug1: identity file /home/pi/.ssh/id_ecdsa type -1
debug1: identity file /home/pi/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/pi/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/pi/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/pi/.ssh/id_ed25519 type -1
debug1: identity file /home/pi/.ssh/id_ed25519-cert type -1
debug1: identity file /home/pi/.ssh/id_ed25519_sk type -1
debug1: identity file /home/pi/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/pi/.ssh/id_xmss type -1
debug1: identity file /home/pi/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Raspbian-5+b1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2+rpt1
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2+rpt1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to MAIN:OTHER_THAN_22 as 'SOME_USER'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:OTC3Mln4Z99L2HGvnClKbT3pyKdTaEdW5fOGWXo5t3A
debug1: Host '[MAIN]:NOT_22' is known and matches the ECDSA host key.
debug1: Found key in /home/pi/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: pi#raspberrypi RSA SHA256:A29ooC6kJ8xwStSziTKZJlkJ5nzabqDR12cMhcgFR+E agent
debug1: Will attempt key: /home/pi/.ssh/id_rsa
debug1: Will attempt key: /home/pi/.ssh/id_dsa
debug1: Will attempt key: /home/pi/.ssh/id_ecdsa
debug1: Will attempt key: /home/pi/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/pi/.ssh/id_ed25519
debug1: Will attempt key: /home/pi/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/pi/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: pi#raspberrypi RSA SHA256:A29ooC6kJ8xwStSziTKZJlkJ5nzabqDR12cMhcgFR+E agent
debug1: Server accepts key: pi#raspberrypi RSA SHA256:A29ooC6kJ8xwStSziTKZJlkJ5nzabqDR12cMhcgFR+E agent
debug1: Authentication succeeded (publickey).
Authenticated to MAIN ([MAIN_IP]:NOT_22).
debug1: Local connections to LOCALHOST:51547 forwarded to remote address 127.0.0.1:51547
debug1: Local forwarding listening on ::1 port 51547.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 51547.
debug1: channel 1: new [port listener]
debug1: Remote connections from LOCALHOST:51547 forwarded to local address 127.0.0.1:51548
debug1: Remote connections from LOCALHOST:8090 forwarded to local address localhost:22
debug1: ssh_init_forwarding: expecting replies for 1 forwards
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: Remote: /home/SOME_USER/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/SOME_USER/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: remote forward failure for: listen 51547, connect 127.0.0.1:51548
Warning: remote port forwarding failed for listen port 51547
debug1: forwarding_success: all expected forwarding replies received
debug1: remote forward failure for: listen 8090, connect localhost:22
Warning: remote port forwarding failed for listen port 8090
So it seems that autossh correctly detects that the tunnel went down, and correctly re-connects to the main server. However, for some reason I do not understand, the ssh tunnel is not restored.
Edit
I went off from my computer to have a cup of coffee, and when I came back, magics, the following had been added to the output:
debug1: Connection to port 51547 forwarding to 127.0.0.1 port 51547 requested.
debug1: channel 2: new [direct-tcpip]
channel 2: open failed: administratively prohibited: open failed
debug1: channel 2: free: direct-tcpip: listening port 51547 for 127.0.0.1 port 51547, connect from 127.0.0.1 port 45238 to 127.0.0.1 port 51547, nchannels 3
debug1: channel 0: free: port listener, nchannels 2
debug1: channel 1: free: port listener, nchannels 1
Transferred: sent 24480, received 9928 bytes, in 614.4 seconds
Bytes per second: sent 39.8, received 16.2
debug1: Exit status 0
OpenSSH_8.4p1 Raspbian-5+b1, OpenSSL 1.1.1k 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
[following is similar to the initial successful connection]
and now the reverse tunnel seems to be active again :) . But it took a good 5 or 10 minutes before the tunnel came back. Any idea how to make sure the tunnel comes back faster? ...
I'm trying to connect to an AWS server with SSH. But for some reason I can't. When I run below:
ssh -i <keypair.pem> ec2-user#<ip> -vvv
I get:
OpenSSH_7.6p1 Ubuntu-4, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "<ip>" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to <ip> [<ip>] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file <keypair.pem> type -1
debug1: key_load_public: No such file or directory
debug1: identity file <keypair.pem>-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4
I'm guessing this is a router ip ban configuration problem? What can I do?
Check the permissions and contents of your remote .ssh directory. Do have your public key in the authorized_keys file? See the following links
https://en.wikibooks.org/wiki/OpenSSH/Logging_and_Troubleshooting
http://www.oracle.com/technetwork/systems/unix/ssh-auth-keys-jsp-135687.html
https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server
I have two machines, m_remote and m_server. I can successfully set up an ssh tunnel from m_remote to m_server using:
ssh -f -N -T -R22222:localhost:22 m_remote_user_on_server#123.123.123.123 -i /path/to/.ssh/m_remote_key
Where 123.123.123.123 is the m_server IP address. And I can successfully use that tunnel from m_server to m_remote using:
ssh m_server_user_on_remote#localhost -p 22222 -i /path/to/.ssh/m_server_key
However after a period of time the tunnel becomes unresponsive and it's no longer possible to use it. I have to access the remote machine directly and restart the tunnel. Is there any way of debugging this? From the remote all I get is:
$ ssh -vvvv m_server_user_on_remote#localhost -p 22222 -i /home/server_user/.ssh/m_server_key
OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /home/server_user/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "localhost" port 22222
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to localhost [127.0.0.1] port 22222.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /home/server_user/.ssh/m_server_key type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/server_user/.ssh/m_server_key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1
Then after a long pause of at least 2 minutes it logs out:
ssh_exchange_identification: Connection closed by remote host
Though this is due to a cron job I've set up to periodically kill and restart the tunnel.
** Edit **
On OSX 10.11 it's the same debug log with an additional line:
...
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: permanently_drop_suid: 501 # <-- additional line before hanging
I'm not certain as I need to diagnose and correlate this but as the m_remote machine is on a flaky home wifi connection this may be dropping and causing the tunnel to hang.
Alternatively there may be some config option such as ControlPersist (which may only be relevant to master connections) which has a low default of perhaps 10 minutes which could be increased.
if I try t connect to my remote server via SSH I get stuck to this, and I can't figure out how to going on.
tiziano#tiziano-HP ~ $ ssh -Tvvv -p 21 username#example.com
OpenSSH_6.2p2 Ubuntu-6ubuntu0.3, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to example.com [10.10.10.10] port 21.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/tiziano/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/tiziano/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/tiziano/.ssh/id_rsa-cert type -1
debug1: identity file /home/tiziano/.ssh/id_dsa type -1
debug1: identity file /home/tiziano/.ssh/id_dsa-cert type -1
debug1: identity file /home/tiziano/.ssh/id_ecdsa type -1
debug1: identity file /home/tiziano/.ssh/id_ecdsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2p2 Ubuntu-6ubuntu0.3
debug1: ssh_exchange_identification: 220 (vsFTPd 2.0.5)
debug1: ssh_exchange_identification: 530 Please login with USER and PASS.
Obviusly the username and the domain(ip addres) showed above are fake, I changed them.
$ ssh -Tvvv -p 21 username#example.com
Port 21 is is the default port for FTP. You're connecting to an FTP server, not an SSH server. The "530 Please login with USER and PASS" message is a typical FTP server error message.
The default port for SSH is 22, not 21.