How to remotely capture traffic across multiple SSH hops? - ssh

I want to debug another machine on my network but have to pass through one or more SSH tunnels to get there.
Currently:
# SSH into one machine
ssh -p 22 me#some_ip -i ~/.ssh/00_id_rsa
# From there, SSH into the target machine
# Note that this private key lives on this machine
ssh -p 1234 root#another_ip -i ~/.ssh/01_id_rsa
# Capture debug traffic on the target machine
tcpdump -n -i eth0 -vvv -s 0 -XX -w tcpdump.pcap
But then it's a pain to successively copy that .pcap out. Is there a way to write the pcap directly to my local machine, where I have wireshark installed?

You should use ProxyCommand to chain ssh hosts and to pipe output of tcpdump directly into wireshark. To achieve that you should create the following ssh config file:
Host some_ip
IdentityFile ~/.ssh/00_id_rsa
Host another_ip
Port 1234
ProxyCommand ssh -o 'ForwardAgent yes' some_ip 'ssh-add ~/.ssh/01_id_rsa && nc %h %p'
I tested this with full paths, so be carefull with ~
To see the live capture you should use something like
ssh another_ip "tcpdump -s0 -U -n -w - -i eth0 'not port 1234'" | wireshark -k -i -
If you want to just dump pcap localy, you can redirect stdout to filename of your choice.
ssh another_ip "tcpdump -n -i eth0 -vvv -s 0 -XX -w -" > tcpdump.pcap
See also:
https://serverfault.com/questions/337274/ssh-from-a-through-b-to-c-using-private-key-on-b
https://serverfault.com/questions/503162/locally-examine-network-traffic-of-remote-machine/503380#503380
How can I have tcpdump write to file and standard output the appropriate data?

Related

Create a config file for ssh command

I have a ssh command as below:
ssh -o ProxyCommand="ssh ubuntu#ip_addr -W %h:%p" ubuntu#ip_addr2 -L port:ip_addr3:port
I want to create a config file for this command, but I don't know what is the option of -L, here is my config file so far:
Host cassandra-khatkesh
User ubuntu
Hostname ip_addr2
ProxyCommand ssh ubuntu#ip_addr -W %h:%p
Anyone knows how can I add -L to config file?
-L corresponds to the LocalForward keyword.
Host cassandra-khatkesh
User ubuntu
Hostname ip_addr2
ProxyCommand ssh ubuntu#ip_addr -W %h:%p
LocalForward port ip_addr3:port
Note that the local and remote endpoints are specified separately, not as single :-delimited string.

How to do ssh jump over two jump hosts in command line

I can't get connection chain with ssh one liner to work.
Chain:
My PC -> jumphost -> Bastion -> my app X host(sharing subnet with Bastion)
-Jumphost expect private key A
-Bastion and X host both expect private key B
my pc> ssh -i /path_to_priv_key_for_X/id_rsa -o StrictHostKeyChecking=no -o
"ProxyCommand ssh -p 22 -W %h:%p -o \"ProxyCommand ssh -p 24 -W %h:%p
-i /path_to_key_jump/id_rsa jumphostuser#jumphostdomain\" -i
/path_to_bastion_key/id_rsa bastionuser#ip_to_bastion" myappuser#subnet_ip
Above does not work, but
ssh -i /path_to_bastion_key/id_rsa -o "ProxyCommand ssh -p 24 -W
%h:%p -i /path_to_key_jump/id_rsa jumphostuser#jumphostdomain"
bastionuser#ip_to_bastion
works, so I can access bastion with one liner, but adding app x host in the command chain does not work, wonder why?
I can step by step manually access the myapp X host like this
mypc> ssh -p 24 -i path_to_key_jump/id_rsa jumphostuser#jumphostdomain
jumphost> ssh -i /path_to_bastion_key/id_rsa bastionuser#ip_to_bastion
bastion> ssh myappuser#subnet_ip
myapp>
How to make in command line two hops over two jump hosts both requiring different key without ssh config?
Something which is working for me surprisingly well is ssh with -J option:
-J destination
Connect to the target host by first making a ssh connection
to the jump host described by destination and then establishing a TCP
forwarding to the ultimate destination from there.
In fact, I's about its feature which I was not aware of for very long time:
Multiple jump hops may be specified separated by comma characters.
So multi-hop like PC -> jump server 1 -> jump server 2 -> target server (in my example: PC -> vpn -> vnc -> ece server can be done with one combo:
$ ssh -J vpn,scs694#tr200vnc rms#tr001tbece11
Of course, most handy is to have ssh keys to open pwd-less connections (PC->vpn and vpn -> vnc and vnc -> target.
I hope it will help,
Jarek
To add to the above. My use-case was a triple-hop to a database server, which looked like Server 1 (Basic Auth) --> Server 2 (Token) --> Server 3 (Basic Auth) --> DB Server (Port Forward).
After quite a few hours of turmoil, the solution was:
ssh -v -4 -J username#server1,username#server2 -N username#Server3 -L 1122:dbserver:{the_database_port_number}
Then I was able to just have the DB client hit localhost:1122 where 1122 can be any free port number on your localhost.

Get incoming ssh forwarded connection port number

I have a server who is forwarding connections to a set of other servers.
Here I forward all incomming connections on:
my.tunnel.com:33199 to my.server2.com:52222
And..
my.tunnel.com:33200 to my.server3.com:52222
.. until
my.tunnel.com:XXXXX to my.serverN.com:52222
I'm initiating this by the following command on each server, except the tunnel my.tunnel.com:
ssh -o StrictHostKeyChecking=no -l root -i /etc/ssh/id_rsa -R *:33199:127.0.0.1:22 -p 443 my.tunnel.com 0 33199
...
ssh -o StrictHostKeyChecking=no -l root -i /etc/ssh/id_rsa -R *:XXXXX:127.0.0.1:22 -p 443 my.tunnel.com 0 XXXXX
Well, this works fine!
But!
At the point of the launching of each of these commands I'd like to check on my.tunnel.com that my.server2.com wants my.tunnel.com to forward exactly from port 33199, but not another port! So at this point I'd like to get this port number.
Please let me know if the problem is still not enough clearly exposed.
Thanks!
To get the forwarded port
There is no such information in the environmental variables, so you must pass it yourself:
ssh -R 33199:127.0.0.1:22 my.tunnel.com "export MY_FWD_PORT=33199; my_command"
(my_command is the script you want to run on the server). More information about passing variables - https://superuser.com/q/163167/93604
To get the source port
Look at the environment variable SSH_CONNECTION in man ssh(1). Its meaning is:
source_ip source_port dest_ip dest_port
You probably want source_port, so just get the second part of it:
echo $SSH_CONNECTION | awk '{ print $2 }'
or
echo $SSH_CONNECTION | cut -d" " -f 2

Mosh via two-level ssh (FreeBSD, jails)

I am fond of mosh but I have problem connecting via two-level ssh. Consider this scenario:
host machine running FreeBSD which has closed all ports from outside
first jail having ssh port 2222 open from the outside is on public IP let's say door.example.com
second jail with private IP address named DEV.example.com that can be ssh-ed from door.example.com on port 2222 as well
redirection is set up to forward udp port 60000 from door.example.com to DEV.example.com
There is generaly some problem with ttys and jails, but I am able to connect this way:
ssh -t -t -p2222 door.example.com -- ssh -p2222 DEV.example.com
being asked for both password to door.example.com and DEV.example.com afterwards.
I have tried this mosh command (also tried all variations with and without -t -t params):
mosh --port 60000 \
--ssh "ssh -t -t -p2222" \
--server "ssh -t -t -p2222 DEV.example.com mosh-server" \
door.example.com
but I always get hanging on password authentication to the second jail with no password prompt.
Funny thing is that from android mosh-flavored irssi connect bot this works when I set up mosh port to 60000 and as mosh server I fill in ssh -t -t -p2222 DEV.example.com mosh-server
I know there are ways to set-up ssh proxy but I don't want to have things like netcat on the door jail. This should work somehow especially because it already works from my phone.
Is there a reason the mosh-server needs to be at the end point (dev) rather than at the entry (door)?
I use something like:
mosh --port 60000 \
--ssh "ssh -t -t -p2222" \
-- door.example.com ssh -t -t -p2222 dev.example.com
For my setup at home.
FWIW, I use something like this for irssi:
mosh --ssh="ssh -p2222" \
-- user#dmz.example.com ssh -q -t user#irssi.example.com \
screen -c /home/user/.screen.irc -UxaA irc
Both my servers are FreeBSD and clients are either MacBook Air or a laptop running Ubuntu. I had gone with a dmz host with host based firewall, to overcome the limited forwards available on my current router.

ssh: check if a tunnel is alive

I have written a small bash script which needs an ssh tunnel to draw data from a remote server, so it prompts the user:
echo "Please open an ssh tunnel using 'ssh -L 6000:localhost:5432 example.com'"
I would like to check whether the user had opened this tunnel, and exit with an error message if no tunnel exist. Is there any way to query the ssh tunnel, i.e. check if the local port 6000 is really tunneled to that server?
Netcat is your friend:
nc -z localhost 6000 || echo "no tunnel open"
This is my test. Hope it is useful.
# $COMMAND is the command used to create the reverse ssh tunnel
COMMAND="ssh -p $SSH_PORT -q -N -R $REMOTE_HOST:$REMOTE_HTTP_PORT:localhost:80 $USER_NAME#$REMOTE_HOST"
# Is the tunnel up? Perform two tests:
# 1. Check for relevant process ($COMMAND)
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND
# 2. Test tunnel by looking at "netstat" output on $REMOTE_HOST
ssh -p $SSH_PORT $USER_NAME#$REMOTE_HOST netstat -an | egrep "tcp.*:$REMOTE_HTTP_PORT.*LISTEN" \
> /dev/null 2>&1
if [ $? -ne 0 ] ; then
pkill -f -x "$COMMAND"
$COMMAND
fi
Autossh is best option - checking process is not working in all cases (e.g. zombie process, network related problems)
example:
autossh -M 2323 -c arcfour -f -N -L 8088:localhost:80 host2
This is really more of a serverfault-type question, but you can use netstat.
something like:
# netstat -lpnt | grep 6000 | grep ssh
This will tell you if there's an ssh process listening on the specified port. it will also tell you the PID of the process.
If you really want to double-check that the ssh process was started with the right options, you can then look up the process by PID in something like
# ps aux | grep PID
Use autossh. It's the tool that's meant for monitoring the ssh connection.
We can check using ps command
# ps -aux | grep ssh
Will show all shh service running and we can find the tunnel service listed
These are more detailed steps to test or troubleshoot an SSH tunnel. You can use some of them in a script. I'm adding this answer because I had to troubleshoot the link between two applications after they stopped working. Just grepping for the ssh process wasn't enough, as it was still there. And I couldn't use nc -z because that option wasn't available on my incantation of netcat.
Let's start from the beginning. Assume there is a machine, which will be called local with IP address 10.0.0.1 and another, called remote, at 10.0.3.12. I will prepend these hostnames, to the commands below, so it's obvious where they're being executed.
The goal is to create a tunnel that will forward TCP traffic from the loopback address on the remote machine on port 123 to the local machine on port 456. This can be done with the following command, on the local machine:
local:~# ssh -N -R 123:127.0.0.1:456 10.0.3.12
To check that the process is running, we can do:
local:~# ps aux | grep ssh
If you see the command in the output, we can proceed. Otherwise, check that the SSH key is installed in the remote. Note that excluding the username before the remote IP, makes ssh use the current username.
Next, we want to check that the tunnel is open on the remote:
remote:~# netstat | grep 10.0.0.1
We should get an output similar to this:
tcp 0 0 10.0.3.12:ssh 10.0.0.1:45988 ESTABLISHED
Would be nice to actually see some data going through from the remote to the host. This is where netcat comes in. On CentOS it can be installed with yum install nc.
First, open a listening port on the local machine:
local:~# nc -l 127.0.0.1:456
Then make a connection on the remote:
remote:~# nc 127.0.0.1 123
If you open a second terminal to the local machine, you can see the connection. Something like this:
local:~# netstat | grep 456
tcp 0 0 localhost.localdom:456 localhost.localdo:33826 ESTABLISHED
tcp 0 0 localhost.localdo:33826 localhost.localdom:456 ESTABLISHED
Better still, go ahead and type something on the remote:
remote:~# nc 127.0.0.1 8888
Hallo?
anyone there?
You should see this being mirrored on the local terminal:
local:~# nc -l 127.0.0.1:456
Hallo?
anyone there?
The tunnel is working! But what if you have an application, called appname, which is supposed to be listening on port 456 on the local machine? Terminate nc on both sides then run your application. You can check that it's listening on the correct port with this:
local:~# netstat -tulpn | grep LISTEN | grep appname
tcp 0 0 127.0.0.1:456 0.0.0.0:* LISTEN 2964/appname
By the way, running the same command on the remote should show sshd listening on port 127.0.0.1:123.
#!/bin/bash
# Check do we have tunnel to example.com server
lsof -i tcp#localhost:6000 > /dev/null
# If exit code wasn't 0 then tunnel doesn't exist.
if [ $? -eq 1 ]
then
echo ' > You missing ssh tunnel. Creating one..'
ssh -L 6000:localhost:5432 example.com
fi
echo ' > DO YOUR STUFF < '
stunnel is a good tool to make semi-permanent connections between hosts.
http://www.stunnel.org/
If you are using ssh in background, use this:
sudo lsof -i -n | egrep '\<ssh\>'