What happens under-the-hood before an ssh connection timeouts? - ssh

Let's look at a scenario. Say I have the domain foo.bar.cc and I'm attempting to connect via ssh:
ssh foo.bar.cc
But, in this scenario, foo.bar.cc:22 requires VPN access. So, this DNS entry is not visible to me. Seeing how I'd never be able to connect, the connection eventually TO's (times out).
Before the TO, what is happening under-the-hood while I am attempting to access the server? What is the connection loop process like during the connection attempt, and what system calls are called, and why? Eventually sshd bails out: how does it determine this? Again, which system calls typically come into play, etc.

Related

How do I ssh into a VPS running tailscale?

I've set up tailscale and connected to an exit node on my VPS on vultr.com. Predictably, I was kicked out and couldn't reconnect, as the VPS's public IP address has changed.
I can reboot the VPS and try again. What steps will I need to take? Does my VPS running behind an exit node even have a unique public address (which?), or does it need to be set up for something like port forwarding?
From looking at tailscale documentation, it looks like they came up with their own ssh, why? Why is the standard ssh inadequate for the purpose? I am not the admin of my tailscale network, and the admin is swamped right now. What can I do?
SSH uses TCP as transport and therefore requires the (srcaddr, srcport, dstaddr, dstport) tuple to be constant over the connection's lifetime.
I believe that since tailscale rotates connections dynamically, it is more suitable for use by clients than servers in a traditional client-server model, unless it provides an 'internal' virtual network over the distributed transport (which would kind of defeat the purpose of covering your tracks).
If you want to connect to your VPS over tailscale, you need to use their tools probably because of that. You can still connect directly to your VPS, though, through plain Internet, if it has any address of its own, and is not firewalled away (or similarly, NATed away). Your provider should either show you the address, or even better, provide access to out-of-band (like serial-port) command line access, where you can query the current addresses using commands like ip addr show.
In your Tailscale Admin console you should be able to see the machine's IP. Just use normal ssh and login that way.
So instead of ssh user#8.8.8.8 you'd do ssh user#100.64.0.1. Tailscale's own ssh client is useful if you want to hook deeper into their MagicDNS stuff, but it's not meant to be the only way to ssh into your machine.
If you run into errors, ping the machine you want to connect to (tailscale ping vps-machine-name). That should help you debug any tailscale client connection problems.

SSH client does not timeout when connection to server has been diconnected

I think there is a simple answer to this question, but everything I find online is about preventing SSH client connections from timing out.
In this case, the client has established a connection to the server, and remains connected. Then the connection is disrupted, say the ethernet cable is unplugged, or the router is powered off.
When this happens, the client connection is not dropped.
The ssh client connection is part of a script and the line that performs the ssh login looks like this:
ssh -Nn script#example.com
The .ssh/config contains the following parameters:
Host *
ServerAliveInterval 60
ServerAliveCountMax 2
When these disconnects occur, I'd like the client ssh connection to timeout, and allow the script to attempt reconnect...
Thanks!
I guess I was wrong about this being a simple question, since no one was able to provide an answer.
My further reading and asking led to one reply on the openssh IRC channel, around 2022-06-06. I was advised that the options:
ServerAliveInterval 60
ServerAliveCountMax 2
Often don't disconnect the client as one might expect.
The ssh_config man page:
ServerAliveCountMax
Sets the number of server alive messages (see below) which
may be sent without ssh(1) receiving any messages back from
the server. If this threshold is reached while server alive
messages are being sent, ssh will disconnect from the server,
terminating the session...
The default value is 3. If, for example, ServerAliveInterval
(see below) is set to 15 and ServerAliveCountMax is left at
the default, if the server becomes unresponsive, ssh will
disconnect after approximately 45 seconds.
Seems to pretty conclusively state that disconnecting on lack of server response is the intention of these parameters. However, in practice this doesn't happen in all cases. Maybe the caveat here is: "while server alive messages are being sent"?
If the application calls for a reliable client disconnect when the server becomes unresponsive, the advice was to implement an external method, separate from the ssh client login script, that monitors server responsiveness, and kills the ssh client process on timeout.

NSURLSession - The request timed out

I'm posting data from my app to my server using NSURLSession when a button is pressed. I can successfully send the data to my server and insert into a database, for the first two occasions, but any time after that, the request times out.
I've tried: changing session configuration (connections per host, timeoutInterval etc), session configuration types, changing the way the data is posted.
Has anyone seen this sort of behaviour before and know how I can fix this issue?
Or is it a server issue? I thought my server was down initially. I couldn't connect to it, nor load certain pages. However, it was only down for me. After rebooting my modem, I could connect back to the server. I didn't have any issues connecting to phpMyAdmin.
If the problem was reproducible after a reboot of the router, then I would look into whether Apple's captive portal test servers were down at the time.
Otherwise, my suspicion is that it is a network problem rather than anything specific to your app.
It is quite possible that the pages you were loading successfully were coming from cache.
Because you said that rebooting your modem fixed the problem, that likely means that your modem stopped responding to either DHCP requests or DNS lookups (either for your domain or for one of the captive portal test domains).
It is also possible that you have a packet loss problem, and that it gets worse the longer your router has been up and running. This could cause some requests to complete and others to fail.
Occasionally, I've seen weird behavior vaguely similar to this when ICMP is getting blocked too aggressively.
I've also seen this when a stateful firewall loses its mind and forgets the state.
This can also be caused by keeping HTTP/HTTPS connections alive past the point at which the server gives up and drops the connection, if your firewall is blocking the packet that tells you that the connection was closed by the remote end.
But without a packet trace, there's no way to be certain. To get one:
If your network code is running on OS X, you can just do this with tcpdump on your network interface.
If you are doing this on iOS, you can do this by connecting your computer via wired Ethernet, enabling network sharing over Wi-Fi, pointing tcpdump at the bridge interface, and pointing your iPhone at that Wi-Fi network.
Either way, that will tell you if there are requests going out that never come back, and more importantly, what type of requests they are, and who is responsible for replying to them. Once you have that information, if the source of the problem isn't obvious, post a link to the packet trace and we'll add more suggestions.

Able to RDP into remote server, but not able to ping or telnet

We have a Win Server 2008 box being hosted (dedicated) for us.
I need to connect to one of it's DB's from a server in our LAN.
What started out as a "sure, I'll just throw that together for you real quick" project has turned into a week-long hair-pulling pile of WTF :)
I am able to RDP into that server without fail or issue.
When I tried to connect to the DB, I got a generic "could not connect" error, so I went hunting.
Telnet attemtps and pings time out.
Since then, we have tried endless variations of firewall settings (including wide open), and still ... no go.
In addition to our firewall, the hosting provider also has a firewall layer.
We turned on all logging, and we don't even see any connection attempts at our FW.
We then had the hosting provider turn on all logging, and they don't see any connection attempts either!
Hrmmmph
I'm at a complete loss.
Any suggestions?
BTW, while I'm comfortable enough with all this to explore and make changes, my experience with firewalls and stuff is fairly limited, so don't hesitate to dumb it down ;)
It is hard to give just one answer to this question, because the interim results of the problem analysis lead to different steps that you need to do next. It will more likely be a step by step help with tracing down the problem.
Do not trust any firewall setting (esp. not any that someone else did, and again esp. not if you don't know him), unless you tested it. Firewall settings are tricky and even experienced professionals get them wrong now and then.
In the guide below, I will write <win2008server> in commands where you have to put the name or IP of the windows 2008 server to which you want to connect. On the other side, I will use the expression "office PC" when I mean your workstation PC in the office from where you are trying to connect to the win2008server.
STEP 1: Checking the Endpoints
1.) Can you telnet to the RDP port?
On your office PC, try this on a command prompt:
telnet <win2008server> 3389
This is to make sure that DNS name resulution works for telnet, as well as network hardware and routing. It should, because you can use RDP to establish this connection. However, anything can get in between, like the telnet command being in any way configured nonstandard or being replaced for whatever reason on a company pc (sysadmins have strange ideas at times...).
2.) Can you telnet locally on the win2008server to the database?
When logged in using RDP on the win2008server, open a command prompt on the server and issue the command
telnet <win2008server> <database port>
That means you are trying to connect from the server to itself. This is to make sure the database port is open on the server.
STEP 2: Checking the Firewalls of the Endpoints
If for 1.) and 2.), your answer is yes it works, you have to test if either the remote side can not be reached or your location can not connect to the internet on the port you are testing (database port). You do this by replacing the respective other side with any other host on the internet for which you know it's reachable or can reach other servers. Typically, you google for a port checker ;)
3.) Check if the win2008server can be reached from another location than yours:
3.1.) Check if the RDP port of the win2008server can be reached from a third party location:
Google for port checker and take the first result (e.g. http://www.yougetsignal.com/tools/open-ports/ ). Type in the name or IP address of the win2008server and the RDP port, usually 3389 . Click on "check" and wait for the success or the timeout.
3.2.) Check if the database port of the win2008server can be reached from a third party location:
Do the same as in 3.1.), just with the database port instead of the RDP port.
4.) Check if you can connect to an outside server on the database port:
For this to work, you need to know a server or create one, which is somewhere outside on the internet, and which listens on the database port. You typically do this by keeping your private PC at home run and accessible through RDP or SSH, and there you open a server and configure your private internet router to forward the connection correctly.
Another way to do this test is webspace with SSH access. Many webspace providers nowadays allow for an SSH login (usually any webspace at $4/month and above).
Let's assume you have SSH access to any such third party place. You can use nc (netcat) there to open a server socket on the database port with this command:
nc -l <database port>
If it's your private PC at home, you usually have to also configure your private router and set up a dynamic DNS name for your internet access for the whole story to work out. You do not have this extra work with a webspace based SSH login. However, there you can not test ports below 1024 because you do not have the privileges. Good luck with this ;)
After you got this, try connecting to the port that you opened:
4.1.) From your office PC with
telnet <third party location> <database port>
4.2.) If 4.1.) does not work, also try with the port checker, because you might have gotten something wrong with setting up the server. Look at 3.) for this, and use the <third party location> and <database port> with the port checker (fourth party check).
STEP 3: Blaming ;)
At least one of the things should have failed by now and you can start calling people and letting them know about your tests and the results. You should be able to combine the results logically, but never start with that. Think about how to convey the information. Start out with your findings and then let them have a moment for their own conclusion. It can be difficult to tell someone in another company or department that their firewall isn't configured correctly. They might deny this even in the presence of proof. Be patient. Explain your findings again. Hint at the conclusion. This can be the trickiest part of the whole problem solution.
I have to say that today I had the same problem.
My solution was just to edit secpol.msc and disable all the FW profiles; then, run services.msc and also disable Windows Firewall service.
After this server was pingable for me.

If possible--how to use an irc server (like gmail chat or facebook) to establish ssh connection between two hosts behind firewalls

I have been looking around for a solution that implements this, but google always gives me tutorials on establishing a live chat over an ssh tunnel--not the other way around.
I suspect this can be implemented just using tunnels (if it is possible at all), but I am not sure how.
I am sorry if this has been asked, but after looking through the related questions, but I have not been able to find one that I can be sure will work for my particular needs (i.e. I cannot create an ssh session directly with gmail.com etc.) If I am wrong, please just post a link to the applicable question.
If you can establish connections between peers via your IRC channel, then there is a solution.
Don't try to fiddle with IRC itself, but build a solution on top of it.
Use ssh yourself on top of IRC.
I mean create a SSH/SSL connection to a dummy socket you can use to intercept the data sent by SSH. Transform this data (if necessary) to make it transportable via IRC. And send it to the remote peer via IRC.
On the remote peer, intercept your data, un-transform it before giving it to your ssh/ssl connection listener. And proceed the same way to send response.
If the connection is successful ssh will tell you and your can start pouring your data through this secure 'channel'.
Your data going via IRC will be safe, because ssh is.