Context:
We have a VPS set up to which we connect through SSH (private keys auth, passwords forbidden, Fail2Ban running).
Recently we've been experiencing issues with the connection when we'd be kicked out (broken pipe) and re-connection attempts would end up with the "Connection closed by remote host" error. After several minutes of "down time" we could connect back as usual. During this time, communication on all other ports is dead as well (AApanel, WebApp ports).
Another thing to mention is that we can verify from VNS that during this down time the server is running and SSH ports are listening as if nothing happened (through lsof).
last command doesn't show any suspicious logins.
The problems persist even after a clean VPS re-install.
Any suggestion what these symptoms might be caused by?
Thanks
Related
I have a ssh server on an old CentOS 5 installation. I can connect to the server without any problems. However, if i disconnect from the server and want to login again after exiting the previous session, the server is not responding and i got a "connection timed out" error. After a while (must be between 1 and 5 minutes) i can login normally. If i then exit the session, the same timeout happens again.
From the network where the client resides, i can connect to other ssh servers without any problems, so i dont think, this is a firewall issue.
Any suggestions, where i can look for the problem?
I tried to login with key instead of password and i stopped the fail2ban service on the ssh server. Both without any success.
I solved my problem:
There is a iptables rule, which is limiting the connections per ip to one attempt per minute. I have whitelisted my ip and now there is no delay when reconnecting.
I have been working on a http server which accepts connections and then based on the host name, loads up the right project from .so, generates the page the client is asking for, then sends them back.
Now that I have several working projects, I am interested in making them available to others but here is my problem :
I am connecting to my dedicated server through ssh, and starting my daemon from there, but after a while, the pages are no longer accessible because my program is no longer running.
I also get kicked by the server after a while. I wonder :
How do I keep my server running ? Does the fact that I keep getting kicked out by ssh after a little idle time explains why my daemon is being shutdown ?
Thanks in advance to whoever will be able to give me some element of answer.
When your SSH session times out SIGHUP was sent to the sub-processes forked from the current interactive shell. That's why the processes were terminated (server no longer running).
To avoid idle SSH connection being kicked by the server, set the ServerAliveInterval to send a request for response from server (e.g. ~/.ssh/config)
Host *
ServerAliveInterval 30
To avoid shell sub-process termination, refer to
https://askubuntu.com/questions/348836/keep-the-running-processes-alive-when-disconneting-the-remote-connection/348921#348921
https://askubuntu.com/questions/349262/run-a-nohup-command-over-ssh-then-disconnect
In short, there are 3 options:
nohup
disown / setsid
start the servers in CLI in tmux or screen session on the server
NOTE: If the server instances are already properly daemonized, try looking at monit or supervisord to keep them running ;-D
I have a google compute engine VM, running ubuntu, and utilising Laravel Forge.
I seem to get blocked by the VM after accessing SSH a few times (2-4), even if I'm logging in correctly. Restarting the VM unblocks me.
I first noticed the issue as I was having trouble logging into SSH, after a few attempts it would become unreachable. My website hosted on it also wouldn't resolve. After restarting the vm, I could try log into ssh again and my website works. This happened a couple time before I figured out how to correctly log in with SSH.
Next, trying to log in to the database with HeidiSQL, which uses plink, I log in fine. But it seems to keep reconnecting via SSH every time I do something, and after 2-4 of these reconnects, I get the same problem with the VM being unreachable by SSH and my website hosted on it being down.
Using SQLyog, which seems to maintain the one SSH connection, rather than constantly reconnecting like HeidiSQL, I have no problems.
When my website is down, I use those "down for everyone or just me" websites to see if it is down, and apparently it's just down for me, so I must be getting blocked.
So I guess my questions are:
1. Is this normal?
2. Can I unblock myself without restarting the VM?
3. Can I make blocking occur in a less strict way?
4. Why does HeidiSQL keep reconnecting via SSH rather than maintaining the one connection like SQLyog seems to?
You have encountered sshguard, which is enabled by default on the GCE Ubuntu images (at least on the 14.10 image, where I encountered it myself). There is a whitelist file at /etc/sshguard/whitelist.
The sshguard default configuration on my VM has a "dangerousness" threshold of 40. Most "attacks" that sshguard detects incur dangerousness of 10, so getting blocked after 4 reconnects sounds about right.
The attack signatures are listed here: http://www.sshguard.net/docs/reference/attack-signatures/
I would bet that you are connecting from an IP that has an invalid reverse DNS configuration (I was). Four connects like that and the default config blocks you for 20 minutes.
I discovered today that if I ssh-forward the local port X to ssh server port Y, and no process is listening on port Y, I can still connect to local port X (I don't get the usual "connection refused" error).
I did my test with 2 different SSH clients on a windows host connecting to a linux server.
After a bit of reflexion, I came to the conclusion that from a pure network point of view, this is the behaviour I should expect: the SSH client is actually listening on localhost:X, so the connection is possible.
Nevertheless, this leads to a problematic situation in which I have an apparently connected socket that talks to nobody. Even sending data on the socket is a successful operation.
So my question: does the SSH protocol manage this situation in some ways, i.e. do I have strategies for detecting this situation? And if yes, may I hope support for this feature on some SSH clients and APIs (today I'm using ssh.net, that does not seem to offer this feature).
If not, how would you proceed for detecting the situation? Timeout on answer?
Thanks for your help,
Alberto.
The only logical behavior would be to close client connection if the server can't connect to the remote side, but that would not be much better than just a hanging connection.
Also there can happen situation when the SSH server is waiting for the remote connection for a minute or two before giving up, so the client's connection will be opened for this period of time anyway.
So there's actually no logical alternative rather than a hanging client connection.
RabbitMQ starts up just fine, but the shovel plugin status is listed as "starting".
I'm using the following rabbitmq.config:
Each broker is running on a separate AWS instance. The remote server is windows 2008 server, the local server is Amazon Linux.
[{rabbitmq_shovel,
[{shovels,
[{scrape_request_shovel,
[{sources, [{broker,"amqp://test_user:test_password#localhost"}]},
{destinations, [{broker, "amqp://test_user:test_password#ec2-###-##-###-###.compute-1.amazonaws.com"}]},
{queue, <<"scp_request">>},
{ack_mode, on_confirm},
{publish_properties, [{delivery_mode, 2}]},
{publish_fields, [{exchange, <<"">>},
{routing_key, <<"scp_request">>}]},
{reconnect_delay, 5}
]}
]
}]
}].
Running the following command:
sudo rabbitmqctl eval 'rabbit_shovel_status:status().'
returns:
[{scrape_request_shovel,starting,{{2012,7,11},{23,38,47}}}]
According to This question, this can result if the users haven't been set up correctly on the two brokers. However, I've double-checked that I've set up the users correctly via rabbitmqctl user_add on both machines -- have even tried it with a different set of users, to be sure.
I also ran an nmap scan of port 5672 on the remote host to verify is was up and running on that port.
UPDATE Problem isn't solved but this does appear to be a result of connection problems with the remote server. I changed "reconnect_delay" to 0 in my config file, to avoid having shovel infinitely re-try the connection. Highly recommend others with this problem do this as well, as it allows you to get error messages out of rabbit_shovel_status. In my case I got the following error:
[{scrape_request_shovel,
{terminated,
{{badmatch,{error,access_refused}},
[{rabbit_shovel_worker,make_conn_and_chan,1},
{rabbit_shovel_worker,handle_cast,2},
{gen_server2,handle_msg,2},
{proc_lib,init_p_do_apply,3}]}},
{{2012,7,12},{0,4,37}}}]
Answering my own question here, in case others encounter this issue. This error (and also a timeout error if you get it, {{badmatch,{error,etimedout}}, ), is almost certainly a communications problem between the two machines, most likely due to port access / firewall settings.
There were a couple of dumb things I was doing here:
1) Was using the wrong DNS for my remote EC2 instance (D'oh! really dumb -- can't tell you how long I spent banging my head against the wall on this one...). Remember that stopping and starting your instance generates a new DNS, if you don't have an elastic IP associated with the instance.
2) My remote instance is a windows server, and I realized you have to open up port 5672 both in windows firewall and in EC2 security groups -- there are two overlapping levels of access controls here, and opening up the port in the EC2 management console isn't sufficient if your machine is windows server on EC2, as you also have to configure the windows server firewall.