I found my httpd processes were using a lot of resources in 'htop' (while my page view is actually low and my webserver has 4 CPU with 4G RAM), so I tried to find out what was happening.
Then I found there is an IP kept visiting my site via httpd access log.
After I blocked this IP, my web server was back to normal.
However, I found the problem after my server was in heavy load, which already caused a lot of connection rejections earlier. Therefore my questions are:
(1) What kind of visiting is this? From the rate of the visiting, it's definitely not a human. I looked up this IP, it's from Neitherland, no other information.
(2) What if they change the IP to visit my site like this again? Any way to prevent this kind of visiting proactively?
Related
I'm using Digitalocean cloud hosting server and apache2 in Ubuntu 16.04 VPS. I can browse the site from my local PC and check apache access.log to see the page requests. However when using a mobile device, I cannot get a response from the website. I can ping the server IP address from my phone successfully. However, any requests for the domain root do not create any record in the access.log.
I have attempted to uninstall fail2ban as per this threads:
https://www.digitalocean.com/community/questions/how-to-debug-solve-a-err_connection_timed_out-error-when-this-error-happens-on-some-browsers-but-not-in-another
http://installion.co.uk/ubuntu/vivid/universe/f/fail2ban/uninstall/index.html
I have also tried simply serving a phpinfo() page. However, no still no records in access.log when trying to access from mobile devices. The site is has https enabled and is serving perfectly to a PC.
Also, using a browser testing site (https://www.browserstack.com/) I also get connection timed out errors, and no response records in the access.log.
Any suggestions on where to start troubleshooting this? Is this possibly a problem with Digitalocean itself? Is there anything in the LAMP stack that would specifically be blocking some browsers or IP addresses?
It sounds to me like one of two things is happening here:
Your DNS is not set to point to that IP, but you set it in your operating system's host file on your computer.
Your DNS is correct, but other systems are not yet seeing the change you've made.
Try visiting the IP of the server directly from your mobile device. If anything occurs besides timing out, be it a redirect (even if failed) or a page load, you will know that DNS resolution is the issue. Given that you can ping the IP from your phone I would suggest fail2ban is not related, as fail2ban should block ping as well.
If it turns out to be #2 there, it's just a game of waiting. DNS changes can take up to 48 hours to be seen by all systems. In most cases 4-6 hours is common, but 48 hours is still the recognized standard of "it could possibly take this long."
Jarland
MISP (Malware Information Sharing Platform) is built with Apache. Previously I had a publicly facing instance on Digital Ocean. No log in lag time at all.
I've now moved behind an Apache reverse proxy which is managed by our network team. When attempting to reach the site for the first time there's either a long lag time to get to the site, and if you get to the site there's long lag time when first logging in.
This seems to go away after you first log in. However, many folks are getting a 502 error unable to process /GET, or they simply can't get to the site. There's competing arguments as to which Apache is doing this.
I've looked up a bunch of items already, tried tuning the MISP server, and have had zero luck. Caveat.
I have internal MISP instances that all use the same network. These are not publicly facing and do not have this issue. Also, we are using Lets Encrypt for the certs.
Here's the basic diagram:
I run a 2GB RAM Linode (Ubuntu) that hosts a few WordPress websites. Recently my server has been OOMing and crashing and I have been up all night trying to find out what's causing it. I have discovered there I get an enormous influx of traffic (a tiny DoS) that brings the whole thing down.
I have access logs setup across all of the virtual hosts and I am using tcptrack to monitor activity on the server.
The traffic appearing in my access logs does not account for the traffic I am seeing on tcptrack. i.e. there are a dozen i.p. addresses that are constantly opening and closing connections on the server, but are nowhere to be seen in the access logs for each virtual host.
Clearly it's because these i.ps are not hitting the virtual hosts, but I have tried to set up access logs to monitor server-wide traffic so that I can see what requests their making but I'm really struggling.
Can anyone please point me in the right direction, perhaps tcptrack is just too simplified to provide any meaningful insight?
Start using mod_security
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#Installation_for_Apache
Debian has it which means Ubuntu likely does as well. You should also make sure the kernel is setup properly, search google for SYN_COOKIES. Look into iptables/shorewall etc. Shorewall is a package that wraps iptables. Iptables can be configured for detect floods and start dropping packets.
We have a small office with 20+ computers that are about 80/20 split Macs vs. PCs. I am a web developer by trade who manages our little network but am, by no means, a networking/DNS expert.
That being said, we are having trouble in that every single web site we visit (stackoverflow.com included) that makes a call to a Google API takes forever to load. They all get stuck with a statusbar message such as: "Connecting to fonts.googleapis.com, ajax.googleapis.com, developers.google.com etc..." Eventually, the api call times out and the site will then load without it. Sometimes we get a pop-up error "accounts.google.com" failed to respond. In fact, when we finally get Stack Overflow to load this message is at the top of the page: "Stack Overflow requires external JavaScript from another domain, which is blocked or failed to load."
This seems to be only happening on our internal network. For instance, we can connect laptops, phones and tablets to LTE/mobile networks and they load up the same sites fine.
Oddly enough, Google.com, itself, loads fine. As do Gmail and Google Docs.
When I ping 'fonts.googleapis.com' from both inside the network and from our firewall I get "Request timed out" for 'googleapis.l.google.com' [74.125.70.95].
I have tried deleting all Google entries from our DNS server, an old Windows 2003 Small Biz Server, which sometimes results in 'googleapis.l.google.com' getting a different IP address from our ISP which alleviates the issue temporarily. But, it seems eventually this same IP of 74.125.70.95 will get tacked on to the API URL and we're back in the same boat.
I tried changing the DNS server address of our Win2003 SBS server, itself, away from our ISP's address to both OpenDNS and Google's own DNS server but this hasn't helped.
This has been happening for about a month.
Any ideas?
Stumbled on this article:
http://www.sophos.com/en-us/support/knowledgebase/2450/2750/4350/120934.aspx
Essentially it details something I hadn't thought about. My firewall's Country Blocking feature. Even though the particular IP I had trouble with seemed to belong to Google here in the US, it may have been routed through China (or my firewall's IP address tables are outdated) so traffic was being blocked.
I've adjusted FW rules to allow this IP and all is well.
I have an Apache web server which several sites are connecting to.
From most sites it is accessible and serves content properly (sites being remote and connected via Cisco VPNs), but there is one site, where the server will serve an incomplete page when requested for the login page of the application we are running.
It does not matter what this application is I guess since it is working fine on 10 other sites, just not on this one.
I am getting exactly 907 bytes of the page (the last 907 bytes out of 4000 bytes).
Wireshark reports that the server response is not the first packet and that there is a packet missing before the capture started. Needless to say I waited minutes to start the browser after the capture started so there is no way a packet really was lost because of Wireshark still trying to start up.
Any idea where I look to resolve this?
As it works everywhere it seems to indicate something goes wrong on the network. Where would I look for such an awkward behaviour where 3000 bytes of a web server response get swallowed?
There is no indication on the Apache logs that anything special has occured on the failed pages.