I run a 2GB RAM Linode (Ubuntu) that hosts a few WordPress websites. Recently my server has been OOMing and crashing and I have been up all night trying to find out what's causing it. I have discovered there I get an enormous influx of traffic (a tiny DoS) that brings the whole thing down.
I have access logs setup across all of the virtual hosts and I am using tcptrack to monitor activity on the server.
The traffic appearing in my access logs does not account for the traffic I am seeing on tcptrack. i.e. there are a dozen i.p. addresses that are constantly opening and closing connections on the server, but are nowhere to be seen in the access logs for each virtual host.
Clearly it's because these i.ps are not hitting the virtual hosts, but I have tried to set up access logs to monitor server-wide traffic so that I can see what requests their making but I'm really struggling.
Can anyone please point me in the right direction, perhaps tcptrack is just too simplified to provide any meaningful insight?
Start using mod_security
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#Installation_for_Apache
Debian has it which means Ubuntu likely does as well. You should also make sure the kernel is setup properly, search google for SYN_COOKIES. Look into iptables/shorewall etc. Shorewall is a package that wraps iptables. Iptables can be configured for detect floods and start dropping packets.
Related
So I'm pretty new to the server and website dev. Self "taught".
I recently setup a home server running Apache on Ubuntu 20.04(MicroK8s, Linux Server). Postgres database. Nextcloud Cloud server. TomCat and OpenCMS system. And Postfix.
I have a domain name pointing to my address. When I'm home, i.e. physically near my server. And I connect to my subdomain. Cloud.example.com I get nextcloud. When I connect to the 8080 port(www.example.com:8080) I get tomcat and OpenCMS. So far, so good.
When I use a VPN, or am not near my server, and go to the subdomain,I sometimes get one of those random "ad" sites that says "this site may be able to purchase".
After more testing it seems like the number of connected users also changes whether I get the rando site or the intended one.
My server is... Old. Likely slow (4GBram and a Core 2 Duo, it's the fastest old tower I had laying around). So I think it's a timeout error within OpenCMS, that serves a rando site when it can't get nextcloud to respond fast enough. But honestly, I'm not even sure where to start, or what to even ask/say or what you would need to see to even start diagnostic...
When I connect to mydomain.com from the vpn i get a 404. Which makes sense, as I haven't built it yet in OpenCMS.
Any pointers on where to start?
What am I missing?
Do I need to delete my /var/www sites or Virtual Hosts, and let OpenCMS handle all the routing?
I'm confused as to how my server knows to point 8080 to tomcat/OpenCMS, as I never setup a virtual host. How will it eventually know to point mydomain.com to the sites I build in OpenCMS? Or will OpenCMS deploy them to /var/www? Will I need to transfer the netcloud site to the OpenCMS directory?
I know this is a lot of free help to ask for, but I'm doing this mostly for fun and to learn, and don't have anyone who knows. I don't want to pay it out as I'd rather learn it. I'm not even sure where to start asking, but have browsed stack overflow for A LOT of excel, Apache, Linux, and other answers in the past, so thought I would ask here first.
I have an Apache2 server running in Debian 9.
I am using it to host a custom MediaWiki Wiki.
To navigate to the Wiki I use something of this form "10.200.200.20/mediawiki" where the Apache2 server is running on 10.200.200.20.
This works fine however sometimes the IP Address (10.200.200.20) will change and then everyone on the local network navigating to the Wiki will have to be notified and use the new IP Address which is a hassle.
I wish to change it to something consistent, for example "OurWikiServer/mediawiki" it doesn't really matter that much as long as it can always be found at the same place.
I know this is possible as the MediaWiki installation was previously maintained by someone else who used XAMPP in Windows 7 and it was configured to be found at "stringHere/mediawiki" on the local network.
I have tried changing it in /etc/hosts and can get it changing on individual machines as expected, however have no idea how to get it working network wide.
The best way to do this is to define the IP of this station static. This can be done via reservation in DHCP server or assign IP outside of the DHCP IPs. Also consider adding small DNS server to provide hostname instead of IP
I'm using Digitalocean cloud hosting server and apache2 in Ubuntu 16.04 VPS. I can browse the site from my local PC and check apache access.log to see the page requests. However when using a mobile device, I cannot get a response from the website. I can ping the server IP address from my phone successfully. However, any requests for the domain root do not create any record in the access.log.
I have attempted to uninstall fail2ban as per this threads:
https://www.digitalocean.com/community/questions/how-to-debug-solve-a-err_connection_timed_out-error-when-this-error-happens-on-some-browsers-but-not-in-another
http://installion.co.uk/ubuntu/vivid/universe/f/fail2ban/uninstall/index.html
I have also tried simply serving a phpinfo() page. However, no still no records in access.log when trying to access from mobile devices. The site is has https enabled and is serving perfectly to a PC.
Also, using a browser testing site (https://www.browserstack.com/) I also get connection timed out errors, and no response records in the access.log.
Any suggestions on where to start troubleshooting this? Is this possibly a problem with Digitalocean itself? Is there anything in the LAMP stack that would specifically be blocking some browsers or IP addresses?
It sounds to me like one of two things is happening here:
Your DNS is not set to point to that IP, but you set it in your operating system's host file on your computer.
Your DNS is correct, but other systems are not yet seeing the change you've made.
Try visiting the IP of the server directly from your mobile device. If anything occurs besides timing out, be it a redirect (even if failed) or a page load, you will know that DNS resolution is the issue. Given that you can ping the IP from your phone I would suggest fail2ban is not related, as fail2ban should block ping as well.
If it turns out to be #2 there, it's just a game of waiting. DNS changes can take up to 48 hours to be seen by all systems. In most cases 4-6 hours is common, but 48 hours is still the recognized standard of "it could possibly take this long."
Jarland
I have a google compute engine VM, running ubuntu, and utilising Laravel Forge.
I seem to get blocked by the VM after accessing SSH a few times (2-4), even if I'm logging in correctly. Restarting the VM unblocks me.
I first noticed the issue as I was having trouble logging into SSH, after a few attempts it would become unreachable. My website hosted on it also wouldn't resolve. After restarting the vm, I could try log into ssh again and my website works. This happened a couple time before I figured out how to correctly log in with SSH.
Next, trying to log in to the database with HeidiSQL, which uses plink, I log in fine. But it seems to keep reconnecting via SSH every time I do something, and after 2-4 of these reconnects, I get the same problem with the VM being unreachable by SSH and my website hosted on it being down.
Using SQLyog, which seems to maintain the one SSH connection, rather than constantly reconnecting like HeidiSQL, I have no problems.
When my website is down, I use those "down for everyone or just me" websites to see if it is down, and apparently it's just down for me, so I must be getting blocked.
So I guess my questions are:
1. Is this normal?
2. Can I unblock myself without restarting the VM?
3. Can I make blocking occur in a less strict way?
4. Why does HeidiSQL keep reconnecting via SSH rather than maintaining the one connection like SQLyog seems to?
You have encountered sshguard, which is enabled by default on the GCE Ubuntu images (at least on the 14.10 image, where I encountered it myself). There is a whitelist file at /etc/sshguard/whitelist.
The sshguard default configuration on my VM has a "dangerousness" threshold of 40. Most "attacks" that sshguard detects incur dangerousness of 10, so getting blocked after 4 reconnects sounds about right.
The attack signatures are listed here: http://www.sshguard.net/docs/reference/attack-signatures/
I would bet that you are connecting from an IP that has an invalid reverse DNS configuration (I was). Four connects like that and the default config blocks you for 20 minutes.
A long time ago I found some guide that showed how to use the Windows Hosts file to forward traffic to another IP/Port. We setup a clients server on one IP/Port, and they wanted it changed to another IP/Port on the same machine, but are now asking that we forward UDP/TCP traffic to it as well.
I am pretty sure this is doable but I cannot for the life of me figure out how to do it now after Google searching for a while.
You're probably remembering incorrectly. The only thing that /etc/hosts controls is local DNS lookups -- you can't use it to swap ports around or forward data sent from other machines.
You may be able to forward connections using a firewall package or router.