How do I monitor bandwidth usage of individual virtual sites on Apache? (Ubuntu 14).
On our IIS server, we use the performance monitor, save to csv file and have MRTG parse the data and display it as graphs.
Can I do this with MRTG? I read of an unsupported module for Apache (mod_monitor??) that some had tried to use but really don't want to go with unsupported software.
The short answer is that you probably cannot do it without a little additional work.
The longer answer is that, while MRTG can graph anything in theory, you have to provide it with a way to obtain the data. The throughput of a network interface is already provided via SNMP, but the network traffic per virtual server is a little harder to come by, and you need to convince Apache to hand this data over in a format you can use.
You are clearly already aware of much of this, since under IIS you used the performance monitor to obtain the data from the perfstats. In fact, with IIS, instead of dumping the stats to a file and parsing it, you can use a plugin like mrtg-nsclient to query the perfstats directly via the Nagios nsclient++ agent. However, you are using Apache...
One way to achieve it would be to run each virtual server on a separate TCP port, and then use iptables logging rules to count the bytes passed. The output of iptables -L can then be parsed by MRTG to get the counters.
If you want to use name virtual hosts, though, only Apache's internals have the relevant data.
I have an MRTG data collection plugin that obtains total traffic counts via the mod_status URL. This allows graphing of the number of active Apache threads, and total traffic. However it is not split by virtual server, so you cannot get the individual statistics. Even with ExtendedStatus on, you only see the activity of the current threads, not counts split by vhost. ExtendedStatus will allow you to see how many threads are active per vhost, but not the total bytes transferred by each vhost.
The output you want appears to exist in mod_watch which will output one line of statistics per vhost on the URL /watch-list. However, this is an older module and may require modification in order for it to compile against Apache 2.4. It is also very hard to come by as the author has apparently tried to bury it. It used to be on github but vanished in 2012.
Try here: https://github.com/pld-linux/apache-mod_watch for the source,
Try here: http://fossies.org/windows/www/httpd-modules-2.4-win64-VC11.zip/index_o.html for the windows binary for Apache 2.4
Related
I run a 2GB RAM Linode (Ubuntu) that hosts a few WordPress websites. Recently my server has been OOMing and crashing and I have been up all night trying to find out what's causing it. I have discovered there I get an enormous influx of traffic (a tiny DoS) that brings the whole thing down.
I have access logs setup across all of the virtual hosts and I am using tcptrack to monitor activity on the server.
The traffic appearing in my access logs does not account for the traffic I am seeing on tcptrack. i.e. there are a dozen i.p. addresses that are constantly opening and closing connections on the server, but are nowhere to be seen in the access logs for each virtual host.
Clearly it's because these i.ps are not hitting the virtual hosts, but I have tried to set up access logs to monitor server-wide traffic so that I can see what requests their making but I'm really struggling.
Can anyone please point me in the right direction, perhaps tcptrack is just too simplified to provide any meaningful insight?
Start using mod_security
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#Installation_for_Apache
Debian has it which means Ubuntu likely does as well. You should also make sure the kernel is setup properly, search google for SYN_COOKIES. Look into iptables/shorewall etc. Shorewall is a package that wraps iptables. Iptables can be configured for detect floods and start dropping packets.
I was wondering if it would be okay to run Tomcat as both the web server and container? On the other hand, it seems that the right way to go about scaling your webapp is to use Apache HTTP listening on port 80 and connecting that to Tomcat listening on another port?
Are both ways acceptable? What is being used nowdays? Whats the prime difference? How do most major websites go about this?
Thanks.
Placing an Apache (or any other webserver) in front of your application server(s) (Tomcat) is a good thing for a number of reasons.
First consideration is about static resources and caching.
Tomcat will probably serve also a lot of static content, or even on dynamic content it will send some caching directives to browsers. However, each browser that hits your tomcat for the first time will cause tomcat to send the static file. Since processing a request is a bit more expensive in Tomcat than it is in Apache (because of Apache being super-optimized and exploiting very low level stuff not always available in Tomcat, because Tomcat extracting much more informations from the request than Apache needs etc...), it may be better for the static files to be server by Apache.
Since however configuring Apache to serve part of the content and Tomcat for the rest or the URL space is a daunting task, it is usually easier to have Tomcat serve everything with the right cache headers, and Apache in front of it capturing the content, serving it to the requiring browser, and caching it so that other browser hitting the same file will get served directly from Apache without even disturbing Tomcat.
Other than static files, also many dynamic stuff may not need to be updated every millisecond. For example, a json loaded by the homepage that tells the user how much stuff is in your database, is an expensive query performed thousands of times that can safely be performed each hour or so without making your users angry. So, tomcat may serve the json with proper one hour caching directive, Apache will cache the json fragment and serve it to any browser requiring it for one hour. There are obviously a ton of other ways to implement it (a caching filter, a JPA cache that caches the query etc...), but sending proper cache headers and using Apache as a reverse proxy is quite easy, REST compliant and scales well.
Another consideration is load balancing. Apache comes with a nice load balancing module, that can help you scale your application on a number of Tomcat instances, supposed that your application can scale horizontally or run on a cluster.
A third consideration is about ulrs, headers etc.. From time to time you may need to change some urls, or remove or override some headers. For example, before a major update you may want to disable caching on browsers for some hours to avoid browsers keep using stale data (same as lowering the DNS TTL before switching servers), or move the old application on another url space, or rewrite old URLs to new ones when possible. While reconfiguring the servlets inside your web.xml files is possible, and filters can do wonders, if you are using a framework that interprets the URLs you may need to do a lot of work on your sitemap files or similar stuff.
Having Apache or another web server in front of Tomcat may help a lot changing only Apache configuration files with modules like mod_rewrite.
So, I always recommend having Apache httpd in front of Tomcat. The small overhead on connection handling is usually recovered thanks to caching of resources, and the additional configuration works is regained the first time you need to move URLs or handle some headers.
It depends on your network and how you wish to have security set up.
If you have a two-firewall DMZ, with applications deployed inside the second firewall, it makes sense to have an Apache or IIS instance in between the two firewalls to handle security and proxy calls into the app server. If it's acceptable to put the Tomcat instance in the DMZ you're free to do so. The only downside that I see is that you'll have to open a port in the second firewall to access a database inside. That might put the database at risk.
Another consideration is traffic. You don't say anything about traffic, sizing servers, and possible load balancing and clustering. A load balancer in front of a cluster of app servers is more likely to be kept inside the second firewall. The Tomcat instance is capable of handling traffic on its own, but there are always volume limitations depending on the hardware it's deployed on and what the application is doing with each request. It's almost impossible to give a yes or no answer without more detailed, application-specific information.
Search the site for "tomcat without apache" - it's been asked before. I voted to close before finding duplicates.
I'm managing a hosting server and one of my customers will launch a high traffic PHP website. It's a penny auction website and we expect between 25k and 30k visitors per day.
Can you tell me please what should I change in my server configuration (PHP and Apache) to avoid problems? I'm afraid that the server crash with a large number of visitors.
Thank you
Using a lighter web server like nginx as a reverse proxy and a static content server should keep the Apache memory and CPU usage to a minimum which will be a problem on larger sites.
APC as an opcode cache will also be useful in a large site because compiling the PHP scripts to opcode is expensive.
Which Apache forking model are you using for the server? Event and Worker MPM's will probably work better for larger sites with higher concurrent connections.
How is PHP setup within Apache, i.e. FastCGI/CGI/DSO/SuPHP/FPM? SuPHP will be slowest while FastCGI, FPM and DSO will give you much better performance and allow you to use opcode caches.
If you don't need SSL support on the site a free service like https://www.cloudflare.com/ will also lessen the load on your servers.
You could put an opcode cache into use, eAccelerator is a good one for this purpose.
You may also want to consider creating Apache vHosts for static content like images/CSS/javascript to be served from. If these can be put into a CDN, then even better.
There are other tools available for benchmarking, including the Apache benchmarking tool "ab". You can use this to stress-test your site.
There are several areas in which tuning can take place, not just PHP.
I currently have one server with nginx that reverse_proxy to apache (same server) for processing php requests. I'm wondering if I drop apache so I'd run nginx/fastcgi to php if I'd see any sort of performance increases. I'm assuming I would since Apache's pretty bloated up, but at the same time I'm not sure how reliable fastcgi/php is especially in high traffic situations.
My sites gets around 200,000 unique visitors a month, with around 6,000,000 page crawls from the search engines monthly. This number is steadily increasing so I'm looking at perfomrance options.
My site is very optimized code wise and there isn't any caching (don't want that either), each page has a max of 2 sql queries without any joins on other tables, indexes are perfect as well.
In a year or so I'll be rewriting everything to use ClearSilver for the templates, and then probably use python or else c++ for extreme performance.
I suppose I'm more or less looking for any advice from anyone who is familiar with nginx/fastcgi and if willing to provide some benchmarks. My sites are one server with 1 quad core xeon, 8gb ram, 150gb velociraptor drive.
nginx will definitely work faster than Apache. I can't tell about fastcgi since I never used it with nginx but this solution seems to make more sense on several servers (one for static contents and one for fastcgi/PHP).
If you are really targeting performance -and even consider C/C++- then you should give a try to G-WAN, an all-in-one server which provides (very fast) C scripts.
Not only G-WAN has a ridiculously small memory footprint (120 KB) but it scales like nothing else. There's work ahead of you if you migrate from PHP, but you can start with the performance-critical tasks and migrate progressively.
We have made the jump and cannot consider to go back to Apache!
Here is a chart showing the respective performances of nginx, apache and g-wan:
g-wan.com/imgs/gwan-lighttpd-nginx-cherokee.png
apache does not seem to lead the pack (and that's a -Quad XEON # 3GHz).
Here is an independent benchmark for g-wan vs nginx, varnish and others http://nbonvin.wordpress.com/2011/03/14/apache-vs-nginx-vs-varnish-vs-gwan/
g-wan handles much more requests per second with much less CPU time.
NGINX is the best choice as a webserver now a days.
The main difference between Apache and NGINX lies in their design
architecture. Apache uses a process-driven approach and creates a
new thread for each request. Whereas NGINX uses an event-driven
architecture to handle multiple requests within one thread.
As far as Static content is concerned, Nginx overpasses Apache.
Both are great at processing Dynamic content.
Apache runs on all operating systems such as UNIX, Linux or BSD and
has full support for Microsoft Windows & NGINX also runs on several
modern Unix-like systems and has support for Windows, but its
performance on Windows is not as stable as that on UNIX platforms.
Apache allows additional configuration on a per-directory basis via
.htaccess files. Where Nginx doesn’t allow additional configuration.
Request Interpretation-Apache pass file System location. Nginx
Passes URI to interpret requests.
Apache have 60 official dynamically loadable modules that can be
turned On/Off.Nginx have 3rd Party core modules (not dynamically
loadable).NGINX provides all of the core features of a web server,
without sacrificing the lightweight and high-performance qualities
that have made it successful.
Apache Supports customization of web server through dynamic modules.
Nginx is not flexible enough to support dynamic modules and loading.
Apache makes sure that all the website that runs on its server are
safe from any harm and hackers. Apache offers configuration tips for
DDoS attack handling, as well as the mod_evasive module for
responding to HTTP DoS, DDoS, or brute force attacks.
When Choose Apache over NGINX?
When needs .htaccess files, you can override system-wide settings on
a per-directory basis.
In a shared hosting environment, Apache works better because of its
.htaccess configuration.
In case of functionality limitations – use Apache
When Choose NGINX over Apache?
Fast Static Content Processing
Great for High Traffic Websites
When Use Both of them -Together
User can use Nginx in front of Apache as a server proxy.
Which one of these two is most commonly used scenario? I want to use the same scenario in my learning process. thanks.
Don't know about the rest of the industry, but where I work we have Apache HTTPD front-ending for Tomcat.
Any static content is directly provided by HTTPD for performance. Pain in the neck to separate every app out, but there is a noticeable payoff.
Also, HTTPD has some nice code for cookie handling, URL rewriting, clustering and so on.
Only if we determine that there's dynamic, database-bound data to show do we forward to Tomcat, which does an admirable job there.
Has been working well for us for almost a decade. Others too, I would wager.