I am running Apache 2.4.52 on x64 Amazon Linux 2 as a reverse-proxy onto a Java app. The only service running on the server is Apache. The server has 2GB RAM. My httpd.conf contains:
<IfModule mpm_event_module>
StartServers 3
ServerLimit 50
MaxRequestWorkers 1250
ThreadsPerChild 25
ThreadLimit 64
</IfModule>
The Java app backs off onto a 3rd-party REST API which sometimes experiences periods of high latency (> 60 seconds to respond). A few times now during one of these periods, Apache stops accepting new connections and needs to be restarted. There is nothing in the error_log to indicate what the issue is, nor anything in the server syslog. If I look at the server-status page, I can see only 3 filled slots, none of which are accepting new connections:
If I restart Apache, everything starts working again (albeit slowly due to the 3rd party API):
I'm confused why Apache doesn't create more servers/slots to handle the incoming traffic while the backend is responding slowly, nor why restarting makes the problem go away even if the backend is still responding slowly. I would have expected it to stop accepting traffic again after only a few minutes.
Can anyone shed some light on this please?
Related
I am trying to configure mod_evasive for Apache 2.4.6 on CentOS, release 7.5.1804. I got clean install of CentOS, with clean install of Apache without serving any pages or anything (just example index.html saying hello world), and I installed mod_evasive using this tutorial: https://www.digitalocean.com/community/tutorials/how-to-protect-against-dos-and-ddos-with-mod_evasive-for-apache-on-centos-7
Everything works fine, unless i have to run the testing script which should send requests to server and get 403 error because of mod_evasive. Instead, I am getting 400 Bad Request.
I switched firewalld for IPtables, and I have port 80 open, in fact, the example page works ok from browser. Also, SELinux mode is set to permissive.
Any suggestions?
You will most likely have to change the perl script (usually saved in /usr/share/doc/libapache2-mod-evasive/examples/test.pl) to make this work, e.g.
Original line:
print $SOCKET "GET /?$_ HTTP/1.0\n\n";
Re-worked line:
print $SOCKET "GET /?$_ HTTP/1.0\r\nHost: 127.0.0.1\r\n\r\n";
From https://centosfaq.org/centos/apache-mod_evasive-problem-with-testpl/
The issue was not with mod_evasive or its configuration. In my case I had to tweak the configuration of mpm_prefork_module like below to get mod_evasive configuration to work:
StartServers 10
MinSpareServers 10
MaxSpareServers 10
MaxRequestWorkers 80
MaxConnectionsPerChild 0
Basically fix the number of servers to constant by setting StartServers = MinSpareServers = MaxSpareServers = {your_magic_number} and set MaxConnectionsPerChild=0, so that no new server processes are spawned and no re-cycling of connections happen, allowing Child to hold infinitely many concurrent connections.
Hope this saves your day!
After a few days, I found that there was an error in testing script provided with mod_evasive...
I corrected it and found out that installation was ok.
This is something new, Apache is running fine as server on a Windows Server 2012. There is plenty of RAM and CPU and this is the only thing this server does, is an Apache server.
I can restart the service and it will run fine for a little bit then stop serving pages on all sites. I have messed with the worker threads to find a fine line but still having issues.
Currently settings in httpd-mpm.conf file
<IfModule mpm_prefork_module>
StartServers 10
MinSpareServers 10
MaxSpareServers 20
MaxRequestWorkers 250
MaxConnectionsPerChild 3000
</IfModule>
At this point I am not sure what is going on and have tried several suggestions from various websites.
Specs of Server: Windows 2012 R2 VM 2 GB of RAM 1 GB available Running on an SSD storage
This server connects to a MySQL database on the same network as well, there is not latency between the servers.
we have a very strong server (32-cores cpu, 96GB ram) and have apache running in prefork mode. our apache2.conf file includes such settings :
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 5
MaxSpareServers 20
ServerLimit 3000
MaxClients 3000
MaxRequestsPerChild 1000
</IfModule>
the problem is, when our website is under heavy load (when apache process count reaches 1000 to be precise) (or when setting StartServers beyond 1000), apache2 freezes and needs to be restarted. Yet there is still plenty of ram, cpu is underused and apache process count is far beyond maxclients.
My question is, what should i do to allow apache to reach the maxclients configured in the conf file ?
please consider we have already played with /etc/security/limits.conf to set max opened files and nprocs to 5000 (ulimit -a showed these values were well taken into account).
No errors are shown in /var/log/apache2/error.log
Your Apache server may have a compiled in hard limit. To change it you need to recompile your webserver. The default is 200000 which should be high enough - but packages from your linux distribution may differ.
I would rather recommend to get of static file serving from your webserver. Put an nginx or lighttp server in front of your apache. Let it serve static content (images, css, javascript, etc.) and forward dynamic request to your apache.
How do I spawn a specific number of processes in Apache when using mod_wsgi with the WSGIDaemonProcess setting?
I have my VirtualHost setup with the following (as a test):
WSGIDaemonProcess webserver user=webserver group=webserver processes=24 threads=8 display-name=%{GROUP} python-path=/var/virtualenv/lib/python2.6/site-packages
While my httpd.conf is setup as follows:
<IfModule prefork.c>
StartServers 8
MinSpareServers 1
MaxSpareServers 8
ServerLimit 24
MaxClients 24
MaxRequestsPerChild 4000
</IfModule>
Note, that I'm running a very constrained 256MB server with PostgreSQL database installed as well.
However, system shows far more than 24 processes for apache (more than 30). I expected that if I set the ServerLimit to the same as processes in WSGIDaemonProcess it would run at the constant 24. However, there seems to be a bunch of spare processes running for unknown reasons?
The ServerLimit directive has got nothing to do with mod_wsgi daemon mode. The 'processes' option to WSGIDaemonProcess is what specifies how many daemon processes mod_wsgi will create. It is a static number and not a dynamic number so just set it to how many you need. For that number of threads per process, there is no point setting it to more that 'processes=3' to start with as you are limited to 24 concurrent requests in the Apache child worker processes which proxy requests to the mod_wsgi daemon processes, so not possible to handle any more requests than that.
In general, if you are running in a memory constrained environment, then you should not be using prefork MPM but worker MPM. Is there a reason you must, such as needing to run PHP code as well? If not, change the MPM used.
How else you could configure things really depends on your code, response times and request throughput, which only you know.
I am running Apache2/PHP on my CentOS 5.8 VPS server (2GB RAM, 2GHz processor) and I tried to do a basic load test. Since I am using the standard installation of Apache I assume that prefork model is the one being used. Here is the config:
<IfModule prefork.c>
StartServers 20
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
I did a short test with ab:
ab -q -c 100 -n 10000 mysite.com
In the same time I was checking mysite.com/server-status and I've never seen the number of requests currently being processed exceeding 10. How is this possible ?
According to my calculations the number of concurrent request should have been more than 10, ideally 100. Am I missing something here or Apache 2 server-status is reporting wrong ?
Thank you all.
You are correct that you could see more than 10 requests. You could in fact get more 256 concurrent requests being processed.
However, there's not enough information here to say why you didn't see more than 10 connections. Here are some possibilities:
A slow connection to the host being tested could reduce the number of parallel connections.
A slow application behind the URL being tested could reduce the number of parallel connections
Limitations in the /client/ could limit the number of the parallel connections. "ab" should provide some reporting on what level of concurrency it was able to achieve.
You could accidentally be setting "MaxClients" to a lower value elsewhere in your Apache configuration
... or you could have some other Apache configuration problem.
To provide a more specific answer, you could consider posting complete-as-possible copies of your "ab" output and the entire Apache configuration.