I'm running tomcat 7 with apache 2.2 & mod_jk 1.2.26 on a debian-lenny x64 server with 2GB of RAM, 8GB of swap & an Intel Xeon CPU X3330. I've a GWT application which uses connection pooling to interact with backend mysql database. this application have to process many request, but each request takes a little time to process (assume 1 second) & then it must free's the allocated resources for using by the other requests. The number of concurrent clients may increase more than 200 at the pick time, but usually is less than 100.
Apache is configured in prefork-mpm mode with this configurations:
<IfModule mpm_prefork_module>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
MaxClients 256
MaxRequestsPerChild 2000
</IfModule>
& this is my worker.properties file:
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
worker.worker1.lbfactor=1
worker.worker1.connect_timeout=6000
worker.worker1.prepost_timeout=6000
worker.worker1.connection_pool_size=256
worker.worker1.connection_pool_minsize=64
worker.worker1.connection_pool_timeout=60
worker.worker1.reply_timeout=1000
worker.worker1.retries=5
worker.worker1.fail_on_status=-500
worker.worker1.cachesize=64
worker.worker1.cache_timeout=60
& this is configuration of tomcat connector in server.xml:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" connectionTimeout="60000" maxThreads="256" acceptorThreadCount="2" />
Now, the problem is that some of my application requests fail with a 504 gateway time-out error. I've changed & tested many settings is all above configuration files, but I could not gain a stable state. In addition, after some hours of running tomcat, many of running application on the server crashes with an error related to fork new processes & leak of system resources (I'm running on a VPS but I'm sure that I have enough resources & even more than I need to run such application & so I'm sure the problem is just because of bad configurations of apache, tomcat & mod_jk).
Any idea? please suggest an ideal configuration for mpm_prefork, mod_jk worker properties & tomcat connector settings with care to this spacial situation (I mean exactly we have an ajax application which process many short-time requests of many users).
Thanks
Related
I am running Apache 2.4.52 on x64 Amazon Linux 2 as a reverse-proxy onto a Java app. The only service running on the server is Apache. The server has 2GB RAM. My httpd.conf contains:
<IfModule mpm_event_module>
StartServers 3
ServerLimit 50
MaxRequestWorkers 1250
ThreadsPerChild 25
ThreadLimit 64
</IfModule>
The Java app backs off onto a 3rd-party REST API which sometimes experiences periods of high latency (> 60 seconds to respond). A few times now during one of these periods, Apache stops accepting new connections and needs to be restarted. There is nothing in the error_log to indicate what the issue is, nor anything in the server syslog. If I look at the server-status page, I can see only 3 filled slots, none of which are accepting new connections:
If I restart Apache, everything starts working again (albeit slowly due to the 3rd party API):
I'm confused why Apache doesn't create more servers/slots to handle the incoming traffic while the backend is responding slowly, nor why restarting makes the problem go away even if the backend is still responding slowly. I would have expected it to stop accepting traffic again after only a few minutes.
Can anyone shed some light on this please?
This is something new, Apache is running fine as server on a Windows Server 2012. There is plenty of RAM and CPU and this is the only thing this server does, is an Apache server.
I can restart the service and it will run fine for a little bit then stop serving pages on all sites. I have messed with the worker threads to find a fine line but still having issues.
Currently settings in httpd-mpm.conf file
<IfModule mpm_prefork_module>
StartServers 10
MinSpareServers 10
MaxSpareServers 20
MaxRequestWorkers 250
MaxConnectionsPerChild 3000
</IfModule>
At this point I am not sure what is going on and have tried several suggestions from various websites.
Specs of Server: Windows 2012 R2 VM 2 GB of RAM 1 GB available Running on an SSD storage
This server connects to a MySQL database on the same network as well, there is not latency between the servers.
I have fronted Tomcat6 with Apache2.
On an Ubuntu instance I have Apache2 running with 8GB RAM, so decided to have following apache2.conf configurations.
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 550
ServerLimit 550
MaxRequestsPerChild 0
</IfModule>
Above configuration was done using below parameters and this blog post(how to configure apache MPM).
Apache Memory Usage (MB): 611.719
Average Proccess Size (MB): 8.26647
On another instance I have a Tomcat6 running with 8GB RAM. In the Tomcat6 server.xml following configuration is used.
<Connector port="8009" protocol="AJP/1.3" redirectPort="8080" maxThreads="500"/>
My problems are,
What's the process/steps to calculate/decide the number maxThreads parameter in Tomcat6?
How should be memory allocation should be done?
Tomcat6 is a Java application and as such memory allocation is done by the JVM. I suppose that you are willing to proxy tomcat through apache, if so usually 1 apache client will end up as 1 apache thread, so having a lower number of threads in tomcat than MaxClients directive in apache is advisable. This said, to calculate the maxThreads parameter might be difficult, depending on your application each thread may vary its memory usage, having an average might be useful but you have to take also into account other JVM memory spaces, eden, permgen, ...
Take a look at JVM memory settings, per thread stack settings, ...I think this is what you might be looking for.
we have a very strong server (32-cores cpu, 96GB ram) and have apache running in prefork mode. our apache2.conf file includes such settings :
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 5
MaxSpareServers 20
ServerLimit 3000
MaxClients 3000
MaxRequestsPerChild 1000
</IfModule>
the problem is, when our website is under heavy load (when apache process count reaches 1000 to be precise) (or when setting StartServers beyond 1000), apache2 freezes and needs to be restarted. Yet there is still plenty of ram, cpu is underused and apache process count is far beyond maxclients.
My question is, what should i do to allow apache to reach the maxclients configured in the conf file ?
please consider we have already played with /etc/security/limits.conf to set max opened files and nprocs to 5000 (ulimit -a showed these values were well taken into account).
No errors are shown in /var/log/apache2/error.log
Your Apache server may have a compiled in hard limit. To change it you need to recompile your webserver. The default is 200000 which should be high enough - but packages from your linux distribution may differ.
I would rather recommend to get of static file serving from your webserver. Put an nginx or lighttp server in front of your apache. Let it serve static content (images, css, javascript, etc.) and forward dynamic request to your apache.
I am using the following Apache config to forward requests to a Tomcat server:
ProxyPass /myapp ajp://localhost:8009/myapp max=2
This is a simplified config, but is enough to reproduce the issue, which is that the max parameter has no effect. If I through 10 concurrent requests to Apache, all 10 are forwarded to Tomcat at the same time, while I would like to have them forwarded 2 by 2. Should I use something other than the max parameter for this?
The max=2 failed to limit the number of requests concurrently forwarded to Tomcat because I was running this on UNIX, and my Apache came preconfigured with prefork MPM, which creates one process per request. The max applies per process, hence doesn't have the desired effect.
If you are in this situation and need to limit the number concurrent request forwarded to Tomcat, then you'll need to replace your Apache with a worker or event MPM Apache, in the config set ServerLimit to 1, and ThreadsPerChild and MaxClients to the same value, which will be the total number of concurrent connections your Apache will be able to process. You can find more information about this in this section documenting the recommended Apache configuration for Orbeon Forms.
service apache2 restart