I an using Apache and Tomcat with mod-jk in my project. My web application is in Tomcat instances. I am using Apache jmeter for testing http load on tomcat. But I'm getting the error Scoreboard is full,not at MaxRequestWorkers even with 1000 threads.
[mpm_event:error] [pid 24313:tid 3075319488] AH00485: scoreboard is full, not at MaxRequestWorkers
[mpm_event:error] [pid 24313:tid 3075319488] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
I've configured http-mpm for approximately 5000 threads.
httd-mpm.conf file is:
<IfModule mpm_event_module>
ServerLimit 200
StartServers 3
MinSpareThreads 75
MaxSpareThreads 250
MaxClients 5000
ThreadsPerChild 25
MaxRequestWorkers 5000
MaxConnectionsPerChild 5000
</IfModule>
Can anybody please guide me what can be the problem in this configuration?
It seems that Apache tries to gracefully finish some of the processes and hangs while doing so. It might be a problem of Apache (https://bz.apache.org/bugzilla/show_bug.cgi?id=53555) or a problem of the application itself.
You can easily check it by looking at the server-status page of apache. In order to do that you need to enable the status.conf apache module. Then go to your-server.org/server-status. You will see all the processes and threads currently being used or gracefully finishing ("G"-state). If you have to many G's that don't go away, you found your problem. It would look sth like this:
Too many threads gracefully finishing...
The solution is pretty simple. Just set MaxConnectionsPerChild to 0 or comment it out (mpm_event.conf). What this does is just let the process continue it's work without restarting it. That's a good thing in our case, since restarting the process let to the problem of having to many G's filling up the scoreboard and eventually crashing Apache.
Hope that helped.
I had this same problem. I tried different Apache versions and MPMs.
I seem to get this alot with MPM Worker. Also error does not reoccur using Apache 2.2.2
Are you using cPanel? IF so try /upcp --force and increase StartServers to a higher amount like 50 as that's all I did to get this error away.
This serverfault Q&A suggests that the event mpm may have a bug. The recommended settings for at least partial workaround:
StartServers 3
MinSpareServers 5
MaxSpareServers 10
ServerLimit 250
MaxRequestWorkers 250
MaxConnectionsPerChild 1000
KeepAlive Off
Try EnableMMAP Off
in 00_default_settings.conf
Related
In our production Apache servers i am observing error AH00286: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
My Environment:
I am using Apache Server version: Apache/2.4.18 (Ubuntu).
MPM worker config as bellow.
<IfModule mpm_worker_module>
ServerLimit 25
StartServers 2
ThreadLimit 64
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
ListenBacklog 511
</IfModule>
During this time there are concurrent request observed to the server however the HTTP response code for all requests are 200 for almost all request excluding very few 302.
I am running Bitbucket server behind reverse proxy.
Question1) how to find my ListenBacklog that is 511 is getting filled OR how to identify its not enough?
Question2) How do i reproduce this error ( i have used AB tool in the past but seeking advice to on this)
Question3) When i get this error, will new user requests & existing user request which are in the queue will be rejected?
Question4) I have enabled monitoring of thread usage based on the scoreboard usage (/usr/sbin/apache2ctl fullstatus) numbers, it prints busy worker thread as 150 and get reset to 0 in next Min when error is observed. is Apache reset all the worker thread when it gets an error as mentioned above?
Thanks in advance for attention on this matter.
I have a webserver that should handle about 800 simultaneous connections.
For this I have configured Apache 2.4 in worker mode (on Centos 6) with this values:
ServerLimit 40
StartServers 25
MaxClients 1000
MinSpareThreads 25
MaxSpareThreads 75
MaxRequestWorkers 1000
ThreadsPerChild 25
MaxConnectionsPerChild 0
However, when I check the server-status I can not see more than 400 workers simultaneously.
In error_log file I can see this: "[mpm_worker:error] [pid 2559:tid 140190319810368] AH00286: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting", but increasing this value has no effect. (with stop/start apache after each change in configuration file).
I checked the ulimits and for apache user the value of nproc is 2048.
I mention that Apache was compiled by me and I didn't set explicitly any limit at compilation time.
What could be the cause of this limitation?
So I figured out what was causing the issue. In httpd.conf the settings for worker module were declared before loading the module, so they were ignored by apache. Moving them after "LoadModule" section fixed the problem.
I am trying to configure Apache to handle a larger number of requests on Ubuntu 14.04. I have done so by adding the following to /etc/apache2/httpd.conf
MaxKeepAliveRequests 500
ServerLimit 20
StartServers 2
MaxRequestWorkers 1000
ThreadsPerChild 50
This doesn't seem to be having the desired affect as I am still receiving the error message "server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers settings"
I would like to confirm that I have set the values correctly and that they are not being overridden anywhere else by checking what values Apache actually has for those settings. Is this possible with Apache?
As it turns out, I missed that httpd.conf is not included/sourced in apache2.conf by default :-/
we have a very strong server (32-cores cpu, 96GB ram) and have apache running in prefork mode. our apache2.conf file includes such settings :
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 5
MaxSpareServers 20
ServerLimit 3000
MaxClients 3000
MaxRequestsPerChild 1000
</IfModule>
the problem is, when our website is under heavy load (when apache process count reaches 1000 to be precise) (or when setting StartServers beyond 1000), apache2 freezes and needs to be restarted. Yet there is still plenty of ram, cpu is underused and apache process count is far beyond maxclients.
My question is, what should i do to allow apache to reach the maxclients configured in the conf file ?
please consider we have already played with /etc/security/limits.conf to set max opened files and nprocs to 5000 (ulimit -a showed these values were well taken into account).
No errors are shown in /var/log/apache2/error.log
Your Apache server may have a compiled in hard limit. To change it you need to recompile your webserver. The default is 200000 which should be high enough - but packages from your linux distribution may differ.
I would rather recommend to get of static file serving from your webserver. Put an nginx or lighttp server in front of your apache. Let it serve static content (images, css, javascript, etc.) and forward dynamic request to your apache.
How do I spawn a specific number of processes in Apache when using mod_wsgi with the WSGIDaemonProcess setting?
I have my VirtualHost setup with the following (as a test):
WSGIDaemonProcess webserver user=webserver group=webserver processes=24 threads=8 display-name=%{GROUP} python-path=/var/virtualenv/lib/python2.6/site-packages
While my httpd.conf is setup as follows:
<IfModule prefork.c>
StartServers 8
MinSpareServers 1
MaxSpareServers 8
ServerLimit 24
MaxClients 24
MaxRequestsPerChild 4000
</IfModule>
Note, that I'm running a very constrained 256MB server with PostgreSQL database installed as well.
However, system shows far more than 24 processes for apache (more than 30). I expected that if I set the ServerLimit to the same as processes in WSGIDaemonProcess it would run at the constant 24. However, there seems to be a bunch of spare processes running for unknown reasons?
The ServerLimit directive has got nothing to do with mod_wsgi daemon mode. The 'processes' option to WSGIDaemonProcess is what specifies how many daemon processes mod_wsgi will create. It is a static number and not a dynamic number so just set it to how many you need. For that number of threads per process, there is no point setting it to more that 'processes=3' to start with as you are limited to 24 concurrent requests in the Apache child worker processes which proxy requests to the mod_wsgi daemon processes, so not possible to handle any more requests than that.
In general, if you are running in a memory constrained environment, then you should not be using prefork MPM but worker MPM. Is there a reason you must, such as needing to run PHP code as well? If not, change the MPM used.
How else you could configure things really depends on your code, response times and request throughput, which only you know.