We are running a LAMP stack on a Dreamhost dedicated server. Since updating to PHP 5.6 FastCGI we have been getting more frequent server outages and slow response times even with nominal server load. Apache is throwing the following error:
[Tue Jul 14 14:58:09 2015] [error] mod_fcgid: domain=<domain>.com too many /dh/cgi-system/php56.cgi processes (current:20, max:20), skip the spawn request
I have read of some FastCGI configurations that might help, but Dreamhost did not suggest modifying these settings. Would it be safe to update these settings on a Dreamhost server and would this help?
We are planning an audit of all CURL requests to add timeouts. Will this help the issue? Is there something else we can do to stop the php spawn requests from being skipped? Is there a way to identify an offending script that might be hanging the server?
Related
I have found several similar questions:
APACHE, PHP Server return randomly empty response
https://serverfault.com/questions/66662/apache-gives-empty-reply
and others
However these does not seem to help to find the cause. I can replicate the behaviour when reloading a specific page ~20 times.
Running current apache2 (= 2.4.38-3+deb10u4). I tried to disable opcache, remove MaxRequestsPerChild with no effect.
Apache log does not show any error. The request is not even logged.
The USE_ZEND_ALLOC=0 seem to have no effect and the problem persists.
I tried to install mod_forensic which shows that the request came in. No error or finished request is then logged.
The container is running in Kubernetes and I cannot replicate the issue locally running directly with Docker machine, that is why I think this might be caused by some memory setting. However I couldn't find what might be causing this as there is no single error message.
Can you think of any reason why this might be happening?
Edit1:
I tried to set log level to trace:
https://gist.github.com/knyttl/861e8a0fe5651408df37cd5c3874946b
The request is handled and then you can see:
[Tue Oct 20 08:37:55.825454 2020] [core:trace4] [pid 1] mpm_common.c(536): mpm child 388 (gen 2/slot 4) exited
With no error and no response.
Edit2:
I updated to php7.4 and the issue persists.
I finally found it:
The process is silently killed by OOM killer of the host machine:
[4019392.626796] Memory cgroup out of memory: Kill process 4178127 (apache2) score 1137 or sacrifice child
[4019392.636520] Killed process 4178127 (apache2) total-vm:143960kB, anon-rss:22856kB, file-rss:10472kB, shmem-rss:28228kB
This is never logged within the container so it was hard to find.
Why don't you use Jorge's answers ?
Finally solved by adding to /etc/apache2/envvars:
export USE_ZEND_ALLOC=0
https://serverfault.com/a/66759
I have the latest MAMP Pro and it woks great most of the time.
Besides other development activities I do WordPress Core and Plugin updates on 22 websites. I always run the updates on a local (MAMP) version before updating the Live site to verify that there are no conflicts that might break the website. Every couple of months I refresh my local copy of the website.
5 out of the 22 sites error when importing the database. These aren't large sites so SQL file size isn't the issue.
Here's the error message I get from PHPMyAdmin:
Internal Server Error
The server encountered an internal error or misconfiguration and was
unable to complete your request.
Please contact the server administrator, you#example.com and inform
them of the time the error occured, and anything you might have done
that may have caused the error.
More information about this error may be available in the server error
log.
There was only this error in the Apache Log (no errors in MySQL or PHP logs.)
[Sun Feb 26 11:13:47 2017] [error] [client ::1] FastCGI: comm with
server "/Applications/MAMP/fcgi-bin/php5.6.28.fcgi" aborted: idle
timeout (30 sec) [Sun Feb 26 11:13:47 2017] [error] [client ::1]
FastCGI: incomplete headers (0 bytes) received from server
"/Applications/MAMP/fcgi-bin/php5.6.28.fcgi"
I was wondering if this is an issue with MySQL Version mismatch. The live sites are running Version 5.7.15 but MAMP Pro is back on 5.6.34. Other websites with the same version import OK, but these 5 sites do not.
I'm reasonably competent, but no expert when dealing with server issues. Any help would be appreciated.
Thanks,
David R
I have a c# Soap service that I have running on my Linux Suse 12.1 VPS server. This has been working fine without problems until I made a small change to the soap service and copied it onto my VPS. I thought it must have been an issue with my change so I rolled back my changes and still not working. Even some methods that haven't been touched are failing.
However, I have tested on my Dev machine which is on windows and is working fine and have also copied the soap interface on to a linux dev machine which is set up in the exact samme way as my VPS, i.e. OpenSuse 12.1 and has all the same stuff as my VPS web server. Both work absolutely fine, not got any problems what so ever.
On the VPS host, however, mod-mono-server is constantly crashing and even though it starts up, the asmx file cannot be read, just displays server error, and I need to run rcapache2 restart to get the test page to load up.
In the apache error log file I have the following:
[Thu Aug 30 20:10:19 2012] [error] (70014)End of file found: read_data
failed [Thu Aug 30 20:10:19 2012] [error] Command stream corrupted,
last command was 1 [Thu Aug 30 20:08:47 2012] [error] Command stream
corrupted, last command was 7
I have no idea what the problem might be, I've tried rebooting the VPS but no difference.
I am using the ASP.net 4 version of mod-mono-server.
Thanks for any help you can provide.
UPDATE 1
I have just noticed something else in the apache error. The log file contains the following
[Thu Aug 30 20:46:31 2012] [notice] caught SIGTERM, shutting down
[Thu Aug 30 20:46:32 2012] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Aug 30 20:46:32 2012] [error] Not running mod-mono-server.exe because no MonoApplications, MonoApplicationsConfigFile or MonoApplicationConfigDir specified.
UPDATE 2
Have just made a discovery, not entirely sure if it helps. The soap service is working fine on the server as long as it doesn't require accessing a mysql database. If it performs a query, it displays an an internal server error 500 within the test page. but if called from PHP it causes it to crash mono. The database is a mysql database and is a local database. There's 34% RAM Free so I don't believe this is a memory issue. I've also emptied the database table to determine if this fixes it if it its something to do with the amount of data, but this hasn't fixed it either.
Thanks to #knocte suggestion I managed to figure out the problem.
When the Soap Service accesses the database it reads a config.xml to determine what username and password to use to access the database, I'm guessing that this config file managed to get corrupted during the transfer. Although I could read it in vi, but maybe there was something that was wrong with the file that stopped the soap service reading the config.
All I did was delete the file and copy and paste the content into the file manually.
For some reason mod-mono when it couldn't access the database it crashes mono, even though all the MySQL stuff within the soap service has MySQL Exception handling. #knocte suggestion of testing the soap service proved useful as when this was used to access the database from the soap service xsp4 would stop but display the error saying that it didn't have permission to access the database, even though it had the correct username and password in the config file.
Once I had re-created the config file the soap service works correctly again.
Thanks for your help.
I have a problem on multiple servers than use Apache module mod_fcgid to serve a cgi script that processes the request (ticket validation and similar processing) then serves files on the server based on the result of the processing.
I keep getting the following errors repeatedly in the logs:
[Mon Jan 30 23:11:41 2012] [warn] [client 95.35.160.193] mod_fcgid: error reading data, FastCGI server closed connection
[Mon Jan 30 23:11:41 2012] [warn] [client 95.35.160.193] (32)Broken pipe: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[Mon Jan 30 23:13:34 2012] [warn] [client 37.8.52.128] mod_fcgid: can't apply process slot for /var/www/cgi-bin/assetx.fcgi
These problems cause the server to be slow and other times result in service temporarily unavailable error.
The servers have large traffic on them, I have currently configured the following fcgi directives as below:
FcgidMaxRequestsPerProcess 0
FcgidMaxProcesses 300
FcgidMinProcessesPerClass 0
FcgidIdleTimeout 240
FcgidIOTimeout 240
FcgidBusyTimeout 300
the average load on the servers is normal, the number of processes is on average 250 processes.
I have done research for days about this issue, some say it is a permission problem, I've followed their suggestion, didn't help. I tried to tune the parameters above, these are the final values I tried, but they didn't work as well. I am also trying out nginx to be used instead of apache but I cannot find a suitable way to run the cgi script with this high load on the server using nginx.
What can I do to fix this problem?
Your app is dying before Apache can contact it successfully. The answer is to find out why the app is dying.
FastCGI process should never die or quit, even in an error condition. Apache expects FastCGI script to just keep on being there.
You mention you have a cgi script. How did you modify it to support FastCGI?
Usually you need to switch to something like CGI::Fast, remove all calls to die and exit, and refactor your script to run using the CGI::Fast while loop.
Can't get to my site. Apache gives the following error message:
[Fri Sep 05 08:47:42 2008] [error] (OS 10060)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. : proxy: HTTP: attempt to connect to 10.10.10.1:80 (10.10.10.1) failed
Can you connect to the proxied host (10.10.10.1) directly? Is it functioning normally?
http://www.checkupdown.com/status/E503.html
Your Web server is effectively 'closed for repair'. It is still functioning minimally because it can at least respond with a 503 status code, but full service is impossible i.e. your Web site is simply unavailable. There are a myriad possible reasons for this, but generally it is because of some human intervention by the operators of your Web server machine. You can usually expect that someone is working on the problem, and normal service will resume as soon as possible.
You need to restart the webserver then figure out why it shut it self down.