Apache/PHP7.3 running in Docker randomly drops connection with empty response - apache

I have found several similar questions:
APACHE, PHP Server return randomly empty response
https://serverfault.com/questions/66662/apache-gives-empty-reply
and others
However these does not seem to help to find the cause. I can replicate the behaviour when reloading a specific page ~20 times.
Running current apache2 (= 2.4.38-3+deb10u4). I tried to disable opcache, remove MaxRequestsPerChild with no effect.
Apache log does not show any error. The request is not even logged.
The USE_ZEND_ALLOC=0 seem to have no effect and the problem persists.
I tried to install mod_forensic which shows that the request came in. No error or finished request is then logged.
The container is running in Kubernetes and I cannot replicate the issue locally running directly with Docker machine, that is why I think this might be caused by some memory setting. However I couldn't find what might be causing this as there is no single error message.
Can you think of any reason why this might be happening?
Edit1:
I tried to set log level to trace:
https://gist.github.com/knyttl/861e8a0fe5651408df37cd5c3874946b
The request is handled and then you can see:
[Tue Oct 20 08:37:55.825454 2020] [core:trace4] [pid 1] mpm_common.c(536): mpm child 388 (gen 2/slot 4) exited
With no error and no response.
Edit2:
I updated to php7.4 and the issue persists.

I finally found it:
The process is silently killed by OOM killer of the host machine:
[4019392.626796] Memory cgroup out of memory: Kill process 4178127 (apache2) score 1137 or sacrifice child
[4019392.636520] Killed process 4178127 (apache2) total-vm:143960kB, anon-rss:22856kB, file-rss:10472kB, shmem-rss:28228kB
This is never logged within the container so it was hard to find.

Why don't you use Jorge's answers ?
Finally solved by adding to /etc/apache2/envvars:
export USE_ZEND_ALLOC=0
https://serverfault.com/a/66759

Related

Apache crashes with Parent: child process exited with status 3221226356

Apache crashes and I get the following error in the apache log:
AH00428: Parent: child process exited with status 3221226356 -- Restarting.
Backtrace:
Count: 2
Exception #: 0XC0000008
Stack:
ntdll!KiRaiseUserExceptionDispatcher+0x3a
KERNELBASE!CloseHandle+0x1b
libapr_1!apr_shm_size_get+0x27d
libapr_1!apr_shm_destroy+0x12
mod_socache_shmcb+0x161c
mod_ssl!ssl_run_proxy_post_handshake+0x8d12
mod_ssl!ssl_run_pre_handshake+0x3d97
libapr_1!apr_pool_clear+0x6e
httpd!OPENSSL_Applink+0xcef
httpd!OPENSSL_Applink+0x1f98
KERNEL32!BaseThreadInitThunk+0x22
ntdll!RtlUserThreadStart+0x34
Seems to have to do with mod_ssl. I do have a http proxy set up in a https virtual host. However, the same error appears in the Apache log even if I comment out the proxy portion of the config file. This error also tends to be followed by a couple of MySQL errors in the Windows application log complaining about a lost connection. I think that might just be because Apache crashed while the connections were open. I am using the latest versions of Apache 2.4.20, PHP 7.0.8, and MySQL 5.7. Any ideas? Thanks!

Apache error - mod_fcgid: too many processes, skip the spawn request- Hangs Server

We are running a LAMP stack on a Dreamhost dedicated server. Since updating to PHP 5.6 FastCGI we have been getting more frequent server outages and slow response times even with nominal server load. Apache is throwing the following error:
[Tue Jul 14 14:58:09 2015] [error] mod_fcgid: domain=<domain>.com too many /dh/cgi-system/php56.cgi processes (current:20, max:20), skip the spawn request
I have read of some FastCGI configurations that might help, but Dreamhost did not suggest modifying these settings. Would it be safe to update these settings on a Dreamhost server and would this help?
We are planning an audit of all CURL requests to add timeouts. Will this help the issue? Is there something else we can do to stop the php spawn requests from being skipped? Is there a way to identify an offending script that might be hanging the server?

ERR_EMPTY_RESPONSE When trying to access phpmyadmin

When accessing http://domain.com/phpmyadmin
I get the following error (this one is in chrome, but it is not working in any other browser either):
No data received
ERR_EMPTY_RESPONSE
When trying to access, apache's error log shows:
apache2: /build/buildd/xcache-1.3.2/stack.c:47: xc_stack_count: Assertion `stack != ((void *)0)' failed.
[Sat Apr 18 23:41:18 2015] [notice] child pid 31027 exit signal Aborted (6)
All worked well a week ago, can't figure out what went wrong.
Rebooted, reinstalled, reconfiugured phpmyadmin, nothing works :(
Working under Ubuntu 10.04,
Any ideas?
Was facing same error. Fixed it by removing a php module: php5-xcache
Received same error in browser.
Apache's error_log shows:
/usr/sbin/httpd: symbol lookup error: /usr/lib64/libnsssysinit.so: undefined symbol: PR_GetEnvSecure
Fixed by restarting Apache via stop and start (apachectl restart is not sufficient).

mod mono server 4 constantly crashes with soap requests

I have a c# Soap service that I have running on my Linux Suse 12.1 VPS server. This has been working fine without problems until I made a small change to the soap service and copied it onto my VPS. I thought it must have been an issue with my change so I rolled back my changes and still not working. Even some methods that haven't been touched are failing.
However, I have tested on my Dev machine which is on windows and is working fine and have also copied the soap interface on to a linux dev machine which is set up in the exact samme way as my VPS, i.e. OpenSuse 12.1 and has all the same stuff as my VPS web server. Both work absolutely fine, not got any problems what so ever.
On the VPS host, however, mod-mono-server is constantly crashing and even though it starts up, the asmx file cannot be read, just displays server error, and I need to run rcapache2 restart to get the test page to load up.
In the apache error log file I have the following:
[Thu Aug 30 20:10:19 2012] [error] (70014)End of file found: read_data
failed [Thu Aug 30 20:10:19 2012] [error] Command stream corrupted,
last command was 1 [Thu Aug 30 20:08:47 2012] [error] Command stream
corrupted, last command was 7
I have no idea what the problem might be, I've tried rebooting the VPS but no difference.
I am using the ASP.net 4 version of mod-mono-server.
Thanks for any help you can provide.
UPDATE 1
I have just noticed something else in the apache error. The log file contains the following
[Thu Aug 30 20:46:31 2012] [notice] caught SIGTERM, shutting down
[Thu Aug 30 20:46:32 2012] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Aug 30 20:46:32 2012] [error] Not running mod-mono-server.exe because no MonoApplications, MonoApplicationsConfigFile or MonoApplicationConfigDir specified.
UPDATE 2
Have just made a discovery, not entirely sure if it helps. The soap service is working fine on the server as long as it doesn't require accessing a mysql database. If it performs a query, it displays an an internal server error 500 within the test page. but if called from PHP it causes it to crash mono. The database is a mysql database and is a local database. There's 34% RAM Free so I don't believe this is a memory issue. I've also emptied the database table to determine if this fixes it if it its something to do with the amount of data, but this hasn't fixed it either.
Thanks to #knocte suggestion I managed to figure out the problem.
When the Soap Service accesses the database it reads a config.xml to determine what username and password to use to access the database, I'm guessing that this config file managed to get corrupted during the transfer. Although I could read it in vi, but maybe there was something that was wrong with the file that stopped the soap service reading the config.
All I did was delete the file and copy and paste the content into the file manually.
For some reason mod-mono when it couldn't access the database it crashes mono, even though all the MySQL stuff within the soap service has MySQL Exception handling. #knocte suggestion of testing the soap service proved useful as when this was used to access the database from the soap service xsp4 would stop but display the error saying that it didn't have permission to access the database, even though it had the correct username and password in the config file.
Once I had re-created the config file the soap service works correctly again.
Thanks for your help.

WebGate Configuration Problem

I have a instance of Oracle Access Manager set up on Server A and a copy of WebGate set up on Apache on Server B which should be working but isn't. When I try to access http(s)://hostname:port/access/oblix/apps/ webgate/bin/webgate.cgi?progid=1 as specified in the directions after everything is set up, I get a web page that says:
No Response from Application Web Server
There was no response from the application web server for the page you requested.
Please notify the site's webmaster and try your request again later.
I get the same error message when accessing other pages which are set up to be protected by WebGate with LocationMatch in Apache's httpd.conf.
Also, I'm not getting any output in webgate/access/oblix/logs/oblog.log even though webgate/access/oblix/conf/oblog_config_wg.xml has the log level set to LOGLEVEL_TRACE.
Does anyone have thoughts on what config files to check that could be causing this problem?
Edit 1: I've found that when I access a page protected by WebGate, the Apache error_log has this:
[Thu Oct 8 09:44:23 2009] [notice] child pid 4970 exit signal Segmentation fault (11)
Edit 2: Also, for some reason, pages which are to be protected by WebGate don't trigger an update to Apache's access_log, even though they do return that error message.
Ah the joy of the webgate install. Here are the usual questions I ask myself:
Does the apache user also own the directory where webgate is installed?
Are you running this on an OS and Apache version verified by Oracle?
Is your httpd.conf correctly setup with the WebGate configs?
Did you put
LD_ASSUME_KERNEL=2.4.19
export LD_ASSUME_KERNEL
in your init.d/httpd file?
Are you clocks in sync between the Access server and the apache host?
Reinstalling Webgate fixed the problem. Not sure what screwed it up.
i got
[2014-03-27T04:41:51.2101+05:30] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [pid: 14931] [tid: 139749613860672] [user: root] [VirtualHost: main] child pid 14952 exit signal Segmentation fault (11)
along with
Directory does not exist for read/write [.../oracle/middleware/Oracle_WT1/log] []
in OHS_HOME/network/log/sqlnet.log
After changing file permission of OHS directory appropriately. This issue got resolved for me.