I am getting the following error intermittently on my server:
**Proxy Error**
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.
Reason: Error reading from remote server
The error logs show the following:
[Sun Feb 06 03:06:00 2011] [error] [client 82.43.154.57] proxy: Error reading from remote server returned by /login, referer: https://demo.XXXXX.us/
[Sun Feb 06 03:06:30 2011] [error] [client 82.43.154.57] (70007)The timeout specified has expired: proxy: error reading status line from remote server XXXXX.us
[Sun Feb 06 03:06:30 2011] [error] [client 82.43.154.57] proxy: Error reading from remote server returned by /
[Sun Feb 06 03:13:31 2011] [error] [client 82.43.154.57] (70007)The timeout specified has expired: proxy: error reading status line from remote server XXXXX.us
[Sun Feb 06 03:13:31 2011] [error] [client 82.43.154.57] proxy: Error reading from remote server returned by /
I have read a lot of posts suggesting connection timeout settings in tomcat and environment settings in Apache. I have set the following in httpd.conf:
<VirtualHost *>
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
</VirtualHost>
I have also set the following in tomcat server.xml:
<Connector port="9080" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" redirectPort="9443" acceptCount="100"
connectionTimeout="60000" disableUploadTimeout="true" />
Also, once the error occurs, I have to start a new browser for the error to disappear as it continues to show even on a refresh. Secondly, I am using htaccess to rewrite the url. Don't know if this has any impact on the error?
EDIT>
My server is running on 150mb of free memory at normal times and can drop quite low but not at the exact times of the above error. Would this cause such an error?
I would appreciate any ideas people have.
Thank you.
This was an issue with Pear Mailer.
We were using Pear Mailer which uses a queue to stack emails ready for sending with a cron job. There was an error in the Pear script which was being called on every action on our site (making posts, sending messages etc..). Pear was crashing which in turn crashed the browser resulting in the above errors.
Disabling Pear resolved the problem, and tweaking the code got it working again.
It took so long to find the issue as we never thought Pear Mailer could cause such a response.
we had a similar problem on our server after a mysql crash, and the only solution was to restart the server.
Related
I have a jboss cluster with 2 nodes (a and b) + 1 apache working as mod_cluster (apache in a separate server)
If one of the nodeA goes down, mod cluster can't connect to another one.
So, if nodeA crashes, I can't access jboss aplication by http://apache_server/myapp, but I can by http://nodeb/myapp and vice-versa
I dig on google almost all i have found say that is related to sessions but I can't fnd whats is wron with my config. (Mod_cluster as configured with this tool Load Balancer Configuration Tool
NodeA Log
15/05/2016 07:45:22,741 ERROR [org.jgroups.protocols.TCP] (http-/nodeA:8080-90) failed sending message to jbossnodeb:jbossnodeb/web (4148 bytes): java.net.SocketException: Socket closed, cause: null
15/05/2016 07:45:22,790 ERROR [org.jgroups.protocols.TCP] (OOB-6464,shared=tcp) failed sending message to jbossnodeb:jbossnodeb/web (4141 bytes): java.net.SocketException: Broken pipe, cause: null
NodeB Log
15/05/2016 07:45:23,126 ERROR [org.jgroups.protocols.TCP] (OOB-4949,shared=tcp) failed sending message to jbossnodea:jbossnodea/web (79 bytes): java.net.SocketException: Broken pipe, cause: null
15/05/2016 07:45:53,457 WARN [org.jgroups.protocols.TCP] (Timer-1,shared=tcp) null: no physical address for jbossnodea:jbossnodea/web, dropping message
Apache mod_cluster server log
[Sun May 15 07:45:04 2016] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (nodeA_IP)
[Sun May 15 07:45:34 2016] [error] (70007)The timeout specified has expired: ajp_cping_cpong: apr_socket_recv failed
[Sun May 15 07:45:38 2016] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
[Sun May 15 07:45:38 2016] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to (null) (nodeA_IP)
[Sun May 15 07:45:44 2016] [error] (70007)The timeout specified has expired: ajp_cping_cpong: apr_socket_recv failed
[Sun May 15 07:45:44 2016] [error] (70007)The timeout specified has expired: proxy: dialog to nodeA_IP:8009 (nodeA_IP) failed
[Sun May 15 07:45:44 2016] [error] ajp_read_header: ajp_ilink_receive failed
[Sun May 15 07:45:44 2016] [error] (70007)The timeout specified has expired: proxy: dialog to nodeA_IP:8009 (nodeA_IP) failed
[Sun May 15 07:45:44 2016] [error] (70007)The timeout specified has expired: proxy: dialog to nodeA_IP:8009 (nodeA_IP) failed
[Sun May 15 07:45:45 2016] [error] ajp_read_header: ajp_ilink_receive failed
[Sun May 15 07:45:45 2016] [error] (70007)The timeout specified has expired: proxy: dialog to (null) (nodeA_IP) failed
[Sun May 15 07:45:45 2016] [error] ajp_read_header: ajp_ilink_receive failed
[Sun May 15 07:45:45 2016] [error] (70007)The timeout specified has expired: proxy: dialog to (null) (nodeA_IP) failed
[Sun May 15 07:45:45 2016] [error] ajp_read_header: ajp_ilink_receive failed
[Sun May 15 07:45:45 2016] [error] proxy: CLUSTER: (balancer://clusterjboss). All workers are in error state
Config apache mod_cluster
AdvertiseGroup 225.0.1.107:23364
KeepAliveTimeout 60
ManagerBalancerName clusterjboss
ServerAdvertise On
AdvertiseFrequency 5
EnableMCPMReceive
CreateBalancers 0
AllowDisplay On
ProxyPass / balancer://clusterjboss/ stickysession=JSESSIONID|jsessionid nofailover=On
Visibility
JBoss worker instances must be able to contact your ```EnableMCPMReceive`` VirtualHost
Your JBoss worker instances report their IP address and AJP port to the Apache HTTP Server
Your Apache HTTP Server must be able to contact them back on those reported addresses
ProxyPass
JGroups, Infinispan, Domains, Clustering
mod_cluster, i.e. modcluster subsystem has nothing to do with the aforementioned whatsoever. The subsystem is completely oblivious to the fact that there is some cluster formed or that you have your instances in a domain -- which is also irrelevant to having your instances in a cluster in the first place. Don't bother with JGroups messages while investigating mod_cluster configuration.
Although, if your JGroups cluster is broken...
Infinispan - i.e. distributed or replicated cache of your web session data in this case, relies on JGroups for forming a cluster and for exchanging messages in this cluster. If your instances cannot for a cluster or fail to exchange messages, you might experience a loss of session data on failover.
For example: Apache HTTP Server mod_cluster balacner decides to send request with JSESSIONID yadayadaXXX.worker-1 to worker-2, because worker-1 is down. Due to a network configuration error, worker-1 and worker-2 has never correctly formed a cluster, so worker-2 does not have the session data of worker-1. The result is a web application with a new session created, i.e. your client lost his context, e.g. shopping cart (popular showcase).
ProxyPass
Don't use it unless you have something specific in mind. The whole point of mod_cluster is that it creates all proxy directives in memory, on the fly dynamically as your worker nodes and their web applications come and go. You start fiddling with additional ProxyPass directives if you want to:
react to special error codes from a special web applciation, e.g. to treat HTTP codes that are supposed to mean an error as valid and vice versa
to serve static content directly from the Apache HTTP Server and not from worker nodes - e.g. pictures...
to load balance some contexts to mod_cluster-aware JBoss worker nodes and some contexts to non-mod_cluster servers, e.g. another Apache HTTP Server running Drupal in PHP...
ManagerBalancerName
It is not clear to me why you would need to change it. If you change the default value, you have to also alter balancer="new_value" in your Jboss modcluster subsystem configuration. What is actually does is that it tells mod_cluster in the Apache HTTP Server to create more separate named ProxyPass Balacners internally. One then could use ProxyPass directives to tweak them separately. Do you need to tweak them? According to the rest of your config I am convinced it is not the case. For example, the session stickiness is configured in JBoss nodes in mod_cluster subsystems - worker ndoes report this to the Apache HTTP Server balancer.
HTH, -K-
Possible changes that need to be done in domain.xml:
1. Under < domain-controller>, add < remote host="< ip-address-of-master-node>" port="< port>" security-realm="ManagementRealm"/>
2. Under < servers>, add < server name="slave-node" group="server-group" auto-start="true">
3. Under mod-cluster subsystem, add < mod-cluster-config advertise-socket="modcluster" proxy-list="< ip-address>:< port-in-mod-cluster-config" connector="ajp">
In mod-cluster configuration:
1. Allow from all
2. ManagerBalancerName server-group (exact name as above)
Also, are you using any virtualization/containers? To deal problems with session replication in such cases, you might need to try out "sticky session".
I have a wierd 504 error occuring in Apache + mod_wsgi when I post a specific SVG string. The backend application is Python Flask, but it does not seem to get to the application at all.
Here is one request that returns a 504 error:
curl 'https://some.domain.com/api/export/png' --data 'svg=%22%3E%28style'
And here is another one that returns with 200:
curl 'https://some.domain.com/api/export/png' --data 'svg=%22%3E%28styl'
The difrrerence is that I only removed one character from the POST data
The SVG data is cut down to the smallest one that could produce an error. The original SVG string is much larger.
The error that appears in the apache error log is:
[Tue Jun 09 14:27:26 2015] [error] [client X.X.X.X] (104)Connection reset by peer: mod_wsgi (pid=19254): Unable to get bucket brigade for request., referer: https://some.domain.com/
[Tue Jun 09 14:27:26 2015] [error] [client X.X.X.X] mod_wsgi (pid=19252): Exception occurred processing WSGI script '/var/www/ade/src/interface.wsgi'.
[Tue Jun 09 14:27:26 2015] [error] [client X.X.X.X] IOError: failed to write data
It is interestig that the the error appears in the error log imidietly after I make the request, before the post data upload finishes.
Here are the version numbers we have:
Ubunut 12.04
Apache/2.2.22 (Ubuntu)
mod_ssl/2.2.22
OpenSSL/1.0.1
mod_wsgi/3.3
Python/2.7.3
Any clues what might be causing this?
UPDATE: We updated mod_wsgi to 4.4.12 and the error message changed in the error.log
[Wed Jun 10 05:48:35 2015] [error] [client X.X.X.X] mod_wsgi (pid=5019): Request data read error when proxying data to daemon process: Connection reset by peer., referer: http://devade.annalect.com/
This might be related to AWS load balancers in front of the web server. Will try to investigate on that end.
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/ts-elb-error-message.html#ts-elb-errorcodes-http504
Hy,
I have succesfully installed hhvm on my ubuntu 14.04 server. I can run codes from terminal with the following code.
hhvm test.php
The problem is when I want to access my webpage from browser I get 500 Internal Server Error. In the error log I have found the following lines.
[Fri Apr 24 14:26:37.377998 2015] [fcgid:warn] [pid 40696] (104)Connection reset by peer: [client 192.168.65.221:50382] mod_fcgid: error reading data from FastCGI server
[Fri Apr 24 14:26:37.378029 2015] [core:error] [pid 40696] [client 192.168.65.221:50382] End of script output before headers: index.php
I am using Apache web server with ISPConfig 3 to handle domains and other stuff.
Does anyone know how to handle this situation?
Thanks in advance, David.
Since a few days, my Apache 2 error log is showing a lot messages like the following (IP addresses and URIs redacted):
[Thu Dec 12 13:46:42 2013] [error] [client 111.222.333.444] 0
[Thu Dec 12 13:52:27 2013] [error] [client 222.333.444.555] 0, referer: http://www.mydomain.com/
[Thu Dec 12 13:52:27 2013] [error] [client 222.333.444.555] 0, referer: http://www.mydomain.com/
[Thu Dec 12 13:53:54 2013] [error] [client 333.444.555.666] 0, referer: http://www.mydomain.com/subdirectory/
[Thu Dec 12 13:46:42 2013] [error] [client 444.555.666.777] 0
[Thu Dec 12 13:54:07 2013] [error] [client aaaa:1111:2222:ffff::] 0, referer: http://www.otherdomain.com/subdirectory/
What is this 0? There are no other messages shown (besides sometimes some other, normal messages, but very rarely).
The IP addresses are both IPv4 and IPv6. I checked the access log for the same date/time and IP addresses. Most of the times, there was an access for the exact same moment from this IP for different URIs on my webpage. But sometimes, there wasn't an access according to the access log.
It's a shared hosting environment, so I can't access the Apache settings (but I have ssh access to my home directory if this helps). I already googled and searched the Apache documentation, but didn't found anything (it's hard to search for "0"...)
/edit: I also asked the webhoster, they said they don't know what it's causing. I cross checked it with the Apache access log, these are requests to PHP scripts (mostly Joomla), but also requests to images as well as JS and CSS files. So I assume it's not a PHP script which is causing this.
If your error_log directive is unset errors will be written in your Apache log file for current VirtualHost.
So double check your PHP configuration (php.ini) or write a simple page with phpinfo()
If this is true, you should look inside your code (may be even into index.php).
Pay attention to this: usually there are two separate php.ini files for Apache /etc/php5/apache2/php.ini and CLI configuration /etc/php5/cli/php.ini.
Please also consider that, if you want change your PHP configuration, you can use ini_set function.
ini_set('error_log', '/var/log/php/error_new.log');
Remember: the destination directory must exist and your web server (or php engine) must have all permission to write into.
error_log format is not customizable, I suspect that it can be set to some higher level: debug or trace, where it can produce additional information.
Also please take into account, that error_log contains debug info from CGI/PHP/Perl scripts, so that 'zero' can be produced by some script that executed through apache as its module.
I'm getting this message in my Apache error console:
[Thu Jun 27 11:56:07 2013] [alert] [client 127.0.0.1] /Users/.......htacces without matching section
It started after my MAMP server wouldnt start and i found some online info that suggested entering ps aux/ grep mysql lsof _-i Killall -9 mysqld when this failed, I tried 2nd suggestion sudo Killall -9 mysqld. This came up with a warning so i called it. I then recalled that i had the same problem before and simple quit mysqld in my activity monitor, and the MAMP server worked again. I tried that, the server worked but got I then got a error message saying:
500 internal server error The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, you#example.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
I checked Apache error console and got this:
[Thu Jun 27 11:56:07 2013] [alert] [client 127.0.0.1] /Users/.......htacces without matching section