I have a simple deployment with some webservers connected to an AWS ELB. This ELB in-turn has some application servers behind it.
The webservers are unable to connect to the application server ELB. The httpd error log is full of:
[Thu Dec 22 15:28:05.897273 2016] [proxy:error] [pid 10188] (70007)The timeout specified has expired: AH00957: HTTP: attempt to connect to 54.254.179.37:80 (elblinkhere) failed
[Thu Dec 22 15:28:05.897348 2016] [proxy:error] [pid 10188] AH00959: ap_proxy_connect_backend disabling worker for (elblinkhere) for 60s
[Thu Dec 22 15:28:05.897361 2016] [proxy_http:error] [pid 10188] [client 10.0.0.54:13789] AH01114: HTTP: failed to make connection to backend: elblinkhere
I have tried to check if this is an SELinux issue but that does not seem so.
I have also read a large number of threads on the internet about this and not come across any solutions.
My question(s):
1. What other methods can I use to resolve this?
2. How do I resolve this?
Did you configure your ELB as external and also enabled necessary port for ELB's security group?
Related
After reading this link, it's clear that HTTP2 and prefork don't work together. I am trying to get around this issue by disabling prefork in MPM.
I went and changed the httpd-mpm.conf and commented out the following statements:
<IfModule mpm_prefork_module>
#StartServers 5
#MinSpareServers 5
#MaxSpareServers 10
#MaxRequestWorkers 250
#MaxConnectionsPerChild 0
</IfModule>
and in my httpd.conf file i have enabled the module
LoadModule http2_module modules/mod_http2.so
and added the following configuration:
Protocols h2 h2c http/1.1
H2Direct on
I see no errors in my error log which are as follows:
[Thu Apr 14 23:03:59.295852 2022] [ssl:warn] [pid 18592:tid 220] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Apr 14 23:03:59.297852 2022] [mpm_winnt:notice] [pid 18592:tid 220] AH00455: Apache/2.4.53 (Win64) OpenSSL/1.1.1n configured -- resuming normal operations
[Thu Apr 14 23:03:59.297852 2022] [mpm_winnt:notice] [pid 18592:tid 220] AH00456: Apache Lounge VS16 Server built: Mar 16 2022 11:26:15
[Thu Apr 14 23:03:59.298852 2022] [core:notice] [pid 18592:tid 220] AH00094: Command line: 'httpd.exe -d C:/Users/naraadia/.softwares/Apache24_2.4.53'
[Thu Apr 14 23:03:59.320854 2022] [mpm_winnt:notice] [pid 18592:tid 220] AH00418: Parent: Created child process 14840
[Thu Apr 14 23:04:00.406963 2022] [ssl:warn] [pid 14840:tid 228] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Thu Apr 14 23:04:00.449967 2022] [mpm_winnt:notice] [pid 14840:tid 228] AH00354: Child: Starting 64 worker threads.
and none of my pages being served by the server have HTTP2 or h2 protocol in developer tools. All the pages have http/1.1 protocol. Is there a better way to achieve HTTP2 protocol being pushed? I feel prefork is still not disabled.
I have downloaded the server zip file and not compiled the server.
EDIT 1 : After Reading the mod_http2 documentation, it's clear that winnt mpm strategy is being used.
I tried to check if curl helps in determining if HTTP2 is enabled or not but when I run
curl -I --http2 http://localhost:8083 | findstr HTTP
It returns nothing.
Can anyone help in enabling HTTP2 or help in determining what is missing?
I went and changed the httpd-mpm.conf and commented out the following statements:
Those statments were wrapped in an <IfModule mpm_prefork_module> so would only be used if that module was enabled.
EDIT 1 : After Reading the mod_http2 documentation, it's clear that winnt mpm strategy is being used.
That's correct. Apache on windows uses it's own mpm module, which is compatible with HTTP/2.
and none of my pages being served by the server have HTTP2 or h2 protocol in developer tools. All the pages have http/1.1 protocol. Is there a better way to achieve HTTP2 protocol being pushed? I feel prefork is still not disabled.
Browsers only support HTTP/2 over HTTPS. Looks like, from your curl command, that you are only using unencrypted HTTP, rather than HTTPS. So this will not work, even though you have enabled H2Direct - both client and server need to support this to work and you have only enabled it on the server and cannot enable this on the browser.
I tried to check if curl helps in determining if HTTP2 is enabled or not but when I run
curl -I --http2 http://localhost:8083 | findstr HTTP
It returns nothing.
Using HTTP/2 over HTTP (rather than HTTPS) requires an upgrade header and round trip, though that is going to be deprecated. You should use --http2-prior-knowledge if wanting to use HTTP/2 over HTTP and are sure the server supports it.
I set my document root to a git repository of three.js, and the root directory lists all subfolders, except examples/. It also doesn't list files and folders starting with . (understandable, I guess) and README.md. Trying to access examples directly gives a 503 error.
There are no .htaccess files anywhere. There is no IndexIgnore anywhere. Error.log lists this:
[Sat Jun 02 13:35:08.169886 2018] [proxy:error] [pid 20844:tid 1996] (OS 10061)No connection could be made because the target machine actively refused it. : AH00957: AJP: attempt to connect to 127.0.0.1:8009 (127.0.0.1) failed
[Sat Jun 02 13:35:08.169886 2018] [proxy:error] [pid 20844:tid 1996] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 5s
[Sat Jun 02 13:35:08.169886 2018] [proxy_ajp:error] [pid 20844:tid 1996] [client 127.0.0.1:57788] AH00896: failed to make connection to backend: 127.0.0.1
In your apache conf folder, see if you have a file named /extra/httpd-ajp.conf with a line like this:
ProxyPass /examples ajp://127.0.0.1:8009/examples smax=0 ttl=60 retry=5
If so, comment out or delete that line. Fixed the problem for me.
There is an issue where I have a cpanel server with cachewall (Xvarnish used to be called) and mod_pagspeed installed.
Cachewall/Xvarnish has https support enabled.
The issue is that even though the header of the website is showing that both Cachewall and modpagespeed is running, I'm getting flooded with this in my apache error log:
[Mon Jul 03 20:45:49.060050 2017] [pagespeed:error] [pid 31223:tid 112490802050816] [mod_pagespeed 1.11.33.4-0 #31223] Serf status 120171(APR does not understand this error code) polling for 1 threaded fetches for 0.05 seconds
[Mon Jul 03 20:45:49.060071 2017] [pagespeed:error] [pid 31223:tid 112490802050816] [mod_pagespeed 1.11.33.4-0 #31223] Serf status 120171(APR does not understand this error code) polling for 1 threaded fetches for 0.05 seconds
[Mon Jul 03 20:45:49.910183 2017] [pagespeed:warn] [pid 31225:tid 112490799363840] [mod_pagespeed 1.11.33.4-0 #31225] Fetch timed out: https://www.example.com/wp-includes/js/jquery/jquery.js?ver=1.12.4 (connecting to:139.xx.xx.xx:82) (1) waiting for 50 ms
Not sure whats going on here.
I think you have to understand the cache flow when using both varnish and page_speed.
Please check the link bellow for details:
https://www.sonassi.com/help/troubleshooting/understanding-cache-flow-when-using-pagespeed-and-varnish
Clear all page_speed cache and varnish cache, then restart both varnish and apache.
I have a jboss cluster with 2 nodes (a and b) + 1 apache working as mod_cluster (apache in a separate server)
If one of the nodeA goes down, mod cluster can't connect to another one.
So, if nodeA crashes, I can't access jboss aplication by http://apache_server/myapp, but I can by http://nodeb/myapp and vice-versa
I dig on google almost all i have found say that is related to sessions but I can't fnd whats is wron with my config. (Mod_cluster as configured with this tool Load Balancer Configuration Tool
NodeA Log
15/05/2016 07:45:22,741 ERROR [org.jgroups.protocols.TCP] (http-/nodeA:8080-90) failed sending message to jbossnodeb:jbossnodeb/web (4148 bytes): java.net.SocketException: Socket closed, cause: null
15/05/2016 07:45:22,790 ERROR [org.jgroups.protocols.TCP] (OOB-6464,shared=tcp) failed sending message to jbossnodeb:jbossnodeb/web (4141 bytes): java.net.SocketException: Broken pipe, cause: null
NodeB Log
15/05/2016 07:45:23,126 ERROR [org.jgroups.protocols.TCP] (OOB-4949,shared=tcp) failed sending message to jbossnodea:jbossnodea/web (79 bytes): java.net.SocketException: Broken pipe, cause: null
15/05/2016 07:45:53,457 WARN [org.jgroups.protocols.TCP] (Timer-1,shared=tcp) null: no physical address for jbossnodea:jbossnodea/web, dropping message
Apache mod_cluster server log
[Sun May 15 07:45:04 2016] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (nodeA_IP)
[Sun May 15 07:45:34 2016] [error] (70007)The timeout specified has expired: ajp_cping_cpong: apr_socket_recv failed
[Sun May 15 07:45:38 2016] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
[Sun May 15 07:45:38 2016] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to (null) (nodeA_IP)
[Sun May 15 07:45:44 2016] [error] (70007)The timeout specified has expired: ajp_cping_cpong: apr_socket_recv failed
[Sun May 15 07:45:44 2016] [error] (70007)The timeout specified has expired: proxy: dialog to nodeA_IP:8009 (nodeA_IP) failed
[Sun May 15 07:45:44 2016] [error] ajp_read_header: ajp_ilink_receive failed
[Sun May 15 07:45:44 2016] [error] (70007)The timeout specified has expired: proxy: dialog to nodeA_IP:8009 (nodeA_IP) failed
[Sun May 15 07:45:44 2016] [error] (70007)The timeout specified has expired: proxy: dialog to nodeA_IP:8009 (nodeA_IP) failed
[Sun May 15 07:45:45 2016] [error] ajp_read_header: ajp_ilink_receive failed
[Sun May 15 07:45:45 2016] [error] (70007)The timeout specified has expired: proxy: dialog to (null) (nodeA_IP) failed
[Sun May 15 07:45:45 2016] [error] ajp_read_header: ajp_ilink_receive failed
[Sun May 15 07:45:45 2016] [error] (70007)The timeout specified has expired: proxy: dialog to (null) (nodeA_IP) failed
[Sun May 15 07:45:45 2016] [error] ajp_read_header: ajp_ilink_receive failed
[Sun May 15 07:45:45 2016] [error] proxy: CLUSTER: (balancer://clusterjboss). All workers are in error state
Config apache mod_cluster
AdvertiseGroup 225.0.1.107:23364
KeepAliveTimeout 60
ManagerBalancerName clusterjboss
ServerAdvertise On
AdvertiseFrequency 5
EnableMCPMReceive
CreateBalancers 0
AllowDisplay On
ProxyPass / balancer://clusterjboss/ stickysession=JSESSIONID|jsessionid nofailover=On
Visibility
JBoss worker instances must be able to contact your ```EnableMCPMReceive`` VirtualHost
Your JBoss worker instances report their IP address and AJP port to the Apache HTTP Server
Your Apache HTTP Server must be able to contact them back on those reported addresses
ProxyPass
JGroups, Infinispan, Domains, Clustering
mod_cluster, i.e. modcluster subsystem has nothing to do with the aforementioned whatsoever. The subsystem is completely oblivious to the fact that there is some cluster formed or that you have your instances in a domain -- which is also irrelevant to having your instances in a cluster in the first place. Don't bother with JGroups messages while investigating mod_cluster configuration.
Although, if your JGroups cluster is broken...
Infinispan - i.e. distributed or replicated cache of your web session data in this case, relies on JGroups for forming a cluster and for exchanging messages in this cluster. If your instances cannot for a cluster or fail to exchange messages, you might experience a loss of session data on failover.
For example: Apache HTTP Server mod_cluster balacner decides to send request with JSESSIONID yadayadaXXX.worker-1 to worker-2, because worker-1 is down. Due to a network configuration error, worker-1 and worker-2 has never correctly formed a cluster, so worker-2 does not have the session data of worker-1. The result is a web application with a new session created, i.e. your client lost his context, e.g. shopping cart (popular showcase).
ProxyPass
Don't use it unless you have something specific in mind. The whole point of mod_cluster is that it creates all proxy directives in memory, on the fly dynamically as your worker nodes and their web applications come and go. You start fiddling with additional ProxyPass directives if you want to:
react to special error codes from a special web applciation, e.g. to treat HTTP codes that are supposed to mean an error as valid and vice versa
to serve static content directly from the Apache HTTP Server and not from worker nodes - e.g. pictures...
to load balance some contexts to mod_cluster-aware JBoss worker nodes and some contexts to non-mod_cluster servers, e.g. another Apache HTTP Server running Drupal in PHP...
ManagerBalancerName
It is not clear to me why you would need to change it. If you change the default value, you have to also alter balancer="new_value" in your Jboss modcluster subsystem configuration. What is actually does is that it tells mod_cluster in the Apache HTTP Server to create more separate named ProxyPass Balacners internally. One then could use ProxyPass directives to tweak them separately. Do you need to tweak them? According to the rest of your config I am convinced it is not the case. For example, the session stickiness is configured in JBoss nodes in mod_cluster subsystems - worker ndoes report this to the Apache HTTP Server balancer.
HTH, -K-
Possible changes that need to be done in domain.xml:
1. Under < domain-controller>, add < remote host="< ip-address-of-master-node>" port="< port>" security-realm="ManagementRealm"/>
2. Under < servers>, add < server name="slave-node" group="server-group" auto-start="true">
3. Under mod-cluster subsystem, add < mod-cluster-config advertise-socket="modcluster" proxy-list="< ip-address>:< port-in-mod-cluster-config" connector="ajp">
In mod-cluster configuration:
1. Allow from all
2. ManagerBalancerName server-group (exact name as above)
Also, are you using any virtualization/containers? To deal problems with session replication in such cases, you might need to try out "sticky session".
We are using glassfish 3.1.2 which is loadbalanced by Apache with mod_jk however, when I turn on the ping_mode to C or A it does not work, it just says
[Wed May 09 08:17:05 2012] [15033:140400029296384] [info] ajp_handle_cping_cpong::jk_ajp_common.c (889): awaited reply cpong, not received
[Wed May 09 08:17:05 2012] [15033:140400029296384] [error] ajp_connect_to_endpoint::jk_ajp_common.c (992): (worker2) cping/cpong after connecting to the backend server failed (errno=0)
[Wed May 09 08:17:05 2012] [15033:140400029296384] [error] ajp_send_request::jk_ajp_common.c (1585): (worker2) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=0)
[Wed May 09 08:17:05 2012] [15033:140400029296384] [info] ajp_service::jk_ajp_common.c (2540): (worker2) sending request to tomcat failed (recoverable), because of error during request sending (attempt=19)
[Wed May 09 08:17:06 2012] [15033:140400029296384] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1223): (worker2) can't receive the response header message from tomcat, tomcat (192.168.42.83:8009) has forced a connection close for socket 15
However, if I comment out the following line it works fine, any ideas?
worker.template.ping_mode=A
My entire workers file looks like this.
worker.list=loadbalancer, status
worker.template.type=ajp13
worker.template.port=8009
worker.template.lbfactor=1
worker.template.socket_timeout=10
worker.template.ping_timeout=10000
worker.template.ping_mode=A
worker.template.connection_pool_size=25
worker.template.connection_pool_timeout=300
worker.template.retries=20
worker.worker1.reference=worker.template
worker.worker1.host=node1
worker.worker2.reference=worker.template
worker.worker2.host=node2
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=worker1,worker2
worker.status.type=status