I have setup mod cluster 2.0.0 with JBOSS EAP 7.3 . The nodes are getting registered with cluster manager but i am not able to access webapps through mod_cluster . It throws service unavailable error. When i checked httpd error log t says "manager_handler STATUS error: MEM: Can't read node with "master:server1" JVMRoute". Any help is appreciated . Below is my jboss socket and proxy config
<outbound-socket-binding name="proxy1">
<remote-destination host="127.0.0.1" port="6677"/>
</outbound-socket-binding>
<subsystem xmlns="urn:jboss:domain:modcluster:5.0">
<proxy name="default" advertise-socket="modcluster" proxies="proxy1" listener="ajp">
<dynamic-load-provider>
<load-metric type="cpu"/>
</dynamic-load-provider>
</proxy>
</subsystem>
Related
I am working with wildfly undertow load balancer , and I added following configuration in standalone-load-balancer.xml but my request is not getting redirected?
I added reverse proxy handler and host to the Undertow subsystem as :
<handlers>
<reverse-proxy name="keycloak-handler">
<host name="keycloak-host" outbound-socket-binding="remote-host" path="/auth"/>
</reverse-proxy>
</handlers>
And then I defined outbound socket bindings for remote hosts as :
<outbound-socket-binding name="remote-host">
<remote-destination host="192.168.1.5" port="8443"/>
</outbound-socket-binding>
And then added the reverse proxy to a location as :
<host name="default-host">
<location name="/auth" handler="keycloak-handler"/>
<filter-ref name="load-balancer"/>
</host>
The mistake I was doing here is , my load balancer was with ssl enabled and I was communicating with keycloak via 8443 which expecting load balancer and keycloak both with same certificates. So to get rid off this , I have connected to keycloak via 8080 port and I have enabled ssl only with load balancer . So after changing below code it worked for me:
<outbound-socket-binding name="remote-host">
<remote-destination host="192.168.1.5" port="8080"/>
</outbound-socket-binding>
I am using Wildfly 13 standalone with JMX enabled:
<remoting-connector/>
The management interface is going through https:
<management-interfaces>
<http-interface security-realm="ManagementRealm">
<http-upgrade enabled="true"/>
<socket-binding https="management-https"/>
SSL is set up:
<security-realm name="ManagementRealm">
<server-identities>
<ssl>
<keystore path=.../>
</ssl>
</server-identities>
When I start JConsole and try to connect on
service:jmx:remote+http://localhost:9993
I don't get connection.
When I revert the binding to http I can connect to service:jmx:remote+http://localhost:9990
Is it possible to have JMX with SSL using the remote+http protocol?
Thanks,
--Ivo
It should be service:jmx:remote+https://<server>:9993
Hi I am new to clustering concept, so I tried to establish a Tomcat cluster with 2 instances. I followed the examples, mainly at http://www.mulesoft.com/tcat/tomcat-clustering and few other web sources.
My sample set up is as,
in workers.properties
# Define worker names
worker.list=jkstatus, loadbalancer
# Create virtual workers
worker.jkstatus.type=status
worker.loadbalancer.type=lb
# Declare Tomcat server worker 1
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=7009
# Declare Tomcat server worker 2
worker.worker2.type=ajp13
worker.worker2.port=9009
worker.worker2.host=localhost
# Associate real workers with virtual LoadBalancer worker
worker.loadbalancer.balance_workers=worker1,worker2
And Apache httpd.conf as,
# ADDED CLUSTER CONFIG
# Load module
LoadModule jk_module modules/mod_jk.so
# Specify path to worker configuration file
JkWorkersFile C:/tomcat_clustered/workers.properties
# Configure logging and memory
JkShmFile C:/tomcat_clustered/log/location/mod_jk.shm
JkLogFile C:/tomcat_clustered/log/location/mod_jk.log
JkLogLevel info
# Configure monitoring
JkMount /jkmanager/* jkstatus
<Location /jkmanager>
Order deny,allow
deny from all
allow from localhost
</Location>
# Configure applications
JkMount /* loadbalancer
# END CLUSTER CONFIG
With each tomcat(worker) server.xml as,
<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker1">
<!--For clustering, please take a look at documentation at:
/docs/cluster-howto.html (simple how to)
/docs/config/cluster.html (reference documentation) -->
<!--
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
-->
<!-- CLUSTER BEGIN -->
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<!---->
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="239.0.0.1"
port="45564" frequency="500"
dropTime="3000"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto" port="4000" autoBind="100"
selectorTimeout="5000" maxThreads="6"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
<!-- CLUSTER END -->
<!-- Use the LockOutRealm to prevent attempts to guess user passwords
via a brute-force attack -->
<Realm className="org.apache.catalina.realm.LockOutRealm">
For worker1, worker2. I am working on Windows 7. It looks like some multicast option is needed but I am not sure. On starting up of tomcat, one is starting well, and on starting second(say worker2) both tomcat console's are throwing errors as,
at org.apache.catalina.tribes.group.ChannelInterceptorBase.heartbeat(Cha
nelInterceptorBase.java:103)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.
ava:155)
at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(Gro
pChannel.java:690)
ep 15, 2014 10:11:03 AM org.apache.catalina.tribes.group.interceptors.TcpFailur
Detector memberAlive
EVERE: Unable to perform failure detection check, assuming member down.
ava.net.SocketException: Permission denied: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketI
pl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja
a:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket
mpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java
Not sure if its due to some multicast permissions on my Windows or any config related issues. Please suggest some good pointers in this regard
You need to enable multicast. see this post how to enable multicast is windows, https://serverfault.com/questions/262634/how-do-i-know-if-ip-multicasting-is-enabled-on-my-network-in-windows
In Linux Environment most of the system kernel is capable to process the multicast address.
but we need to add route entry in kernel routing table.
sudo route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
The error that you're reporting looks like it might be OS / networking related. Hard to say exactly. I can tell you this though.
You've declared a TcpFailureDetector.
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
This is part of Tomcat's cluster configuration and it's responsibility is to make a TCP connection to any node that is suspected of failure. If the listener can't connect to the node, it'll be marked as down. If it can connect, then the node remains active.
In this case, the listener is attempting to make a connection to one of your nodes and it's failing with an error from the JDK. Googling that specific error turns up some suggestions, one of which is to use "-Djava.net.preferIPv4Stack=true".
I get java.net.SocketException: Permission denied: connect when sending an email in Jenkins
On a separate note, if you've note seen the official documentation on Tomcat clustering, I would strongly recommend you check that out. Here's the link.
http://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html
I have a clustered environment with Apache 2.2.6 and mod_proxy pointing to Tomcat 7.0.26 through AJP13 with Sticky Sessions.
The httpd.conf configuration is like this:
<Proxy balancer://mycluster2>
BalancerMember ajp://192.168.0.1:8009 route=tomcat1
BalancerMember ajp://192.168.0.2:8009 route=tomcat2
ProxySet lbmethod=byrequests
</Proxy>
ProxyPass /MyApp balancer://myCluster2/MyApp stickysession=JSESSIONID
ProxyPassReverse /MyApp https://apache_server/MyApp
In my tomcat server.xml file, I have properly configured the cluster inside the <Host> tag (posted just the tomcat1 file, tomcat2 is the same changing only the ip):
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="150" minSpareThreads="4"/>
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
...
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false" notifyListenersOnReplication="true" />
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4" port="45564" frequency="500" dropTime="3000" />
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="192.168.0.1" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6" />
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender" />
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector" />
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor" />
<Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor" />
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="" />
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve" />
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener" />
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener" />
</Cluster>
This configuration works like a charm with any jsp webapp, it replicates sessions and works perfectly on failover with the classic failover step test case:
1.- Tomcat1 starts.
2.- Tomcat2 starts.
3.- A request is processed by the balancer: https://apache_server/MyApp and sent to Tomcat1.
4.- Some operations are performed (i. e. refresh page with a counter as session attribute).
5.- Tomcat1 is killed.
6.- User refresh page and the session counter follows counting in Tomcat2.
So, at that point I have clear that there are no misconfigurations neither on apache nor in Tomcat. Then I go with MyApp. First of all, it has the <distributable/> tag in web.xml.
Next, I deploy it on Tomcat1 and Tomcat2 successfully and I see that Tomcat is multicasting and sharing information between nodes for my application:
INFO: Gestor [/MyApp], requiriendo estado de sesión desde org.apache.catalina.tribes.membership.MemberImpl[tcp://{192, 168, 0, 1}:4000,{192, 168, 0, 1},4000, alive=5113068, securePort=-1, UDP Port=-1, id={-31 113 14 29 99 -58 77 -75 -111 66 -103 86 102 -108 120 61 }, payload={}, command={}, domain={}, ]. Esta operación se agotará si no se recibe estado de sesión dentro de 60 segundos.
19-sep-2013 18:49:51 org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor report
INFO: ThroughputInterceptor Report[
Tx Msg:1 messages
Sent:0,00 MB (total)
Sent:0,00 MB (application)
Time:0,00 seconds
Tx Speed:0,12 MB/sec (total)
TxSpeed:0,12 MB/sec (application)
Error Msg:0
Rx Msg:1 messages
Rx Speed:0,00 MB/sec (since 1st msg)
Received:0,00 MB]
19-sep-2013 18:49:51 org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions
INFO: Gestor [/MyApp]; estado de sesión enviado a las 19/09/13 18:49 recibido en 106 ms.
I try to reproduce the before mentioned navigation, and I can see in Tomcat Managers for both nodes my domain objects getting replicated (all of them implement Serializable).
For some reason, my JSF com.sun.faces.renderkit.ServerSideStateHelper.LogicalViewMap does not replicate all the objects. In fact, the Backup node has always one object less than Primary node in that session attribute.
At point 6, after killing Tomcat1 and refreshing page, session is not recovered and user is sent to logout screen invalidating the session.
MyApp worked before in a non-clustered environment. Here is the configuration for STATE-SAVING of MyApp web.xml:
<context-param>
<param-name>javax.faces.STATE_SAVING_METHOD</param-name>
<param-value>server</param-value>
</context-param>
Even I tried to upgrade from JSF 2.1.4 to 2.1.21 and I get the same error. I can't upgrade to 2.2.3 without changing many things in MyApp (this is in production and it's a long time development project, so I must try all before considering the refactorization of the whole project).
I tried to put also this in my web.xml and it is even worse, since it replicates less objects in LogicalViewMap:
<context-param>
<param-name>com.sun.faces.serializeServerState</param-name>
<param-value>true</param-value>
</context-param>
My faces-config.xml has nothing special.
I also tried to upgrade JSF to 2.2.3 but in this case project wouldn't work at all since I'm using Richfaces 4.0.0-final and I would need to refactorizate some more code.
At this time I'm thinking that JSF 2 is not compatible with Tomcat Clustering. Does anybody configured a project with Tomcat Clustering and JSF 2?
Any help will be appreciated.
I had a similar issue but on a weblogic environment. Please see jsf session failover
Hopefully this might assist somehow.
I have the following configuration of Jetty server inside my REST layer of Apache CXF JAX-RS:
<httpj:engine-factory bus="cxf">
<!--
you just need to specify the TLS Server configuration for the certain port
-->
<httpj:engine port="${port}">
<httpj:tlsServerParameters>
<sec:keyManagers keyPassword="${keystorePassword}">
<sec:keyStore type="JKS"
password="${keystorePassword}"
file="${keystoreFile}"/>
</sec:keyManagers>
<sec:trustManagers>
<sec:keyStore type="JKS"
password="${keystorePassword}"
file="${bookshelf.portal.http.keystoreFile}"/>
</sec:trustManagers>
</httpj:tlsServerParameters>
<httpj:threadingParameters minThreads="5"
maxThreads="15" />
<httpj:sessionSupport>true</httpj:sessionSupport>
</httpj:engine>
</httpj:engine-factory>
I would like to configure Jetty to use ThrottlingFilter to throw HTTP 503 error upon a high load on server.
How do I do that?