Tomcat Clustering Multicast throws error - Unable to perform failure detection check, assuming member down - apache

Question

Hi I am new to clustering concept, so I tried to establish a Tomcat cluster with 2 instances. I followed the examples, mainly at http://www.mulesoft.com/tcat/tomcat-clustering and few other web sources.
My sample set up is as,
in workers.properties
# Define worker names
worker.list=jkstatus, loadbalancer
# Create virtual workers
worker.jkstatus.type=status
worker.loadbalancer.type=lb
# Declare Tomcat server worker 1
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=7009
# Declare Tomcat server worker 2
worker.worker2.type=ajp13
worker.worker2.port=9009
worker.worker2.host=localhost
# Associate real workers with virtual LoadBalancer worker
worker.loadbalancer.balance_workers=worker1,worker2
And Apache httpd.conf as,
# ADDED CLUSTER CONFIG
# Load module
LoadModule jk_module modules/mod_jk.so
# Specify path to worker configuration file
JkWorkersFile C:/tomcat_clustered/workers.properties
# Configure logging and memory
JkShmFile C:/tomcat_clustered/log/location/mod_jk.shm
JkLogFile C:/tomcat_clustered/log/location/mod_jk.log
JkLogLevel info
# Configure monitoring
JkMount /jkmanager/* jkstatus
<Location /jkmanager>
Order deny,allow
deny from all
allow from localhost
</Location>
# Configure applications
JkMount /* loadbalancer
# END CLUSTER CONFIG
With each tomcat(worker) server.xml as,
<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker1">
<!--For clustering, please take a look at documentation at:
/docs/cluster-howto.html (simple how to)
/docs/config/cluster.html (reference documentation) -->
<!--
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
-->
<!-- CLUSTER BEGIN -->
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<!---->
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="239.0.0.1"
port="45564" frequency="500"
dropTime="3000"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto" port="4000" autoBind="100"
selectorTimeout="5000" maxThreads="6"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
<!-- CLUSTER END -->
<!-- Use the LockOutRealm to prevent attempts to guess user passwords
via a brute-force attack -->
<Realm className="org.apache.catalina.realm.LockOutRealm">
For worker1, worker2. I am working on Windows 7. It looks like some multicast option is needed but I am not sure. On starting up of tomcat, one is starting well, and on starting second(say worker2) both tomcat console's are throwing errors as,
at org.apache.catalina.tribes.group.ChannelInterceptorBase.heartbeat(Cha
nelInterceptorBase.java:103)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.
ava:155)
at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(Gro
pChannel.java:690)
ep 15, 2014 10:11:03 AM org.apache.catalina.tribes.group.interceptors.TcpFailur
Detector memberAlive
EVERE: Unable to perform failure detection check, assuming member down.
ava.net.SocketException: Permission denied: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketI
pl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja
a:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket
mpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java
Not sure if its due to some multicast permissions on my Windows or any config related issues. Please suggest some good pointers in this regard

Answer 1

You need to enable multicast. see this post how to enable multicast is windows, https://serverfault.com/questions/262634/how-do-i-know-if-ip-multicasting-is-enabled-on-my-network-in-windows
In Linux Environment most of the system kernel is capable to process the multicast address.
but we need to add route entry in kernel routing table.
sudo route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

Answer 2

The error that you're reporting looks like it might be OS / networking related. Hard to say exactly. I can tell you this though.
You've declared a TcpFailureDetector.
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
This is part of Tomcat's cluster configuration and it's responsibility is to make a TCP connection to any node that is suspected of failure. If the listener can't connect to the node, it'll be marked as down. If it can connect, then the node remains active.
In this case, the listener is attempting to make a connection to one of your nodes and it's failing with an error from the JDK. Googling that specific error turns up some suggestions, one of which is to use "-Djava.net.preferIPv4Stack=true".
I get java.net.SocketException: Permission denied: connect when sending an email in Jenkins
On a separate note, if you've note seen the official documentation on Tomcat clustering, I would strongly recommend you check that out. Here's the link.
http://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html