How to use firewall with ActiveMQ? - activemq

I have this ports configured in my 3 virtual machines running Zookeeper and ActiveMQ.
root#mom3:~# ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To Action From
-- ------ ----
22 ALLOW IN Anywhere
2881 ALLOW IN Anywhere
2888 ALLOW IN Anywhere
3888 ALLOW IN Anywhere
61616 ALLOW IN Anywhere
61617 ALLOW IN Anywhere
22 (v6) ALLOW IN Anywhere (v6)
2881 (v6) ALLOW IN Anywhere (v6)
2888 (v6) ALLOW IN Anywhere (v6)
3888 (v6) ALLOW IN Anywhere (v6)
61616 (v6) ALLOW IN Anywhere (v6)
61617 (v6) ALLOW IN Anywhere (v6)
When I tried to start ActiveMQ, it gets a random port to use it:
INFO | Master started: tcp://mom1.company.com:37649
WARN | Store update waiting on 1 replica(s) to catch up to log position 0.
WARN | Store update waiting on 1 replica(s) to catch up to log position 0.
WARN | Store update waiting on 1 replica(s) to catch up to log position 0.
But when I disable my firewall, ActiveMQ start normally.
How can I use the same port every time, in order to create a new rule in my firewall ?
EDIT
Based on #Daniel's suggestion this is my configuration for activemq.xml file.
<persistenceAdapter>
<replicatedLevelDB
directory="${activemq.data}/leveldb"
replicas="3"
bind="tcp://0.0.0.0:0:61616"
zkAddress="mom1.company.com:2881,mom2.company.com:2881,mom3.company.com:2881"
zkPassword="password"
zkPath="/activemq/leveldb-stores"
hostname="mom3.company"
/>
</persistenceAdapter>
...
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ssl" uri="ssl://0.0.0.0:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>

Since you are writting about a zookeeper and I vaguelly rememeber this log line from when I was working with a master/slave replicated levelDB Setup I'll go ahead and assume you are also using one. If this is indeed the case then the port you are seeing there is the "bind" port the master starts up for clients to attach themselves to and start replicating data. This port can easily be configured in your brokers XML configuration using the bind parameter in the replicatedLevelDB section, for example
<broker brokerName="broker" ... >
...
<persistenceAdapter>
<replicatedLevelDB
directory="activemq-data"
replicas="3"
bind="tcp://0.0.0.0:<myDesiredPort>"
zkAddress="zoo1.example.org:2181,zoo2.example.org:2181,zoo3.example.org:2181"
zkPassword="password"
zkPath="/activemq/leveldb-stores"
hostname="broker1.example.org"
/>
</persistenceAdapter>
...
</broker>
will then always use "myDesiredPort" for the bind port. Since normally 61619 is the default port when this parameter is not set at all you probably already have this element configured right now, however with bind="tcp://0.0.0.0:0" which dynamically choses one. For more explanation and a full list of available parameters for the replicated levelDB see the documentation
Hope this solves your Problem, if this is however not your setup please add your Broker configuration to your question this will make it easier to find the actual culprit without guessing.

Related

jmx doesnt seem to be working with activeMQ

I'm trying to use JMX with activeMQ for monitoring so far I've been using this and this as a reference but so far I'm unable to connect to jmx remotely and also I don't see any mention of jmx url in activemq logs. I'm wondering if there is another way to make sure jmx is working? is it supposed to be indicated in activemq logs?
PS I'm using jdk1.7 and activeMQ 5.14.2.
Thanks in advance!
EDIT
I set useJmx="true" in my activemq.xml file:
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="primary" useJmx="true" dataDirectory="${activemq.data}">
I tried two steps:
FIRST
I tried changing management context from createConnector="false" to :
<managementContext>
<managementContext createConnector="true" connectorPort="1099"/>
</managementContext>
FOR FIRST TIME THE PORT IS OPEN AND ACTIVEMQ RUNS FINE AND JMX URL GETS REPORTED IN LOGS ALTHOUGH I CAN NOT CONNECT IT TO IT REMOTLEY BUT IM ASSUMING ITS WORKING
SECOND
I reverted back the changes I made for managmentContext and I tried setting:
ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_BASE}/jmx.password -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_BASE}/jmx.access"
in bin/activemq script and I set a username in conf/jmx.access file as:
admin readwrite
And also have set a password in conf/jmx.password:
admin activemq
NOW ACTIVEMQ IS NOT RUNNINT AT ALL BUT IT WILL RUN IF I SET
AUTHENTICATE=FALSE AND DELETE JMX.ACCESS AND JMX.PASSWORD
CONFIGURATION IN BIN/ACTIVEMQ FILE BUT I NEED USER NAME AND PASSWORD
FOR SECURITY REASONS
I found this post which has the exact same issue as mine. any ideas?
Password authentication for remote monitoring is enabled by default. To disable it, set the following system property when you start the JVM:
-Dcom.sun.management.jmxremote.authenticate=false like you done in second test but you need to add system property -Dcom.sun.management.jmxremote
Try to add these jvm param to env file and update host ip
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=X.X.X.X
UPDATE
SO, to resume, i think that the FIRST step you tried is the best, for making it working these are the steps :
revert all jmx env file changes, like this :
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099 "
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote"
<broker useJmx="true" ...
<managementContext>
<managementContext createConnector="true" connectorPort="1099" />
</managementContext>
verify that in AMQ logs you have
INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi |
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
NOTE : Assuming that 10.10.10.16 is the IP of AMQ host.
try to connect with jconsole from another machine than AMQ host with url "service:jmx:rmi:///jndi/rmi://10.10.10.16:1099/jmxrmi" without user/pwd.
if you cannot connect, try like this :
<managementContext>
<managementContext createConnector="true" connectorPort="1099" connectorHost="10.10.10.16" />
</managementContext>
verify that in AMQ logs you have
INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://10.10.10.16:1099/jmxrmi |
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
retry to connect, step 4
at this step normally you can connect with jconsole.
if you want to add security and authorizations, use this :
<managementContext>
<managementContext createConnector="true" connectorPort="1099" connectorHost="10.10.10.16" >
<property xmlns="http://www.springframework.org/schema/beans" name="environment">
<map xmlns="http://www.springframework.org/schema/beans">
<entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.password.file"
value="${activemq.conf}/jmx.password"/>
<entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.access.file"
value="${activemq.conf}/jmx.access"/>
</map>
</property>
</managementContext>
</managementContext>
Please try these steps and let me know in which one you fails to connect and provide error message from jconsole.
A couple troubleshooting steps:
Start jconsole or visualvm on the same system and connect using the "pid" attach method. Browse the MBeans and confirm org.apache.activemq beans are present
Run netstat -na and confirm ports 1099 (and 44444) are in LISTEN
Look at logs and confirm you do not have any "java.net.BindException: Address already in use.." messages that indicate a port conflict with an already running Java process.
Edit bin/env to configure JMX (this disables requiring SSL, sets the port to 1099 and disables requiring username and password.
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099 "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote "
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"`
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"

Infinispan and jgroups Implementation on Openshift Stack

Question-1
I am trying run an Infinispan cluster on Openshift Tomcat gears. With nodes sitting on two different hosts.I am using TCP as the data transfer protocol and MPING as the discovery protocol.
If I try to use any of the JGROUPS provided key word for bind address like GLOBAL, SITE_LOCAL, LINK_LOCAL, NON_LOOPBACK, match-interface, match-host, match-address, except for LOOPBACK it binds the service to public IP(64.X.X.X) and for LOOPBACK it binds it to 127.0.0.1.This is not what I want to achieve.
I want it to run the JGROUPS service on the custom IP address provided by Openshift which looks some what like this 127.2.155.1. If I am able to run it in the given IP then it will be easy for me to write Port forwarding rules so that the cluster members will be able to discover each other even if they exist in different hosts.
Using environment property
Map<String, String> envKeys = System.getenv();
for (String keys : envKeys.keySet()) {
System.out.println(keys + ":" + envKeys.get(keys));
if (keys.equalsIgnoreCase("OPENSHIFT_JBOSSEWS_IP")) {
System.setProperty("OPENSHIFT_JBOSSEWS_IP", envKeys.get(keys));
}
}
It fails while doing above saying could not find the IP address or 127.2.155.1 is an invalid IP address. Please find the sample jgroups.xml I am using in my project.
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema /JGroups-3.4.xsd">
<TCP
bind_addr="${OPENSHIFT_JBOSSEWS_IP}"
bind_port="${jgroups.tcp.port:7800}"
port_range="0"
recv_buf_size="20m"
send_buf_size="640k"
max_bundle_size="31k"
use_send_queues="true"
enable_diagnostics="false"
bundler_type="sender-sends-with-timer"
thread_naming_pattern="pl"
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="30"
thread_pool.keep_alive_time="60000"
thread_pool.queue_enabled="true"
thread_pool.queue_max_size="100"
thread_pool.rejection_policy="Discard"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="2"
oob_thread_pool.max_threads="30"
oob_thread_pool.keep_alive_time="60000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="Discard"
internal_thread_pool.enabled="true"
internal_thread_pool.min_threads="2"
internal_thread_pool.max_threads="4"
internal_thread_pool.keep_alive_time="60000"
internal_thread_pool.queue_enabled="true"
internal_thread_pool.queue_max_size="100"
internal_thread_pool.rejection_policy="Discard"
/>
<!-- Ergonomics, new in JGroups 2.11, are disabled by default in TCPPING until JGRP-1253 is resolved -->
<!--
<TCPPING timeout="3000"
initial_hosts="localhost[7800],localhost[7801]"
port_range="5"
num_initial_members="3"
ergonomics="false"
/>
-->
<MPING bind_addr="{OPENSHIFT_JBOSSEWS_IP}"
break_on_coord_rsp="true"
mcast_addr="${jgroups.mping.mcast_addr:228.2.4.6}"
mcast_port="${jgroups.mping.mcast_port:43376}"
ip_ttl="${jgroups.udp.ip_ttl:2}"
num_initial_members="3"/>
<MERGE3/>
<FD_SOCK/>
<FD timeout="3000" max_tries="5"/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK2 use_mcast_xmit="false"
xmit_interval="1000"
xmit_table_num_rows="100"
xmit_table_msgs_per_row="10000"
xmit_table_max_compaction_time="10000"
max_msg_batch_size="100"/>
<UNICAST3 xmit_interval="500"
xmit_table_num_rows="20"
xmit_table_msgs_per_row="10000"
xmit_table_max_compaction_time="10000"
max_msg_batch_size="100"
conn_expiry_timeout="0"/>
<pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="1m"/>
<pbcast.GMS print_local_addr="false" join_timeout="3000" view_bundling="true"/>
<tom.TOA/> <!-- the TOA is only needed for total order transactions-->
<MFC max_credits="2m" min_threshold="0.40"/>
<FRAG2 frag_size="30k"/>
<RSVP timeout="60000" resend_interval="500" ack_on_delivery="false" />
</config>
Question-2
When Infinispan is successfully started, It runs 2 java processes one on the port 7800 as mentioned in the above config file and other on a port number randomly picked up by Infinispan. I would like to understand more about the processes.
**COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME**
java 5640 1334 44u IPv4 1556890779 0t0 TCP 127.2.155.1:7800 (LISTEN)
java 5640 1334 44u IPv4 1556890779 0t0 TCP 127.2.155.1:20772 (LISTEN)
Question-1:
Setting bind_addr to bind_addr=match-address=127.2.\* should pick the correct address (127.2.155.1).
Does 127.2.155.1 show when you do an ifconfig?
Question-2:
The second port is probably opened by FD_SOCK. You can control this by setting attributes bind_addr and start_port in FD_SOCK.
[1] http://www.jgroups.org/manual/index.html#CommonProps

How to run two instances of Apache ActiveMQ in one system?

I am using apache-activemq-5.11.1 which is the stable version runs on JDK 7 (Major version 51.0), I am using JDK 7 Update 80. I had error if I run the same on JDK 6.
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/ac
tivemq/console/Main : Unsupported major.minor version 51.0
Coming to my problem I need to have two running instances of ActiveMQ in my system. I had followed the following steps to create two instance.
C:\>cd \apache-activemq-5.11.1
C:\apache-activemq-5.11.1>.\bin\activemq create instance1
C:\apache-activemq-5.11.1>.\bin\activemq create instance2
I had changed to different set of port numbers for instance2 as below,
<!--EDITED: apache-activemq-5.11.1\instance2\conf\activemq.xml-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61716?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5772?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61713?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1983?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61714?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
Now I am starting instance1 & instance2 as follows.....
C:\apache-activemq-5.11.1\instance1\bin>instance1 start
C:\apache-activemq-5.11.1\instance1\bin>instance2 start
Among these the second instance which I am trying to start gives the following kahadb lock problem.....
INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1#7209d9af: startup date [Thu May 07 16:16:23 IST 2015]; root of context hierarchy
INFO | PListStore:[C:\apache-activemq-5.11.1\data\localhost\tmp_storage] started
INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\apache-activemq-5.11.1\data\kahadb]
INFO | Database C:\apache-activemq-5.11.1\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File 'C:\apache-activemq-5.11.1\data\kahadb\lock' could not be locked.
Please give a solution for this db lock issue.
Make a replica of your ActiveMQ like apache-activemq-x.xx.x to apache-activemq-x.xx.x_2
Change the ports of apache-activemq-x.xx.x_2\conf\activemq.xml. Make sure the port numbers that you are changing are not in clash.
<!--EDIT: apache-activemq-5.11.1_2\conf\activemq.xml-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61716?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5772?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61713?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1983?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61714?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
And along with port changes we have to correct jetty.xml http management console port as well.
<bean id="jettyPort" class="org.apache.activemq.web.WebConsolePort" init-method="start">
<!-- the default port number for the web console -->
<property name="host" value="0.0.0.0"/>
<property name="port" value="8162"/>
</bean>
In this way you can run two services of ActiveMQ in one system.
Running more instances created for ActiveMQ gives the Fail Over option.
In this way when one instance goes down for some reason. The other instance coming up automatically, since the first instance which locks the KahaDB is been released.
For this ports are not to be changed,- as we're configuring instances for Fail Over Mode.
C:\>cd \apache-activemq-5.11.1
C:\apache-activemq-5.11.1>.\bin\activemq create instance1
C:\apache-activemq-5.11.1>.\bin\activemq create instance2
Please start instances without changing any of the configuration. So when instance1 goes down for any reason, instance2 coming up.
C:\apache-activemq-5.11.1\instance1\bin>instance1 start
C:\apache-activemq-5.11.1\instance2\bin>instance2 start
I hope this must be the purpose of creating multiple instances under ActiveMQ. And more than this some more tweaky config. also available for Kahadb.
I did do the following instructions from:
Running Multiple ActiveMQ Instances on One Machine (Dzone)
It works fine for Mac (Not tested in Linux)
Note: the instances must be started through instanceNumber start (the console argument/parameter is not valid anymore).
I had the same problem about the kahadb locking only for Windows, it for ActiveMQ 5.13.3 and 5.14.5 versions
The same author from DZone wrote practically the same post in his blog
Running multiple ActiveMQ instances on one machine (Blog)
But there exists an important update.
You must open each instanceNumber.bat file for each instance from each bin directory and add these two lines:
set ACTIVEMQ_CONF="ACTIVEMQ_HOME/instanceNumber/conf"
set ACTIVEMQ_DATA="ACTIVEMQ_HOME/instanceNumber/data"
Where ACTIVEMQ_HOME represents the path location of your ActiveMQ and instanceNumber is the instance being edited such as: instanceA and instanceB
what is happening is you have changed the port numbers correctly but both the instances that you created use a same Database(in this case file system KahaDB) to store their messages,
So when one instance is up and running, it holds the lock for that database and other instance of activeMQ will be waiting to gain a lock of this DB.
Essentially this is becoming a master slave configuration .
look at this line in activeMQ.xml
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
this will be pointing so same location for both instances.
what my solution is to copy entire folder apache-activemq-x.xx.x in different location change the port numbers for second instance and run them differently
by this you will have 2 instances of activeMQ running on same machine
hope this helps!
Good luck!
Although because of KahaDB restriction load balancing/fault tolerant configuration is restricted. We can use following kind of connection URL to utilize ActiveMQ load....
failover://(tcp://192.nnn.nn.nn:61616,tcp://192.nnn.nn.nn:61616)?randomize=false
randomize=true will made message shuffles between two AciveMQ in active mode, rather not by just fail-over of ActiveMQ......
Complete reference for this can be found under the following Apache Site link....
http://activemq.apache.org/failover-transport-reference.html
But Still high availability (i.e, cluster) configuration make things stable for your App although Apache must advance ActiveMQ High Availability, hence things can work smoother.
The present Apache ActiveMQ High Availability configuration available in the following link.
http://activemq.apache.org/clustering.html
Although KahaDB has file lock restriction, following tweaking/alternates ways of configuration can be done...
1)Shared File System Master Slave,- A shared file system such as a SAN
http://activemq.apache.org/shared-file-system-master-slave.html
2)JDBC Master Slave,- A Shared database
http://activemq.apache.org/jdbc-master-slave.html
3)Replicated LevelDB Store,- ZooKeeper Server
http://activemq.apache.org/replicated-leveldb-store.html
Over & above by having JCA connectors,- AS like JBoss, Weblogic, Websphere, Geronimo, Glassfish,- ActimeMQ patching as a kind of Resource Adapter can be done. And with Apache Camel (karaf), JBoss Fuse ESB kind of products HA & clustering of ActiveMQ can be done.

Tomcat Clustering Multicast throws error - Unable to perform failure detection check, assuming member down

Hi I am new to clustering concept, so I tried to establish a Tomcat cluster with 2 instances. I followed the examples, mainly at http://www.mulesoft.com/tcat/tomcat-clustering and few other web sources.
My sample set up is as,
in workers.properties
# Define worker names
worker.list=jkstatus, loadbalancer
# Create virtual workers
worker.jkstatus.type=status
worker.loadbalancer.type=lb
# Declare Tomcat server worker 1
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=7009
# Declare Tomcat server worker 2
worker.worker2.type=ajp13
worker.worker2.port=9009
worker.worker2.host=localhost
# Associate real workers with virtual LoadBalancer worker
worker.loadbalancer.balance_workers=worker1,worker2
And Apache httpd.conf as,
# ADDED CLUSTER CONFIG
# Load module
LoadModule jk_module modules/mod_jk.so
# Specify path to worker configuration file
JkWorkersFile C:/tomcat_clustered/workers.properties
# Configure logging and memory
JkShmFile C:/tomcat_clustered/log/location/mod_jk.shm
JkLogFile C:/tomcat_clustered/log/location/mod_jk.log
JkLogLevel info
# Configure monitoring
JkMount /jkmanager/* jkstatus
<Location /jkmanager>
Order deny,allow
deny from all
allow from localhost
</Location>
# Configure applications
JkMount /* loadbalancer
# END CLUSTER CONFIG
With each tomcat(worker) server.xml as,
<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker1">
<!--For clustering, please take a look at documentation at:
/docs/cluster-howto.html (simple how to)
/docs/config/cluster.html (reference documentation) -->
<!--
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
-->
<!-- CLUSTER BEGIN -->
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<!---->
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="239.0.0.1"
port="45564" frequency="500"
dropTime="3000"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto" port="4000" autoBind="100"
selectorTimeout="5000" maxThreads="6"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
<!-- CLUSTER END -->
<!-- Use the LockOutRealm to prevent attempts to guess user passwords
via a brute-force attack -->
<Realm className="org.apache.catalina.realm.LockOutRealm">
For worker1, worker2. I am working on Windows 7. It looks like some multicast option is needed but I am not sure. On starting up of tomcat, one is starting well, and on starting second(say worker2) both tomcat console's are throwing errors as,
at org.apache.catalina.tribes.group.ChannelInterceptorBase.heartbeat(Cha
nelInterceptorBase.java:103)
at org.apache.catalina.tribes.group.GroupChannel.heartbeat(GroupChannel.
ava:155)
at org.apache.catalina.tribes.group.GroupChannel$HeartbeatThread.run(Gro
pChannel.java:690)
ep 15, 2014 10:11:03 AM org.apache.catalina.tribes.group.interceptors.TcpFailur
Detector memberAlive
EVERE: Unable to perform failure detection check, assuming member down.
ava.net.SocketException: Permission denied: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketI
pl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja
a:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket
mpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java
Not sure if its due to some multicast permissions on my Windows or any config related issues. Please suggest some good pointers in this regard
You need to enable multicast. see this post how to enable multicast is windows, https://serverfault.com/questions/262634/how-do-i-know-if-ip-multicasting-is-enabled-on-my-network-in-windows
In Linux Environment most of the system kernel is capable to process the multicast address.
but we need to add route entry in kernel routing table.
sudo route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
The error that you're reporting looks like it might be OS / networking related. Hard to say exactly. I can tell you this though.
You've declared a TcpFailureDetector.
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
This is part of Tomcat's cluster configuration and it's responsibility is to make a TCP connection to any node that is suspected of failure. If the listener can't connect to the node, it'll be marked as down. If it can connect, then the node remains active.
In this case, the listener is attempting to make a connection to one of your nodes and it's failing with an error from the JDK. Googling that specific error turns up some suggestions, one of which is to use "-Djava.net.preferIPv4Stack=true".
I get java.net.SocketException: Permission denied: connect when sending an email in Jenkins
On a separate note, if you've note seen the official documentation on Tomcat clustering, I would strongly recommend you check that out. Here's the link.
http://tomcat.apache.org/tomcat-7.0-doc/cluster-howto.html

ActiveMQ 5.8 network of broker with custom jmx port

I am trying to run a network of broker with 2 brokers on the same network but on 2 different virtual machines.
Because of some internal constraints I have to use a custom jmx port.
I am using the Tanuki wrapper to launch ActiveMQ on an Ubuntu server.
Here is the relevant part of my activemq.xml
<broker xmlns="http://activemq.apache.org/schema/core" advisorySupport="false" useJmx="true" brokerName="test1" dataDirectory=".../data/activemq">
<networkConnectors>
<networkConnector uri="multicast://1.2.3.4:101234?group=test"
dynamicOnly="true"
networkTTL="3"
prefetchSize="1"
decreaseNetworkConsumerPriority="true"
userName="user"
password="password"/>
</networkConnectors>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616" rebalanceClusterClients="true" updateClusterClients="true" />
<transportConnector name="nio" uri="nio://0.0.0.0:61617" rebalanceClusterClients="true" updateClusterClients="true" discoveryUri="multicast://1.2.3.4:101234?group=test" />
</transportConnectors>
...
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
...
</broker>
Here is the relevant part of wrapper.conf:
# Uncomment to enable jmx
wrapper.java.additional.1=-Dcom.sun.management.jmxremote
wrapper.java.additional.2=-Dcom.sun.management.jmxremote.port=4321
wrapper.java.additional.3=-Dcom.sun.management.jmxremote.authenticate=false
wrapper.java.additional.4=-Dcom.sun.management.jmxremote.ssl=false
When running ActiveMQ on both broker, I see the process with the expected options:
activemq 30682 30680 3 13:27 ? 00:00:30 java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=4321 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djavax.net.ssl.keyStore=../../conf/broker.ks -Djavax.net.ssl.trustStore=../../conf/broker.ts -Dcom.sun.management.jmxremote -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dactivemq.conf=../../conf -Dactivemq.data=../../data -Xms2048m -Xmx2048m -Djava.library.path=../../bin/linux-x86-64/ -classpath ../../bin/wrapper.jar:../../bin/activemq.jar -Dwrapper.key=y4TuwO32Hj6kN7w8 -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=30680 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp org.apache.activemq.console.Main start
The port is open on the running shorewall.
The network of broker is up but I cannot connect to the jmx using jvisualvm with server_dns:4321. It returns the error
"cannot connect to server_dns:4321 using service:jmx:rmi:///jndi/rmi://server_dns:4321/jmxrmi
I can see some traffic on the port via tcpdump.
Could anybody tell me what I am doing wrong or how I should use ActiveMQ as a Network Of Broker with a custom jmx port?
JMX needs 2 open ports. An extra one is necessary for rmi.
I figured it out thanks to this post: Apache ActiveMQ browser can't connect to JMX console
In my case the fix is to change the configuration of my wrapper to expose the rmi port and open the port on the firewall
# Uncomment to enable jmx
wrapper.java.additional.1=-Dcom.sun.management.jmxremote
wrapper.java.additional.2=-Dcom.sun.management.jmxremote.port=4321
wrapper.java.additional.3=-Dcom.sun.management.jmxremote.port=8765
wrapper.java.additional.4=-Dcom.sun.management.jmxremote.authenticate=false
wrapper.java.additional.5=-Dcom.sun.management.jmxremote.ssl=false