How to run two instances of Apache ActiveMQ in one system? - apache

I am using apache-activemq-5.11.1 which is the stable version runs on JDK 7 (Major version 51.0), I am using JDK 7 Update 80. I had error if I run the same on JDK 6.
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/ac
tivemq/console/Main : Unsupported major.minor version 51.0
Coming to my problem I need to have two running instances of ActiveMQ in my system. I had followed the following steps to create two instance.
C:\>cd \apache-activemq-5.11.1
C:\apache-activemq-5.11.1>.\bin\activemq create instance1
C:\apache-activemq-5.11.1>.\bin\activemq create instance2
I had changed to different set of port numbers for instance2 as below,
<!--EDITED: apache-activemq-5.11.1\instance2\conf\activemq.xml-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61716?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5772?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61713?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1983?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61714?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
Now I am starting instance1 & instance2 as follows.....
C:\apache-activemq-5.11.1\instance1\bin>instance1 start
C:\apache-activemq-5.11.1\instance1\bin>instance2 start
Among these the second instance which I am trying to start gives the following kahadb lock problem.....
INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1#7209d9af: startup date [Thu May 07 16:16:23 IST 2015]; root of context hierarchy
INFO | PListStore:[C:\apache-activemq-5.11.1\data\localhost\tmp_storage] started
INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\apache-activemq-5.11.1\data\kahadb]
INFO | Database C:\apache-activemq-5.11.1\data\kahadb\lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File 'C:\apache-activemq-5.11.1\data\kahadb\lock' could not be locked.
Please give a solution for this db lock issue.

Make a replica of your ActiveMQ like apache-activemq-x.xx.x to apache-activemq-x.xx.x_2
Change the ports of apache-activemq-x.xx.x_2\conf\activemq.xml. Make sure the port numbers that you are changing are not in clash.
<!--EDIT: apache-activemq-5.11.1_2\conf\activemq.xml-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61716?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5772?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61713?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1983?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61714?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
And along with port changes we have to correct jetty.xml http management console port as well.
<bean id="jettyPort" class="org.apache.activemq.web.WebConsolePort" init-method="start">
<!-- the default port number for the web console -->
<property name="host" value="0.0.0.0"/>
<property name="port" value="8162"/>
</bean>
In this way you can run two services of ActiveMQ in one system.

Running more instances created for ActiveMQ gives the Fail Over option.
In this way when one instance goes down for some reason. The other instance coming up automatically, since the first instance which locks the KahaDB is been released.
For this ports are not to be changed,- as we're configuring instances for Fail Over Mode.
C:\>cd \apache-activemq-5.11.1
C:\apache-activemq-5.11.1>.\bin\activemq create instance1
C:\apache-activemq-5.11.1>.\bin\activemq create instance2
Please start instances without changing any of the configuration. So when instance1 goes down for any reason, instance2 coming up.
C:\apache-activemq-5.11.1\instance1\bin>instance1 start
C:\apache-activemq-5.11.1\instance2\bin>instance2 start
I hope this must be the purpose of creating multiple instances under ActiveMQ. And more than this some more tweaky config. also available for Kahadb.

I did do the following instructions from:
Running Multiple ActiveMQ Instances on One Machine (Dzone)
It works fine for Mac (Not tested in Linux)
Note: the instances must be started through instanceNumber start (the console argument/parameter is not valid anymore).
I had the same problem about the kahadb locking only for Windows, it for ActiveMQ 5.13.3 and 5.14.5 versions
The same author from DZone wrote practically the same post in his blog
Running multiple ActiveMQ instances on one machine (Blog)
But there exists an important update.
You must open each instanceNumber.bat file for each instance from each bin directory and add these two lines:
set ACTIVEMQ_CONF="ACTIVEMQ_HOME/instanceNumber/conf"
set ACTIVEMQ_DATA="ACTIVEMQ_HOME/instanceNumber/data"
Where ACTIVEMQ_HOME represents the path location of your ActiveMQ and instanceNumber is the instance being edited such as: instanceA and instanceB

what is happening is you have changed the port numbers correctly but both the instances that you created use a same Database(in this case file system KahaDB) to store their messages,
So when one instance is up and running, it holds the lock for that database and other instance of activeMQ will be waiting to gain a lock of this DB.
Essentially this is becoming a master slave configuration .
look at this line in activeMQ.xml
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
this will be pointing so same location for both instances.
what my solution is to copy entire folder apache-activemq-x.xx.x in different location change the port numbers for second instance and run them differently
by this you will have 2 instances of activeMQ running on same machine
hope this helps!
Good luck!

Although because of KahaDB restriction load balancing/fault tolerant configuration is restricted. We can use following kind of connection URL to utilize ActiveMQ load....
failover://(tcp://192.nnn.nn.nn:61616,tcp://192.nnn.nn.nn:61616)?randomize=false
randomize=true will made message shuffles between two AciveMQ in active mode, rather not by just fail-over of ActiveMQ......
Complete reference for this can be found under the following Apache Site link....
http://activemq.apache.org/failover-transport-reference.html
But Still high availability (i.e, cluster) configuration make things stable for your App although Apache must advance ActiveMQ High Availability, hence things can work smoother.
The present Apache ActiveMQ High Availability configuration available in the following link.
http://activemq.apache.org/clustering.html
Although KahaDB has file lock restriction, following tweaking/alternates ways of configuration can be done...
1)Shared File System Master Slave,- A shared file system such as a SAN
http://activemq.apache.org/shared-file-system-master-slave.html
2)JDBC Master Slave,- A Shared database
http://activemq.apache.org/jdbc-master-slave.html
3)Replicated LevelDB Store,- ZooKeeper Server
http://activemq.apache.org/replicated-leveldb-store.html
Over & above by having JCA connectors,- AS like JBoss, Weblogic, Websphere, Geronimo, Glassfish,- ActimeMQ patching as a kind of Resource Adapter can be done. And with Apache Camel (karaf), JBoss Fuse ESB kind of products HA & clustering of ActiveMQ can be done.

Related

jmx doesnt seem to be working with activeMQ

I'm trying to use JMX with activeMQ for monitoring so far I've been using this and this as a reference but so far I'm unable to connect to jmx remotely and also I don't see any mention of jmx url in activemq logs. I'm wondering if there is another way to make sure jmx is working? is it supposed to be indicated in activemq logs?
PS I'm using jdk1.7 and activeMQ 5.14.2.
Thanks in advance!
EDIT
I set useJmx="true" in my activemq.xml file:
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="primary" useJmx="true" dataDirectory="${activemq.data}">
I tried two steps:
FIRST
I tried changing management context from createConnector="false" to :
<managementContext>
<managementContext createConnector="true" connectorPort="1099"/>
</managementContext>
FOR FIRST TIME THE PORT IS OPEN AND ACTIVEMQ RUNS FINE AND JMX URL GETS REPORTED IN LOGS ALTHOUGH I CAN NOT CONNECT IT TO IT REMOTLEY BUT IM ASSUMING ITS WORKING
SECOND
I reverted back the changes I made for managmentContext and I tried setting:
ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_BASE}/jmx.password -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_BASE}/jmx.access"
in bin/activemq script and I set a username in conf/jmx.access file as:
admin readwrite
And also have set a password in conf/jmx.password:
admin activemq
NOW ACTIVEMQ IS NOT RUNNINT AT ALL BUT IT WILL RUN IF I SET
AUTHENTICATE=FALSE AND DELETE JMX.ACCESS AND JMX.PASSWORD
CONFIGURATION IN BIN/ACTIVEMQ FILE BUT I NEED USER NAME AND PASSWORD
FOR SECURITY REASONS
I found this post which has the exact same issue as mine. any ideas?
Password authentication for remote monitoring is enabled by default. To disable it, set the following system property when you start the JVM:
-Dcom.sun.management.jmxremote.authenticate=false like you done in second test but you need to add system property -Dcom.sun.management.jmxremote
Try to add these jvm param to env file and update host ip
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=X.X.X.X
UPDATE
SO, to resume, i think that the FIRST step you tried is the best, for making it working these are the steps :
revert all jmx env file changes, like this :
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099 "
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote"
<broker useJmx="true" ...
<managementContext>
<managementContext createConnector="true" connectorPort="1099" />
</managementContext>
verify that in AMQ logs you have
INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi |
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
NOTE : Assuming that 10.10.10.16 is the IP of AMQ host.
try to connect with jconsole from another machine than AMQ host with url "service:jmx:rmi:///jndi/rmi://10.10.10.16:1099/jmxrmi" without user/pwd.
if you cannot connect, try like this :
<managementContext>
<managementContext createConnector="true" connectorPort="1099" connectorHost="10.10.10.16" />
</managementContext>
verify that in AMQ logs you have
INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://10.10.10.16:1099/jmxrmi |
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
retry to connect, step 4
at this step normally you can connect with jconsole.
if you want to add security and authorizations, use this :
<managementContext>
<managementContext createConnector="true" connectorPort="1099" connectorHost="10.10.10.16" >
<property xmlns="http://www.springframework.org/schema/beans" name="environment">
<map xmlns="http://www.springframework.org/schema/beans">
<entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.password.file"
value="${activemq.conf}/jmx.password"/>
<entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.access.file"
value="${activemq.conf}/jmx.access"/>
</map>
</property>
</managementContext>
</managementContext>
Please try these steps and let me know in which one you fails to connect and provide error message from jconsole.
A couple troubleshooting steps:
Start jconsole or visualvm on the same system and connect using the "pid" attach method. Browse the MBeans and confirm org.apache.activemq beans are present
Run netstat -na and confirm ports 1099 (and 44444) are in LISTEN
Look at logs and confirm you do not have any "java.net.BindException: Address already in use.." messages that indicate a port conflict with an already running Java process.
Edit bin/env to configure JMX (this disables requiring SSL, sets the port to 1099 and disables requiring username and password.
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099 "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote "
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"`
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"

ActiveMQ doesn't recognize Stomp: Transport scheme NOT recognized: [stomp+ssl]

I'm trying to set up mcollective/activemq on a puppetmaster (open source puppet). I am having a problem where ActiveMQ does not recognize the Stomp protocol. Here is the relevant snippet in my /etc/activemq/instances-enabled/activemq/activemq.xml file that should enable stomp+ssl:
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
<transportConnector name="stomp+ssl" uri="stomp+ssl://0.0.0.0:61614?needClientAuth=true&transport.enabledProtocols=TLSv1,TLSv1.1,TLSv1.2"/>
</transportConnectors>
When I start ActiveMQ via service activemq start, I notice that the daemon doesn't end up running (I don't see it as a process). Then I tried running service activemq console activemq, and it looks like the problem is that it can't find the stomp Transport scheme. Here is the first error that I see in the output (and the error persists throughout the output):
ERROR | Failed to start Apache
ActiveMQ ([localhost,
ID:my-servers-hostname.example.com-40447-1475514312306-0:1], java.io.IOException: Transport
Connector could not be registered in
JMX: java.io.IOException: Transport
scheme NOT recognized: [stomp+ssl])
ActiveMQ recognizes openwire just fine. When using openwire+ssl only, without using stomp+ssl, the ActiveMQ daemon starts fine with no errors. However, when I try running mco find, I get an error because it seems that mco is still trying to use stomp+ssl (and ActiveMQ only has openwire+ssl enabled):
error 2016/10/03 17:26:59: activemq.rb:149:in `on_ssl_connectfail' SSL session creation with stomp+ssl://mcollective#localhost:61614 failed: Connection refused - connect(2) for "localhost" port 61614
Perhaps I need to adjust my mco config to use openwire instead of stomp? I wasn't sure where or what file that configuration would be in. Not sure why it doesn't recognize stomp, but I was wondering what my options are here. Is it even possible to use MCollective/ActiveMQ using only openwire+ssl, or is using stomp a requirement if I want to use mco? I don't think this is a port issue, as the relevant ports are open on the server I believe.
Here are the relevant packages/versions installed on my machine:
OS: Ubuntu 16.04 (xenial)
puppet: 4.7.0
ActiveMQ: 5.13.2
ruby-stomp: 1.3.5-1
MCollective (mco) version: 2.9.0
I've run into the same problem with the embedded ActiveMQ server in my project. Turns out I needed to add the following dependencies to my pom.
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-stomp</artifactId>
<version>5.15.0</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-kahadb-store</artifactId>
<version>5.15.0</version>
</dependency>
In your case, I think you need to get hold of those 2 jars and add them to your ActiveMQ installation.
In activemq package provided by ubuntu 16+ library stomp transport was not included. I do not know why.
Yoy can download it manually and place in /usr/share/activemq/lib:
cd /usr/share/activemq/lib
# check your actviemq version before (apt-cache policy activemq) and use the relevant version of library.
wget https://repository.apache.org/content/repositories/releases/org/apache/activemq/activemq-stomp/5.13.5/activemq-stomp-5.13.5.jar
service activemq restart

Liferay 6.2 clustering issue with multicast

I am trying to cluster ehcache and lucene with Liferay 6.2 EE sp2 bundle on 2 servers with mutlicast enabled. WE have Apache HTTPD servers fronting tomcat servers using reverse proxy. A valid 6.2 license is deployed on both the nodes.
We user the following properties in the portal-ext.properties:
cluster.link.enabled=true
lucene.replicate.write=true
ehcache.cluster.link.replication.enabled=true
# Since we are using SSL on the frontend
web.server.protocol=https
# set this to any server that is visible to both the nodes
cluster.link.autodetect.address=dbserverip:dbport
#ports and ips we know work in our environment for multicast
multicast.group.address["cluster-link-control"]=ip
multicast.group.port["cluster-link-control"]=port1
multicast.group.address["cluster-link-udp"]=ip
multicast.group.port["cluster-link-udp"]=port2
multicast.group.address["cluster-link-mping"]=ip
multicast.group.port["cluster-link-mping"]=port3
multicast.group.address["hibernate"]=ip
multicast.group.port["hibernate"]=port4
multicast.group.address["multi-vm"]=ip
multicast.group.port["multi-vm"]=port5
We are running into issues with the ehcache and lucene clustering not working. The following tests fail :
Moving a portlet on node 1, does not show up on node 2
There are no errors except for a startup error with lucene.
14:19:35,771 ERROR
[CLUSTER_EXECUTOR_CALLBACK_THREAD_POOL-1][LuceneHelperImpl:1186]
Unable to load index for company 10157
com.liferay.portal.kernel.exception.SystemException:
java.net.ConnectException: Connection refused at
com.liferay.portal.search.lucene.LuceneHelperImpl.getLoadIndexesInputStreamFromCluster(LuceneHelperImpl.java:488)
at
com.liferay.portal.search.lucene.LuceneHelperImpl$LoadIndexClusterResponseCallback.callback(LuceneHelperImpl.java:1176)
at
com.liferay.portal.cluster.ClusterExecutorImpl$ClusterResponseCallbackJob.run(ClusterExecutorImpl.java:614)
at
com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask(ThreadPoolExecutor.java:682)
at
com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask.run(ThreadPoolExecutor.java:593)
at java.lang.Thread.run(Thread.java:745) Caused by:
java.net.ConnectException: Connection refused at
java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
java.net.Socket.connect(Socket.java:579) at
sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:625) at
sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:160)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at
sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at
sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at
sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:275)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:371)
We verified that the jgroups multicast works outside of liferay by running the following commands and using a downloaded copy of the jgroups.jar and replacing with the 5 multicast ips and ports.
Testing with JGROUPS
1) McastReceiver -
java -cp ./jgroups.jar org.jgroups.tests.McastReceiverTest -mcast_addr 224.10.10.10 -port 5555
ex. java -cp jgroups-final.jar org.jgroups.tests.McastReceiverTest -mcast_addr 224.10.10.10 -port 5555
2) McastSender -
java -cp ./jgroups.jar org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10 -port 5555
ex. java -cp jgroups-final.jar org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10 -port 5555
From there, typing things into the McastSender will result in the Receiver printing it out.
Thanks!
After a lot of troubleshooting and help from various folks in my team and at liferay support, we switched to using unicast and it worked a lot better.
Here is what we did:
Extracted jgroups.jar from the tomcat home/webappts/ROOT/WEB_INF/lib, saved locally.
Unzipped the jgroups.jar file and extracted and save the tcp.xml from the jar's WEB_INF folder
As a base line test, changed the section in the tcp.xml and saved
TCPPING timeout="3000"
initial_hosts="${jgroups.tcpping.initial_hosts:servername1[7800],servername2[7800]}"
port_range="1"
num_initial_members="10"
Copy the tcp.xml to the liferay home on both the nodes
Change the portal-ext.properties to remove the mutlicast properties and add the following lines.
cluster.link.channel.properties.control=${liferay.home}/tcp.xml
cluster.link.channel.properties.transport.0=${liferay.home}/tcp.xml
Start node 1
start node 2
check logs
Do the cluster cache test:
Moving a portlet on node 1, shows up on node 2
Under control panel -> License manager both the nodes show up with valid licenses.
searching for user on node 2 after adding in node 1 in control panel -> user and organizations.
All of the above tests worked.
So we shutdown servers and changed the tcp.xml to use jdbc rather than the tcpping so we don't have to specify node names manually.
Step for the jdbc config:
Create the table in the liferay database manually.
CREATE TABLE JGROUPSPING (own_addr varchar(200) not null, cluster_name varchar(200) not null, ping_data blob default null, primary key (own_addr, cluster_name))
change tcp.xml and remove the tcpping section and add the following.
Note: Please replace the leading \ with less than symbol in the following code block. There are issues with the leading less than sign in the SO editor/parser hiding whatever comes after it:
\JDBC_PING datasource_jndi_name="java:comp/env/jdbc/LiferayPool"
initialize_sql="" />
Save and push the file manually to both the nodes.
Start the servers and repeat tests above.
It should work seamlessly.
It was invaluable to have the debug logging on for jgroups mentioned in the following the post:
https://bitsofinfo.wordpress.com/2014/05/21/clustering-liferay-globally-across-data-centers-gslb-with-jgroups-and-relay2/
tomcat home/webapps/ROOT/WEB-INF/classes/META-INF/portal-log4j-ext.xml file I used to triage various issues on bootup related to clustering.
<?xml version="1.0"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<category name="com.liferay.portal.cluster">
<priority value="TRACE" />
</category>
<category name="com.liferay.portal.license">
<priority value="TRACE" />
</category>
We also found that the Lucene cluster replication startup errors were fixed in a fix pack and are getting a patch for it.
https://issues.liferay.com/browse/LPS-51714
https://issues.liferay.com/browse/LPS-51428
We added the following portal instance properties for lucene replication to work better between the 2 nodes:
portal.instance.http.port=port that the app servers listen on ex. 8080
portal.instance.protocol=http
Hope this helps someone.
Update
The lucene index load in a cluster issue was resolved by a Liferay 6.2 EE patch from support for the LPS's mentioned above.

activeMQ master/slave cluster with zookeeper

I have my activeMQ connected to zookeeper (a cluster of 5 zookeepers), in the config file "activemq.xml", I have
<persistenceAdapter>
<replicatedLevelDB
directory="${activemq.data}/leveldb"
replicas="3"
bind="tcp://0.0.0.0:0"
zkAddress="blablabla:2181"
zkPassword="password"
zkPath="/activemq/leveldb-stores"
hostname="blabla"
/>
</persistenceAdapter>
now I have activeMQ-server1 started, successfully become the master; activeMQ-server2 with the same "activemq.xml" config file, successfully become the slave; activeMQ-server3 with the same "activemq.xml" config file, successfully become the slave, but kicks out activeMQ-server2 (start to give connection error)
I think I put the wrong number for replicas, I changed all the 3 config files with "replicas="4"", still not work
what would be the correct replicas number with 3 activeMQ servers, or I am wrong with some other parts. (I only have 1 zookeeper listed in config, since the 5 zookeepers can connect to each other, already a cluster there)
Thanks :)
You need to list all zookeeper servers in the zkAddress portion, zkAddress="zoo1.example.org:2181,zoo2.example.org:2181,zoo3.example.org:2181", taken from activemq replicated levelDB
The replicas value is the number of activemq nodes, not the number of zookeeper nodes. So if you have 3 amq nodes, set replicas="3", not more. http://activemq.apache.org/replicated-leveldb-store.html :
Replicas property :
The number of nodes that will exist in the cluster.
At least (replicas/2)+1 nodes must be online to avoid service outage.
Another thing, all amq nodes in the cluster must get the same name (MyBroker below):
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="MyBroker" dataDirectory="${activemq.data}">

Unable to start AMQ broker with persistent store on an NFSv3 share

I've been struggling to start AMQ broker node with persistent store on an NFSv3 share.
I keep getting the below error complaining of unavailable locks.
I've made sure that all java processes are killed and the lock file on the shared folder is deleted before starting the AMQ master broker.
When I start AMQ, it seems to create a lock file on the shared folder and after that it complains of unavailable locks.
Loading message broker from: xbean:activemq.xml
INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1#73cf56e9: startup date [Mon Dec 23 05:28:23 UTC 2013]; root of context hierarchy
INFO | PListStore:[/home/pnarayan/apache-activemq-5.9.0/activemq-data/notificationsBroker/tmp_storage] started
INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/home/y/share/nfs/amqnfs]
INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
INFO | Database /home/y/share/nfs/amqnfs/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: No locks available
Below is the activemq xml configuration file I'm using:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<broker
xmlns="http://activemq.apache.org/schema/core"
xmlns:spring="http://www.springframework.org/schema/beans"
brokerName="notificationsBroker"
useJmx="true"
start="true"
persistent="true"
useShutdownHook="false"
deleteAllMessagesOnStartup="false">
<persistenceAdapter>
<kahaDB directory="/home/y/share/nfs/amqnfs" />
</persistenceAdapter>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
</broker>
</beans>
Is this because of the reason that I'm using NFSv3, but not NFSv4 as recommended by AMQ?
I believe the issue with NFSv3 is that it cannot cleanup the lock if at all the broker process dies abruptly. However it shouldn't be having any issues in starting the broker. If my understanding is right why am I observing the above error?
You are absolutely right in what you say - NFS3 does not clean up its locks properly. When using KahaDB, the broker creates a file in $ACTIVEMQ_DATA/lock. If that file exists, chances are that something has a hold on it (or at least NFS3 thinks that it does) and the broker will be blocked. Check to see whether the file is there, and if so, use the lsof command to determine the process id of its holder.