ActiveMQ HA On failover - activemq

Does ActiveMQ cluster (master-slave) offer high availability on failover mode?
I am publishing messages on a topic but when I kill active node and my consumer connect to the other node It loses several messages.
I need only topic mode, because I need to have different pods consuming messages at the same time. We have made test with durable messages and subscriptions wit we got the same problem.
If ActiveMQ doesn't support this functionality, is there other broker that could do this?
We have used Spring boot to develop sender and consumer apps.
SENDER:
To send messages we used and API REST that sends 5000 messages to a topic. If some convertAndSend fails (ActiveMQ node 1 killed) we retry it until the failover.
This is my sender class:
#RestController
public class RestApiController {
#Autowired
private JmsTemplate jmsTemplate;
#RequestMapping(value = "/produce")
public String produce() {
String result = "Done";
for (int i = 0; i < 5000; i++) {
boolean repetir = true;
while (repetir) {
try {
jmsTemplate.convertAndSend("TestTopic", "Message " + i);
repetir = false;
result = "Done";
} catch (JmsException e) {
e.printStackTrace();
result = "ERROR";
}
}
}
return result;
}
}
This is de sender's application.properties:
spring.activemq.broker-url=failover:(tcp://172.18.13.45:61616,tcp://172.18.13.45:61626)?initialReconnectDelay=1&backup=true spring.activemq.user=admin
spring.activemq.password=admin
spring.jms.pub-sub-domain=true
server.port=8081
CONSUMER:
This is my listener class:
#Service
public class ContactTransactionReceiver {
#JmsListener(destination = "TestTopic")
public void receiveMessageSendMessage(Message message) throws Exception {
System.out.println(((TextMessage) message).getText());
}
}
This is the consumer's application.properties:
spring.activemq.broker-url=failover:(tcp://172.18.13.45:61616,tcp://172.18.13.45:61626)?initialReconnectDelay=1&backup=true
spring.activemq.user=admin
spring.activemq.password=admin
spring.jms.pub-sub-domain=true
server.port=8082
ACTIVEMQ NODE 1
We have included this configuration in activemq.xml for HA, that refers to node2:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<networkConnectors>
<networkConnector uri="static:(tcp://172.18.13.45:61616,tcp://172.18.13.45:61626)" />
</networkConnectors>
We have proved too master-slave:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<networkConnectors>
<networkConnector uri="masterslave:(tcp://172.18.13.45:61616,tcp://172.18.13.45:61626)" />
</networkConnectors>
ACTIVEMQ NODE 2
We have included this configuration in activemq.xml for HA, that refers to node2:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<networkConnectors>
<networkConnector uri="static:(tcp://172.18.13.45:61626,tcp://172.18.13.45:61616)" />
</networkConnectors>
We have proved too master-slave:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<networkConnectors>
<networkConnector uri="masterslave:(tcp://172.18.13.45:61626,tcp://172.18.13.45:61616)" />
</networkConnectors>
You can find the full code and ActiveMQ configuration files in:
https://github.com/PedroRamirezTOR/ActiveMQ-HA-Sender-Consumer.git
Thanks in advance!

ActiveMQ 5.x does support high availability via a master-slave configuration. See the documentation for more details.

I noticed in your xmls that both master & slave are referring to a local KashaDB. I believe to have true HA Failover, you need to reference a shared network drive.
http://activemq.apache.org/shared-file-system-master-slave
<persistenceAdapter>
<kahaDB directory="/sharedFileSystem/sharedBrokerData"/>
</persistenceAdapter>

Related

activemq client not receiving the message from queue

We are trying to run JMS application using ActiveMQ, we are facing a problem with the message receiver. Please find below process used by the application and test.
A scheduler based JMS application starts
A persistent activemq test message is placed into a queue
The scheduled event tries to receive the message from the queue(this is not a consumer but it is just a receiver jmsTemplate.receive(queue))
Even though the message is there in the queue which can be browsed using activemq console, the application doesn't see it and says no message received in the queue. This issue happening even for a single input message, I have tried all the config options suggested over the support sites but nothing works. This is how my activemq.xml broker tag is now:
<?xml version="1.0" encoding="UTF-8"?>
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" persistent="true" dataDirectory="${activemq.data}" schedulePeriodForDestinationPurge="10000" deleteAllMessagesOnStartup="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" maxPageSize="5000" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000">
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000" />
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false" />
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb" />
</persistenceAdapter>
</broker>
Can someone please suggest anything that I am missing?
Edit:
Here is the receiver code
private javax.jms.Message getMessageFromQueue(String jmsCorrelationID, String queueName)
throws JMSException, EdiRoutingException, DcQueueNameMapperException {
String messageSelector = "JMSCorrelationID = '" + jmsCorrelationID + "'";
javax.jms.Message message = jmsTemplate.receiveSelected(queueName, messageSelector);
if(null != message) {
TextMessage textMessage = (TextMessage) message;
auditMessage("DATA", textMessage.getText());
}
return message;
}
Here is the producer code:
public void sendMessage(String queueName, final String jmsCorrelationId, final String message) {
jmsTemplate.send(queueName, session -> {
TextMessage textMessage = session.createTextMessage(message);
if (StringUtils.isNotBlank(jmsCorrelationId)) {
textMessage.setJMSCorrelationID(jmsCorrelationId);
}
return textMessage;
});
}

ava RabbitMQ client (amqp-client-3.6.5.jar) is blocking indefinitely on "Channel.Open" command

I am using Spring Integration AMQP 4.1.2, Spring Rabbit 1.4.3, Spring Amqp 1.4.3, amqp-client-3.6.5.jar to publish messages to RabbitMQ server 3.5.3
As part of negative testing, I am sending messages to Non Existing Exchange.
I have a negative acknowledgement handler configured using Spring Integration Amqp. This negative acknowledgement handler got invoked with the failed message and even this message contains the reason for negative acknowledgement.
Everything is perfect up to here.
I need to Retry the failed message again as part of requirement. So the negative acknowledgement handler retires to publish the same message again.
At this time, when the Java RabbitMQ client (amqp-client-3.6.5.jar) trying to issue the command "Channel.Open" to the RabbitMQ server.
But this call getting blocked indefinitely (AMQP Connection thread is indefinitely waiting on the Object BlockingValueOrException which is responsible to notify)
and the Java client is indefinitely waiting for the response to the command "Channel.Open". But I could see a new channel got created in RabbitMQ server using the admin console.
Why my "Channel.Open" call getting blocked? Is RabbitMQ server failed to send response to the command "Channel.Open"?
How to check the command requests and responses passed in between Java RabbitMQ client and RabbitMQ server? Do we have any plugins that need be installed in RabbitMQ server?
Please help me in this regard. Configuration information is below.
Spring Integration Amqp configuration that publishes messages and registers ack/nack, return handlers
<!-- AMQP/RMQ Publishing -->
<int-amqp:outbound-channel-adapter
default-delivery-mode="PERSISTENT" exchange-name="${prism.rabbitmq.exchange}"
routing-key-expression="headers['${prism.rabbitmq.message.header.routingKey}']" amqp-template="amqpTemplate"
mapped-request-headers="*" channel="outgoingRabbit"
confirm-ack-channel="successfullyPublishedChannel"
confirm-nack-channel="mailPublishingExceptionChannel"
confirm-correlation-expression="#this" lazy-connect="false" return-channel="mailPublishingExceptionChannel"/>
<!-- AMQP client connection factory -->
<bean id="amqpClientConnectionFactory" class="com.rabbitmq.client.ConnectionFactory">
<property name="uri"
value="amqp://guest:guest#127.0.0.1:5672" />
<property name="automaticRecoveryEnabled"
value="true" />
</bean>
<rabbit:connection-factory id="amqpConnectionFactory"
host="127.0.0.1" connection-factory="amqpClientConnectionFactory"
publisher-confirms="true" publisher-returns="true" channel-cache-size="5"/>
<rabbit:template id="amqpTemplate" connection-factory="amqpConnectionFactory"
exchange="${prism.rabbitmq.exchange}" retry-template="retryTemplate" mandatory="true"/>
<bean id="retryTemplate" class="org.springframework.retry.support.RetryTemplate">
<property name="retryPolicy">
<bean class="org.springframework.retry.policy.SimpleRetryPolicy">
<property name="maxAttempts" value="4" />
</bean>
</property>
<property name="backOffPolicy">
<bean class="org.springframework.retry.backoff.ExponentialBackOffPolicy">
<property name="initialInterval" value="1000" />
<property name="multiplier" value="5.0" />
<property name="maxInterval" value="60000" />
</bean>
</property>
</bean>
Negative Acknowledgement Handler configuration
<int:service-activator input-channel="mailPublishingExceptionChannel" ref="mailPublishingExceptionHandler" method="handleError" />
Negative Acknowledgement Handler class's handle method.
#Autowired
#Qualifier("outgoingRabbit")
private MessageChannel outgoingRabbit;
#Override
public void handleError(Message<?> genMessage) {
try {
// Retry !!
// Get the failed RMQ Message whose payload is JSON and has Message
// Headers as well.
Message failedRMQMessage = (Message) genMessage.getPayload();
MessageBuilder rmqMessageWithRetry = MessageBuilder.withPayload(failedRMQMessage.getPayload());
rmqMessageWithRetry.copyHeaders(failedRMQMessage.getHeaders());
new MessagingTemplate().send(outgoingRabbit, rmqMessageWithRetry.build()); --> this call again publishes the payload
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

Why is Sender not sending data to the queue in RabbitMQ?

I have implemented RabbitMQ and stuck in one place where it seems as if my sender is not able to send any data to the queue.
My Producer class:
#Service
public class MessageSender {
#Autowired
private AmqpTemplate template;
public void send(String text) {
template.convertAndSend(text);
}
}
my spring config file looks like this:
<rabbit:connection-factory id="connectionFactory"
addresses="${connectionFactory.addresses}" channel-cache-size="${connectionFactory.channel-cache-size}"
virtual-host="${connectionFactory.vhost}" username="${connectionFactory.user}"
password="${connectionFactory.password}" />
<rabbit:template id="amqpTemplate" connection-factory="connectionFactory" queue="myQueue" exchange="myExchange" routing-key="dummyKey"/>
<rabbit:queue name="myQueue" />
<rabbit:direct-exchange name="myExchange">
<rabbit:bindings>
<rabbit:binding queue="myQueue" />
</rabbit:bindings>
</rabbit:direct-exchange>
<rabbit:listener-container connection-factory="connectionFactory">
<rabbit:listener ref="messageHandler" method="onMessage" queue-names="myQueue" />
</rabbit:listener-container>
<rabbit:admin connection-factory="connectionFactory" />
<bean id="messageHandler" class="com.tm.email.sender.spring.MessageHandler" />
I cannot find out what the issue is.
The below mentioned thing works perfectly. I can easily push the messages on to the queue and then my sender class works perfectly fine.
public class Send {
private final static String QUEUE_NAME = "myQueue";
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
int i = 0;
while(i < 10000){
HashMap message = new HashMap();
message.put("message number", i);
channel.basicPublish("", QUEUE_NAME, null, SerializationUtils.serialize(message));
i++;
}
channel.close();
connection.close();
}
}
You have default routing-key="dummyKey" on the <rabbit:template> definition. That means that your template.convertAndSend(text); will send a Message to the myExchange through the dummyKey routing-key.
Since you don't provide the binding for that myQueue via dummyKey you messages just lost.
From the raw AMQP code you just send message to the default Exchange ("") using a default routing-key for each queue - its name.
For the listener it doesn't matter from where and how the message has appeared in the queue.
That's why the second 'send' variant works well.
So, to fix your use-case with AmqpTemplate, you have to add this:
<rabbit:binding queue="myQueue" key="dummyKey"/>

ActiveMQ failover seems not to work

I have super simple scenario: one broker and one consumer with durable subscription.
This is the code of my consumer app:
package test;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.Destination;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.Session;
import javax.jms.TextMessage;
import javax.jms.Topic;
import org.apache.activemq.ActiveMQConnectionFactory;
import pojo.Event;
import pojo.StockUpdate;
public class Consumer
{
private static transient ConnectionFactory factory;
private transient Connection connection;
private transient Session session;
public static int counter = 0;
public Consumer(String brokerURL) throws JMSException
{
factory = new ActiveMQConnectionFactory(brokerURL);
connection = factory.createConnection();
connection.setClientID("CLUSTER_CLIENT_1");
connection.start();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
}
public void close() throws JMSException
{
if (connection != null)
{
connection.close();
}
}
public static void main(String[] args) throws JMSException
{
try
{
// extract topics from the rest of arguments
String[] topics = new String[2];
topics[0] = "CSCO";
topics[1] = "ORCL";
// define connection URI
Consumer consumer = new Consumer("failover:(tcp://localhost:61616)?maxReconnectAttempts=-1&useExponentialBackOff=true");
for (String stock : topics)
{
try
{
Destination destination = consumer.getSession().createTopic("STOCKS." + stock);
// consumer.getSession().
MessageConsumer messageConsumer = consumer.getSession().createDurableSubscriber((Topic) destination, "STOCKS_DURABLE_CONSUMER_" + stock);
messageConsumer.setMessageListener(new Listener());
}
catch (JMSException e)
{
e.printStackTrace();
}
}
}
catch (Throwable t)
{
t.printStackTrace();
}
}
public Session getSession()
{
return session;
}
}
class Listener implements MessageListener
{
public void onMessage(Message message)
{
try
{
TextMessage textMessage = (TextMessage) message;
String json = textMessage.getText();
Event event = StockUpdate.fromJSON(json, StockUpdate.class);
System.out.println("Consumed message #:" + ++Consumer.counter + "\n" + event);
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
Here is my activemq.xml
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="R6_cluster_broker1" persistent="true">
<networkConnectors>
<networkConnector uri="static:(failover:(tcp://remote_master:61616,tcp://remote_slave:61617))"/>
</networkConnectors>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see:
http://activemq.apache.org/slow-consumer-handling.html
-->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see:
http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<kahaDB directory="/work/temp/kahadb"/>
</persistenceAdapter>
<!--
The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see:
http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<!-- <transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/> -->
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
<!--
Enable web consoles, REST and Ajax APIs and demos
The web consoles requires by default login, you can disable this in the jetty.xml file
Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/>
</beans>
When I have both broker and consumer running and then stop the broker my consumer exits few moments after. As far I understood it must attempt to reconnect, but it is not the case. What am I doing wrong, please advise.
!NOTE! I launch my consumer in Eclipse, i do not build a standalone jar for this task.
I have updated my broker to the latest 5.9.1 and did the same to my consumer. Result is the same - after I stop the broker my consumer dies few seconds after. It works fine if broker is up and running.
For anyone who can't setup the failover via the URI/URL parameter in the ConnectionFactory ,because of schema not found in ARTEMIS :
As far as I know in the ActiveMQ the URL is the following:
failover:(tcp://localhost:61616,tcp://localhost:51516)?randomize=false
But in Artemis the above will fail ,because of schema not found so remove just the failover: prefix:
(tcp://localhost:61616,tcp://localhost:51516)?randomize=false
Alright, the problem was actually in my code: there was nothing that would prevent main thread from exiting. Since thread that implements failover is a daemon thread, consumer app terminated right after there was nothing to hold on to (no non-daemon threads).
Most likely you are using a version of ActiveMQ that has a bug which causes there to be all daemon threads which means there's nothing to keep the client running. Upgrade to a later version such as v5.9.1 and see if that helps. If not post more information as you haven't really provided much.

ActiveMQ with JMS topic - some messages not dequeued by consumer

We're trying to set up ActiveMQ 5.9.0 as a message broker using JMS topics, but we're having some issues with the consumption of the messages.
For testing purposes, we have a simple configuration of 1 topic, 1 event producer, and 1 consumer. We send 10 messages one after the other, but every time we run the application, 1-3 of these messages are not consumed! The other messages are consumed and proceesed fine.
We can see that all the messages we're published to the topic in the ActiveMQ managment console, but they never reach the consumer, even if we reastart the application (we can see that the numbers in the "Enqueue" and "Dequeue" columns are different).
EDIT: I should also mention that when using queues instead of topic, this problem does not occur.
Why is this happening? Could it have something to do with atomikos (which is the transaction manger)? Or maybe something else in the configuration? Any ideas/suggestions are welcome. :)
This is the ActiveMQ/JMS spring configuration:
<bean id="connectionFactory" class="com.atomikos.jms.AtomikosConnectionFactoryBean"
init-method="init" destroy-method="close">
<property name="uniqueResourceName" value="amq" />
<property name="xaConnectionFactory">
<bean class="org.apache.activemq.spring.ActiveMQXAConnectionFactory"
p:brokerURL="${activemq_url}" />
</property>
<property name="maxPoolSize" value="10" />
<property name="localTransactionMode" value="false" />
</bean>
<bean id="cachedConnectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="connectionFactory" />
</bean>
<!-- A JmsTemplate instance that uses the cached connection and destination -->
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="cachedConnectionFactory" />
<property name="sessionTransacted" value="true" />
<property name="pubSubDomain" value="true"/>
</bean>
<bean id="testTopic" class="org.apache.activemq.command.ActiveMQTopic">
<constructor-arg value="test.topic" />
</bean>
<!-- The Spring message listener container configuration -->
<jms:listener-container destination-type="topic"
connection-factory="connectionFactory" transaction-manager="transactionManager"
acknowledge="transacted" concurrency="1">
<jms:listener destination="test.topic" ref="testReceiver"
method="receive" />
</jms:listener-container>
The producer:
#Component("producer")
public class EventProducer {
#Autowired
private JmsTemplate jmsTemplate;
#Transactional
public void produceEvent(String message) {
this.jmsTemplate.convertAndSend("test.topic", message);
}
}
The consumer:
#Component("testReceiver")
public class EventListener {
#Transactional
public void receive(String message) {
System.out.println(message);
}
}
The test:
#Autowired
private EventProducer eventProducer;
public void testMessages() {
for (int i = 1; i <= 10; i++) {
this.eventProducer.produceEvent("message" + i);
}
That's the nature of JMS topics - only current subscribers receive messages by default. You have a race condition and are sending messages before the consumer has established its subscription, after the container is started. This is a common mistake with unit/integration tests with topics where you are sending and receiving in the same application.
With newer versions of Spring, there is a method you can poll to wait until the subscriber is established (since 3.1, I think). Or, you can just wait a little while before starting to send, or you can make your subscriptions durable.