ActiveMQ with JMS topic - some messages not dequeued by consumer - activemq

We're trying to set up ActiveMQ 5.9.0 as a message broker using JMS topics, but we're having some issues with the consumption of the messages.
For testing purposes, we have a simple configuration of 1 topic, 1 event producer, and 1 consumer. We send 10 messages one after the other, but every time we run the application, 1-3 of these messages are not consumed! The other messages are consumed and proceesed fine.
We can see that all the messages we're published to the topic in the ActiveMQ managment console, but they never reach the consumer, even if we reastart the application (we can see that the numbers in the "Enqueue" and "Dequeue" columns are different).
EDIT: I should also mention that when using queues instead of topic, this problem does not occur.
Why is this happening? Could it have something to do with atomikos (which is the transaction manger)? Or maybe something else in the configuration? Any ideas/suggestions are welcome. :)
This is the ActiveMQ/JMS spring configuration:
<bean id="connectionFactory" class="com.atomikos.jms.AtomikosConnectionFactoryBean"
init-method="init" destroy-method="close">
<property name="uniqueResourceName" value="amq" />
<property name="xaConnectionFactory">
<bean class="org.apache.activemq.spring.ActiveMQXAConnectionFactory"
p:brokerURL="${activemq_url}" />
</property>
<property name="maxPoolSize" value="10" />
<property name="localTransactionMode" value="false" />
</bean>
<bean id="cachedConnectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="connectionFactory" />
</bean>
<!-- A JmsTemplate instance that uses the cached connection and destination -->
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="cachedConnectionFactory" />
<property name="sessionTransacted" value="true" />
<property name="pubSubDomain" value="true"/>
</bean>
<bean id="testTopic" class="org.apache.activemq.command.ActiveMQTopic">
<constructor-arg value="test.topic" />
</bean>
<!-- The Spring message listener container configuration -->
<jms:listener-container destination-type="topic"
connection-factory="connectionFactory" transaction-manager="transactionManager"
acknowledge="transacted" concurrency="1">
<jms:listener destination="test.topic" ref="testReceiver"
method="receive" />
</jms:listener-container>
The producer:
#Component("producer")
public class EventProducer {
#Autowired
private JmsTemplate jmsTemplate;
#Transactional
public void produceEvent(String message) {
this.jmsTemplate.convertAndSend("test.topic", message);
}
}
The consumer:
#Component("testReceiver")
public class EventListener {
#Transactional
public void receive(String message) {
System.out.println(message);
}
}
The test:
#Autowired
private EventProducer eventProducer;
public void testMessages() {
for (int i = 1; i <= 10; i++) {
this.eventProducer.produceEvent("message" + i);
}

That's the nature of JMS topics - only current subscribers receive messages by default. You have a race condition and are sending messages before the consumer has established its subscription, after the container is started. This is a common mistake with unit/integration tests with topics where you are sending and receiving in the same application.
With newer versions of Spring, there is a method you can poll to wait until the subscriber is established (since 3.1, I think). Or, you can just wait a little while before starting to send, or you can make your subscriptions durable.

Related

Number of connections and channels

I am new to Rabbitmq and Spring. I want to know how to manage the number of connections and channels.
In my architecture there are 2 queues where messages are published from single producer based on routing key on direct exchange. As per my understanding I would need a single connection with 2 channels which will be persistent and messages will be published through them. I assumed this is managed by Spring automatically. But a connection, consisting of single channel, is created every time a message is published.
- How do I manage the channels and connections? Is it the right approach to create a single channel for each queue in a connection? If the queue size increases to 10 then 10 channels should be used in a single connection?
Configuration File:
<bean id="connectionFactory" class="org.springframework.amqp.rabbit.connection.CachingConnectionFactory">
<property name="username" value="test"/>
<property name="password" value="test"/>
<property name="host" value="50.16.11.22"/>
<property name="port" value="5672"/>
</bean>
<bean id="publisher" class="com.test.code.Publisher">
<constructor-arg ref="amqpTemplate"></constructor-arg>
</bean>
<bean id="amqpTemplate" class="org.springframework.amqp.rabbit.core.RabbitTemplate">
<property name="connectionFactory" ref="connectionFactory"/>
<property name="mandatory" value="true"></property>
<property name="exchange" value="x.direct"></property>
</bean>
<rabbit:admin connection-factory="connectionFactory" />
<rabbit:queue name="q.queue1" />
<rabbit:queue name="q.queue2" />
<rabbit:direct-exchange name="x.direct">
<rabbit:bindings>
<rabbit:binding queue="q.queue1" key="key1" />
<rabbit:binding queue="q.queue2" key="key2" />
</rabbit:bindings>
</rabbit:direct-exchange>
</beans>
This is my Publisher class
public class Publisher {
public Publisher(RabbitTemplate rabbitTemplate) {
this.rabbitTemplate = rabbitTemplate;
}
public void messageToQueue1(JSONObject message) {
amqpTemplate.convertAndSend("key1", message.toString());
}
public void messageToQueue2(JSONObject message) {
amqpTemplate.convertAndSend("key2", message.toString());
}
}
But a connection, consisting of single channel, is created every time a message is published.
That is not true. There is also no dedicated channel for each routing key.
The CachingConnectionFactory maintains a single persistent connection (by default) and channels are cached.
The first publish creates a channel and puts it in the cache. The next publish gets it from the cache. Only if the cache is empty is a new channel created (and then you'll end up with 2 cached channels).
You'll only get as many channels as you need concurrently.

RabbitMQ listener stops listening messages when MessageListener throws exception first time

I am facing a unusual problem, after an exception is thrown during processing of first message, Spring Amqp MessageListener stops picking up any further messages.In my code I am not explicitly doing any resource locking. Can anyone please suggest what is the problem.Any help will be appreciated as I am falling short of ideas.
MQ configuraion:
<rabbit:admin id="rabbitAdmin" connection-factory="rabbitConnectionFactory" />
<rabbit:connection-factory
id="rabbitConnectionFactory"
host="${rabbitmq.host}"
port="${rabbitmq.port}"
username="${rabbitmq.username}"
password="${rabbitmq.password}"/>
<!--
Configure the rabbitTemplate helper class to simplify rabbitMQ
access (sending and receiving message).
The default Exchange is used here.
-->
<rabbit:template id="rabbitTemplate" connection-factory="rabbitConnectionFactory" message-converter="rabbitSimpleMessageConverter" retry-template="rabbitRetryTemplate" />
<bean id="rabbitRetryTemplate" class="org.springframework.retry.support.RetryTemplate">
<property name="backOffPolicy">
<bean class="org.springframework.retry.backoff.ExponentialBackOffPolicy">
<property name="initialInterval" value="500" />
<property name="multiplier" value="5" />
<property name="maxInterval" value="90000" />
</bean>
</property>
<property name="retryPolicy">
<bean class="org.springframework.retry.policy.SimpleRetryPolicy">
<property name="maxAttempts" value="5"/>
</bean>
</property>
</bean>
<bean id="rabbitRetryInterceptor" class="org.springframework.amqp.rabbit.config.StatefulRetryOperationsInterceptorFactoryBean">
<property name="messageRecoverer" ref="rejectAndDontRequeueRecoverer"/>
<property name="retryOperations" ref="rabbitRetryTemplate" />
</bean>
<!-- Consumers -->
<bean id="genericMessageConsumer" class="org.bla.GenericMessageConsumer" />
<rabbit:listener-container
connection-factory="rabbitConnectionFactory"
advice-chain="rabbitRetryInterceptor"
concurrency="${rabbitmq.concurrency}"
max-concurrency="${rabbitmq.maxconcurrency}"
acknowledge="auto" >
<rabbit:listener ref="genericMessageConsumer" queue-names="${rabbitmq.orion.genericNotificationdata.queueName}" />
</rabbit:listener-container>
MessageListener class
public class GenericMessageConsumer implements MessageListener{
....
#Override
public void onMessage(Message message) {
try{
...Some logic...
} catch (Exception e) {
throw new RuntimeException(e.getMessage(), e);
}
After exception is thrown while processing first message, The container remains up but the remaining messages are not processed until i restart the container. Also the first messages stays in unacknowledged state.
Tried checking other posts same as my issue:
1) RabbitMQ listener stops listening messages when MessageListener throws exception
but am unable to figure out the solution.

ava RabbitMQ client (amqp-client-3.6.5.jar) is blocking indefinitely on "Channel.Open" command

I am using Spring Integration AMQP 4.1.2, Spring Rabbit 1.4.3, Spring Amqp 1.4.3, amqp-client-3.6.5.jar to publish messages to RabbitMQ server 3.5.3
As part of negative testing, I am sending messages to Non Existing Exchange.
I have a negative acknowledgement handler configured using Spring Integration Amqp. This negative acknowledgement handler got invoked with the failed message and even this message contains the reason for negative acknowledgement.
Everything is perfect up to here.
I need to Retry the failed message again as part of requirement. So the negative acknowledgement handler retires to publish the same message again.
At this time, when the Java RabbitMQ client (amqp-client-3.6.5.jar) trying to issue the command "Channel.Open" to the RabbitMQ server.
But this call getting blocked indefinitely (AMQP Connection thread is indefinitely waiting on the Object BlockingValueOrException which is responsible to notify)
and the Java client is indefinitely waiting for the response to the command "Channel.Open". But I could see a new channel got created in RabbitMQ server using the admin console.
Why my "Channel.Open" call getting blocked? Is RabbitMQ server failed to send response to the command "Channel.Open"?
How to check the command requests and responses passed in between Java RabbitMQ client and RabbitMQ server? Do we have any plugins that need be installed in RabbitMQ server?
Please help me in this regard. Configuration information is below.
Spring Integration Amqp configuration that publishes messages and registers ack/nack, return handlers
<!-- AMQP/RMQ Publishing -->
<int-amqp:outbound-channel-adapter
default-delivery-mode="PERSISTENT" exchange-name="${prism.rabbitmq.exchange}"
routing-key-expression="headers['${prism.rabbitmq.message.header.routingKey}']" amqp-template="amqpTemplate"
mapped-request-headers="*" channel="outgoingRabbit"
confirm-ack-channel="successfullyPublishedChannel"
confirm-nack-channel="mailPublishingExceptionChannel"
confirm-correlation-expression="#this" lazy-connect="false" return-channel="mailPublishingExceptionChannel"/>
<!-- AMQP client connection factory -->
<bean id="amqpClientConnectionFactory" class="com.rabbitmq.client.ConnectionFactory">
<property name="uri"
value="amqp://guest:guest#127.0.0.1:5672" />
<property name="automaticRecoveryEnabled"
value="true" />
</bean>
<rabbit:connection-factory id="amqpConnectionFactory"
host="127.0.0.1" connection-factory="amqpClientConnectionFactory"
publisher-confirms="true" publisher-returns="true" channel-cache-size="5"/>
<rabbit:template id="amqpTemplate" connection-factory="amqpConnectionFactory"
exchange="${prism.rabbitmq.exchange}" retry-template="retryTemplate" mandatory="true"/>
<bean id="retryTemplate" class="org.springframework.retry.support.RetryTemplate">
<property name="retryPolicy">
<bean class="org.springframework.retry.policy.SimpleRetryPolicy">
<property name="maxAttempts" value="4" />
</bean>
</property>
<property name="backOffPolicy">
<bean class="org.springframework.retry.backoff.ExponentialBackOffPolicy">
<property name="initialInterval" value="1000" />
<property name="multiplier" value="5.0" />
<property name="maxInterval" value="60000" />
</bean>
</property>
</bean>
Negative Acknowledgement Handler configuration
<int:service-activator input-channel="mailPublishingExceptionChannel" ref="mailPublishingExceptionHandler" method="handleError" />
Negative Acknowledgement Handler class's handle method.
#Autowired
#Qualifier("outgoingRabbit")
private MessageChannel outgoingRabbit;
#Override
public void handleError(Message<?> genMessage) {
try {
// Retry !!
// Get the failed RMQ Message whose payload is JSON and has Message
// Headers as well.
Message failedRMQMessage = (Message) genMessage.getPayload();
MessageBuilder rmqMessageWithRetry = MessageBuilder.withPayload(failedRMQMessage.getPayload());
rmqMessageWithRetry.copyHeaders(failedRMQMessage.getHeaders());
new MessagingTemplate().send(outgoingRabbit, rmqMessageWithRetry.build()); --> this call again publishes the payload
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

How to configure simple RabbitMQ message producer with Spring Integration

I'd like to configure a simple RabbitMQ message producer using Spring Integration constructs. The requirement is very basic: just a simple fire-and-forget, sending an event message to a queue, no response required.
I've configured the connection factory, RabbitTemplate and outbound channel adapter (see below), but missing the last piece: the code that actually sends the message out to the channel.
Thanks in advance.
<rabbit:connection-factory id="producerRabbitConnectionFactory"
channel-cache-size="${amqp.channel.cache.size}"
host="${amqp.hostname}"
port="${amqp.port}"
virtual-host="${amqp.vhost}"
username="${amqp.username}"
password="${amqp.password}"
requested-heartbeat="${amqp.heartbeat}"
/>
<bean id="producerRabbitTemplate" class="org.springframework.amqp.rabbit.core.RabbitTemplate">
<property name="connectionFactory" ref="producerRabbitConnectionFactory" />
<property name="exchange" value="${amqp.exchange.event}" />
<property name="routingKey" value="${amqp.routingKey.event}" />
</bean>
<int:channel id="outboundAmqpChannel" />
<int-amqp:outbound-channel-adapter id="outboundAmqpChannelAdapter"
channel="outboundAmqpChannel"
amqp-template="producerRabbitTemplate"
default-delivery-mode="NON_PERSISTENT"
lazy-connect="true"/>
The simplest is a Messaging Gateway. That way your code doesn't know your talking to an integration flow.
public interface Foo {
void bar(String foo);
}
<int:gateway service-interface="foo.Foo" default-request-channel="outboundAmqpChannel" />
Inject a Foo into your code and call it.

Receiving RabbitMQ message from one channel, set the messageId in the transformer and send it to other channel using Spring Integration

I'm new to RabbitMQ and Spring Integration.
I have a use case to consume JSON message from a channel, convert it to an object. One of the field that I need to set in the object is the message Id(delivery.getEnvelope().getDeliveryTag()) of the message that we receive from rabbitMQ which we need for ack handling after all the business logic.
How to do it using spring integration?
Here is my xml configuration.
<bean id="devRabbitmqConnectionFactory" class="com.rabbitmq.client.ConnectionFactory">
<property name="brokerURL" value="#{props[rabbitmq_inputjms_url]}" />
<property name="redeliveryPolicy" ref="redeliveryPolicy" />
</bean>
<bean id="devJMSCachingConnectionFactory"
class="org.springframework.amqp.rabbit.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="devRabbitmqConnectionFactory" />
<property name="sessionCacheSize" value="10" />
<property name="cacheProducers" value="false" />
</bean>
<int-jms:channel id="devJMSChannel" acknowledge="transacted"
connection-factory="devJMSCachingConnectionFactory" message-driven="false"
queue-name="devJMSChannel">
</int-jms:channel>
<bean id="redeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<property name="initialRedeliveryDelay" value="5000" />
<property name="maximumRedeliveries" value="5" />
</bean>
<int:transformer id="devObjectTransformer" input-channel="devJMSChannel" ref="devService" method="readEventFromRabbitMQ"
output-channel="devPacketChannel">
<int:poller fixed-rate="10" task-executor="devObjectTransformerExecutor" />
</int:transformer>
The transformer method "readEventFromRabbitMQ" gets the message String from msg.getPayload() converts it into object and sends it to the output channel. But not sure how to get the message Id in the transformer class. Can somebody help me with this?
public List<DevEventRecord> readEventFromRabbitMQ(Message<EventsDetail> msg){
DevEventRecord[] eventRecords=null;
EventsDetail expEvent = null;
long receivedTime =System.currentTimeMillis();
int packetId = -1;
try{
monitorBean.incrementDeviceExceptionPacketCount();
expEvent = msg.getPayload();
LogUtil.debug("readExceptionEvent :: consumed JMS Q "+expEvent);
eventRecords = dispatchPacket(expEvent);
}
catch(ProcessingException pe){
notifyAck(expEvent.getUniqueId(),,,,);
}
catch(Exception ex){
notifyAck(expEvent.getUniqueId(),,,,);
LogUtil.error("Exception occured while reading object in readEvent , "+ex.toString());
}
return getEventRecordList(eventRecords);
}
The deliveryTag is presented as message header after an <int-amqp:inbound-channel-adapter> under the key AmqpHeaders.DELIVERY_TAG.
I don't understand why you mix AMQP and JMS, but anyway those channel implementations don't populate headers from received message. It is out of their responcibity.
Please, use <int-amqp:inbound-channel-adapter> and here is a sample how to ack message manually using deliveryTag header.