I am using camel-rabbitmq.
Here is my route defination
camelContext.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
from("rabbitmq:TEST?queue=TEST&concurrentConsumers=5")
.routeId("jms")
.autoStartup(false)
.throttle(10)
.asyncDelayed()
.log("Consuming message ${body} to ${header.deliveryAddress}")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
System.out.println(atomicLong.decrementAndGet());
}
})
;
}
});
When I push 500 messages to this queue , when stop and start route all message on channel will be lost ,wonder where they are going.
If I configure same route with &autoAck=false it is working properly but losing performance. Why camel not offering same behavior with and without autoAck.
I managed my problem doing following change in rabbitmqconsumer of camel-rabbitmq
public void handleCancelOk(String consumerTag) {
// no work to do
log.info("Received cancelOk signal on the rabbitMQ channel");
**downLatch.countDown();**
}
#Override
protected void doStop() throws Exception {
if (channel == null) {
return;
}
this.requeueChannel=openChannel(consumer.getConnection());
if (tag != null && isChannelOpen()) {
channel.basicCancel(tag);
}
stopping=true;
downLatch.await();
try {
lock.acquire();
if (isChannelOpen()) {
channel.close();
}
} catch (TimeoutException e) {
log.error("Timeout occured");
throw e;
} catch (InterruptedException e1) {
log.error("Thread Interrupted!");
} finally {
lock.release();
}
}
By doing this camel route will for message to consumed and avoided message loss.
You need to check rabbitmq consumer prefetch count
consumer prefetch
I think By default consumer picks all the messages in queue to its memory buffers.
If you set the prefetch count to 1, consumer will acknowledge messages one by one.
All the other unacknowledged will be present in the queue in ready state. Waiting to be picked up, after the consumer completes it task on the previous message picked.
Related
I have a scenario where I want to "pull" messages of a RabbitMQ queue/topic and process them one at a time.
Specifically if there are already messages sitting on the queue when the consumer starts up.
I have tried the following with no success (meaning, each of these options reads the queue until it is either empty or until another thread closes the context).
1.Stopping route immediately it is first processed
final CamelContext context = new DefaultCamelContext();
try {
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
RouteDefinition route = from("rabbitmq:harley?queue=IN&declare=false&autoDelete=false&hostname=localhost&portNumber=5672");
route.process(new Processor() {
Thread stopThread;
#Override
public void process(final Exchange exchange) throws Exception {
String name = exchange.getIn().getHeader(Exchange.FILE_NAME_ONLY, String.class);
String body = exchange.getIn().getBody(String.class);
// Doo some stuff
routeComplete[0] = true;
if (stopThread == null) {
stopThread = new Thread() {
#Override
public void run() {
try {
((DefaultCamelContext)exchange.getContext()).stopRoute("RabbitRoute");
} catch (Exception e) {}
}
};
}
stopThread.start();
}
});
}
});
context.start();
while(!routeComplete[0].booleanValue())
Thread.sleep(100);
context.stop();
}
Similar to 1 but using a latch rather than a while loop and sleep.
Using a PollingConsumer
final CamelContext context = new DefaultCamelContext();
context.start();
Endpoint re = context.getEndpoint(srcRoute);
re.start();
try {
PollingConsumer consumer = re.createPollingConsumer();
consumer.start();
Exchange exchange = consumer.receive();
String bb = exchange.getIn().getBody(String.class);
consumer.stop();
} catch(Exception e){
String mm = e.getMessage();
}
Using a ConsumerTemplate() - code similar to above.
I have also tried enabling preFetch and setting the max number of exchanges to 1.
None of these appear to work, if there are 3 messages on the queue, all are read before I am able to stop the route.
If I were to use the standard RabbitMQ Java API I would use a basicGet() call which lets me read a single message, but for other reasons I would prefer to use a Camel consumer.
Has anyone successfully been able to process a single message on a queue that holds multiple messages using a Camel RabbitMQ Consumer?
Thanks.
This is not the primary intention of the component as its for continued received. But I have created a ticket to look into supporting a basicGet (single receive). There is a new spring based rabbitmq component coming in 3.8 onwards so its going to be implemeneted there (first): https://issues.apache.org/jira/browse/CAMEL-16048
I am developing system where messages to be processed are pushed in ActiveMQ. I have stringent requirement that consumer must process messages in incoming sequence. If a message processing fails in consumer, it need to rollback/recover and keep on re-trying infinitely. Only when a message processing is successful, consumer need to commit and proceed to next message.
How to prevent rolled-back message auto-forwarding to DLQ and what's proper way of configuring re-delivery policy for such requirement?
when set RedeliveryPolicy re-trying infinitely like below the messages will never be sent to DLQ.
policy.setMaximumRedeliveries(RedeliveryPolicy.NO_MAXIMUM_REDELIVERIES);
with ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE you acknowledge messages one by one.
http://activemq.apache.org/redelivery-policy.html
import javax.jms.Connection;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.ActiveMQMessageConsumer;
import org.apache.activemq.ActiveMQSession;
import org.apache.activemq.RedeliveryPolicy;
public class SimpleConsumerIndividualAcknowledge {
public static void main(String[] args) throws JMSException {
Connection conn = null;
try {
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("tcp://localhost:61616");
RedeliveryPolicy policy = new RedeliveryPolicy();
policy.setMaximumRedeliveries(RedeliveryPolicy.NO_MAXIMUM_REDELIVERIES);
cf.setRedeliveryPolicy(policy);
conn = cf.createConnection();
ActiveMQSession session = (ActiveMQSession) conn.createSession(false,
ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE);
ActiveMQMessageConsumer consumer = (ActiveMQMessageConsumer) session
.createConsumer(session.createQueue("test"));
consumer.setMessageListener(new MessageListener() {
#Override
public void onMessage(Message message) {
try {
//do your stuff
message.acknowledge();
} catch (Exception e) {
throw new RuntimeException(e);//ActiveMQMessageConsumer.rollback() is called automatically
}
}
});
conn.start();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (conn != null) {
try {
conn.close();
} catch (Exception e) {
}
}
}
}
}
if you want to manually stop and restart consumer take a look here activemq-redelivery-does-not-work
I use SimpleMessageListenerContainer to consume message from rabbitMQ queue.
EVN:
rabbitmq:3.6.6
spring-rabbit:1.6.2
CODE:
#Bean
public SimpleMessageListenerContainer listenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory());
container.setQueues(new Queue("spring.demo"));
container.setMaxConcurrentConsumers(1);
container.setConcurrentConsumers(1);
container.setAcknowledgeMode(AcknowledgeMode.MANUAL);
container.setMessageListener(new ChannelAwareMessageListener() {
#Override
public void onMessage(Message message, Channel channel) throws Exception {
long deliveryTag = message.getMessageProperties().getDeliveryTag();
try {
byte[] body = message.getBody();
String result = new String(body);
logger.debug("result:" + result);
// doSomething(for a long time)
channel.basicAck(deliveryTag, false);
} catch (Exception e) {
e.printStackTrace();
channel.basicNack(deliveryTag, false, false);
}
}
});
return container;
}
There has 1 consumer to handle message. Now If a message will handle in a long time(5minutes), then other message can not to be consumed.
I want to set a timeout for each handle message. For example,set timeout is 30 seconds. If a Message handle over this value(30 seconds) then the consumer throw Exception to over handle. Thus this consumer can handle other message.
But now I do not know how to set this timeout?
We are trying to implement Retry mechanism on client exceptions. We want to be able to set different routing key, ttl and retry count based on the content in each message. We want to keep the handler simple, i.e; for handleMessage to throw exception. How do we handle this exception and send the message to DLX with appropriate parameters. On retry if the failure happens again - message would be discarded (acknowledged) , or will be put back on DLX with incrementing the retry count. where would we implement this logic and how would be wired?
========================
With Gary's direction, I was able to implement. Here are excerpts ..
#Bean
public SimpleMessageListenerContainer listenerContainer() {
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
jsonMessageHandler.setQueueName(queueName);
container.setQueueNames(queueName);
container.setMessageListener(jsonMessageListenerAdapter());
container.setAdviceChain(new Advice[]{retryOperationsInterceptor()});
return container;
}
#Bean
public MessageListenerAdapter messageListenerAdapter() {
return new MessageListenerAdapter(messageHandler,messageConverter);
}
#Bean
public MessageListenerAdapter jsonMessageListenerAdapter() {
return new MessageListenerAdapter(jsonMessageHandler);
}
#Bean
RetryOperationsInterceptor retryOperationsInterceptor() {
return RetryInterceptorBuilder.stateless().recoverer(republishMessageRecoverer).maxAttempts(1).build();
}
#Bean
RepublishMessageRecoverer republishMessageRecoverer() {
return new MyRepublishMessageRecoverer(rabbitTemplate());
}
==========
public class MyRepublishMessageRecoverer extends RepublishMessageRecoverer {
// - constructor
#Override
public void recover(Message message, Throwable cause) {
//Deal with headers
long currentCount = 0;
List xDeathList = (List)message.getMessageProperties().getHeaders().get("x-death");
if(xDeathList != null && xDeathList.size() > 0) {
currentCount = (Long)((Map)(xDeathList.get(0))).get("count");
}
if(currentCount < context.getRules().getNumberOfRetries()) {
//message sent to DLX
this.retryTemplate.send(handlerProperties.getSystem(), message);
} else {
//message ignored
}
throw new AmqpRejectAndDontRequeueException(cause);
}
}
You can't modify a rejected message, it is routed to the DLX/DLQ unchanged (except x-death headers are added by the broker).
You have to republish to the DLX/DLQ yourself if you want to change message properties.
You can use Spring Retry with a customized RepublishMessageRecoverer to do this.
Spring batch documentation recommends using the JmsItemReader, which is a wrapper around the JMSTemplate. However, I have discovered that the JMSTemplate has some issues - see http://activemq.apache.org/jmstemplate-gotchas.html .
This post came to my attention only because the queue was appearing to disappear before I could actually read the data of of it. The opportunity to miss messages seems like a fairly significant issue to me.
For consumers atleast try using DefaultMessageListenerContainer coupled with a SingleConnectionFactory or any such connection factory , it not need a scheduler to wake them up.there are log of examples explaining this , this one is really good in explaining stuff
http://bsnyderblog.blogspot.com/2010/05/tuning-jms-message-consumption-in.html
Here is the solution I ended up with. Since the query was about the "cleanest" way to listen to JMS from within a spring-batch step, I'm going to leave the question open for a while longer just in case there's a better way.
If someone can figure out why the code isn't formatting correctly, please let me know how to fix it.
1. In the a job listener, implement queue setup and teardown inside the beforeJob and afterJob events, respectively:
public void beforeJob(JobExecution jobExecution) {
try {
jobParameters = jobExecution.getJobParameters();
readerConnection = connectionFactory.createConnection();
readerConnection.start();
} catch (JMSException ex) {
// handle the exception as appropriate
}
}
public void afterJob(JobExecution jobExecution) {
try {
readerConnection.close();
} catch (JMSException e) {
// handle the exception as appropriate
}
}
2. In the reader, implement the StepListener and beforeStep / afterStep methods.
public void beforeStep(StepExecution stepExecution) {
this.stepExecution = stepExecution;
this.setJobExecution(stepExecution.getJobExecution());
try {
this.connection = jmsJobExecutionListener.getReaderConnection();
this.jmsSession = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
this.messageConsumer = jmsSession.createConsumer(jmsJobExecutionListener.getQueue());
}
catch (JMSException ex)
{
// handle the exception as appropriate
}
}
public ExitStatus afterStep(StepExecution stepExecution) {
try {
messageConsumer.close();
jmsSession.close();
} catch (JMSException e) {
// handle the exception as appropriate
}
return stepExecution.getExitStatus();
}
3. Implement the read() method:
public TreeModel<SelectedDataElementNode> read() throws Exception,
UnexpectedInputException, ParseException,
NonTransientResourceException {
Object result = null;
logger.debug("Attempting to receive message on connection: ", connection.toString());
ObjectMessage msg = (ObjectMessage) messageConsumer.receive();
logger.debug("Received: {}", msg.toString());
result = msg.getObject();
return result;
}
4. Add the listeners to the Spring Batch context as appropriate:
<batch:job id="doStuff">
<batch:listeners>
<batch:listener ref="jmsJobExecutionListener" />
</batch:listeners>
... snip ...
<batch:step id="step0003-do-stuff">
<batch:tasklet transaction-manager="jtaTransactionManager"
start-limit="100">
<batch:chunk reader="selectedDataJmsReader" writer="someWriter"
commit-interval="1" />
</batch:tasklet>
<batch:listeners>
<batch:listener ref="selectedDataJmsReader" />
</batch:listeners>
</batch:step>
</batch:job>