My Scenario: I publish two messages to my Rabbit broker, and an unhandled exception occurs while processing the first message.
My Question: Why does the message remain Unack'd in the broker and as a consequence why is the second message not be dequeued and processed?
Some info:
I am using Spring AMQP 1.5.4 with Spring Integration 4.2.4. (See code below)
I have a Dead Letter Exchange set up and it is working as expected (i.e. When I Nack a message, it is forwarded to the DLX where it expires. It is then forwarded to the main Exchange).
What I want:
I would like unhandled exceptions (i.e. exceptions that are caught by the SimpleMessageListenerContainer) to result in the amqp-message being Nack'd rather than remaining Unack'd.
What I see:
There are 3 retry attempts to process the message which of course fail because of my forced exception (see code below in ErrorHandler).
The consumer tag of the BlockingQueueConsumer is the same so I'm guessing that the BlockingQueueConsumer is not restarted. However, the logs below show that it does continue to wait for messages.
I would like to know why the BlockingQueueConsumer does not nack the message and why subsequent message are not consumed despite the evidence in the logs that the Consumer is waiting for messages.
Any suggestions or background info would be very welcome!
#Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer(ConnectionFactory connectionFactory, Queue mainQueue, RetryOperationsInterceptor retryOperationsInterceptor) {
SimpleMessageListenerContainer retVal = new SimpleMessageListenerContainer(connectionFactory);
retVal.addQueues(mainQueue);
retVal.setAcknowledgeMode(AcknowledgeMode.MANUAL);
retVal.setDefaultRequeueRejected(false);
retVal.setAdviceChain(new Advice[]{retryOperationsInterceptor});
return retVal;
}
#Bean
public RetryOperationsInterceptor retryOperationsInterceptor () {
return stateless().recoverer(new RejectAndDontRequeueRecoverer()).build();
}
<int-amqp:inbound-channel-adapter
channel="fromRabbitChannel"
error-channel="errorChannel"
listener-container="simpleMessageListenerContainer"
/>
<int:service-activator ref="errorHandler" input-channel="errorChannel" method="handleError"/>
#MessageEndpoint
public class ErrorHandler {
public void handleError(Message<MessagingException> message) throws IOException {
throw new IllegalStateException("FORCED EXCEPTION");
}
}
09:49:38.219 [SimpleAsyncTaskExecutor-1] INFO c.p.a.f.ErrorHandler - Throwing an exception!!
09:49:38.219 [SimpleAsyncTaskExecutor-1] DEBUG o.s.retry.support.RetryTemplate - Checking for rethrow: count=3
09:49:38.219 [SimpleAsyncTaskExecutor-1] DEBUG o.s.retry.support.RetryTemplate - Retry failed last attempt: count=3
09:49:38.220 [SimpleAsyncTaskExecutor-1] WARN o.s.a.r.r.RejectAndDontRequeueRecoverer - Retries exhausted for message (Body:'[B#c78ef32(byte[97])'MessageProperties [blah blah])
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener threw exception
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:865) [spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:760) [spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:680) [spring-rabbit-1.5.2.RELEASE.jar:na]
....
....
09:49:38.221 [SimpleAsyncTaskExecutor-1] WARN o.s.a.r.l.ConditionalRejectingErrorHandler - Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Retry Policy Exhausted
at org.springframework.amqp.rabbit.retry.RejectAndDontRequeueRecoverer.recover(RejectAndDontRequeueRecoverer.java:44) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.config.StatelessRetryOperationsInterceptorFactoryBean$1.recover(StatelessRetryOperationsInterceptorFactoryBean.java:59) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.amqp.rabbit.config.StatelessRetryOperationsInterceptorFactoryBean$1.recover(StatelessRetryOperationsInterceptorFactoryBean.java:53) ~[spring-rabbit-1.5.2.RELEASE.jar:na]
at org.springframework.retry.interceptor.RetryOperationsInterceptor$ItemRecovererCallback.recover(RetryOperationsInterceptor.java:124) ~[spring-retry-1.1.2.RELEASE.jar:na]
at org.springframework.retry.support.RetryTemplate.handleRetryExhausted(RetryTemplate.java:458) ~[spring-retry-1.1.2.RELEASE.jar:na]
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:320) ~[spring-retry-1.1.2.RELEASE.jar:na]
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:168) ~[spring-retry-1.1.2.RELEASE.jar:na]
....
....
09:49:38.222 [SimpleAsyncTaskExecutor-1] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-XVCBQNXxCMFERaF1kbeI3Q=debitCardStatusQueue}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5671/,1), acknowledgeMode=MANUAL local queue size=0
09:49:39.222 [SimpleAsyncTaskExecutor-1] DEBUG o.s.a.r.l.BlockingQueueConsumer - Retrieving delivery for Consumer: tags=[{amq.ctag-XVCBQNXxCMFERaF1kbeI3Q=debitCardStatusQueue}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5671/,1), acknowledgeMode=MANUAL local queue size=0
retVal.setAcknowledgeMode(AcknowledgeMode.MANUAL);
With manual acks, you are responsible to ack or reject the message; the container will only ack/nack if you set the mode to AUTO; it will then do exactly as you require.
Related
I noticed that when brokers are in a blocked state due to high watermark messages will not be accepted. However when they get unblocked the messages that are sent when the brokers were in a blocked state are accepted again (while the publisher is down, so they are not being republished).
Where are the messages kept? Is there a maximum amount of messages that can be kept like this, and how do I see how many? Is this behavior configurable?
I'm using a CachingConnectionFactory with a publisherConfirm in order to confirm messages are ack'd, but in this case it results in false information. The publisher confirm times out, but the broker eventually processes the message anyway.
The com.rabbitmq.client.impl.AMQChannel has a logic like this:
public void quiescingTransmit(AMQCommand c) throws IOException {
synchronized (_channelMutex) {
if (c.getMethod().hasContent()) {
while (_blockContent) {
try {
_channelMutex.wait();
} catch (InterruptedException ignored) {
Thread.currentThread().interrupt();
}
// This is to catch a situation when the thread wakes up during
// shutdown. Currently, no command that has content is allowed
// to send anything in a closing state.
ensureIsOpen();
}
}
this._trafficListener.write(c);
c.transmit(this);
}
}
Pay attention to that while (_blockContent) {, so technically a publishing thread is blocked over here and there is no any internal queues to buffer messages until it is unblocked. We just don't go anywhere else.
See more info in official RabbitMQ docs: https://www.rabbitmq.com/connection-blocked.html
And also see Spring AMQP docs: https://docs.spring.io/spring-amqp/docs/current/reference/html/#blocked-connections-and-resource-constraints
There is a behavior in RabbitMQ server that it will not accept subsequent connections / operations when it reaches to watermark value till the time it rebalances itself.
RabbitMQ client elegantly gets timeout when such situations happen after the connection timeout , But we are using Spring AMQP it continues to hang.
Steps to Reproduce
o Create a RabbitMQ HA cluster
o Create a simple program which produces and consumes message
a) Using Spring AMQP
b) Using RabbitMQ client
o Make RabbitMQ server reach high watermark value in memory so that it cannot accept any new connections or perform any operations say for 10 min
o Create Q, Send message from
a) Spring AMQP ( It will hang )
b) RabbitMQ client ( It will get timeout ) say after 1 min if connection timeout is being set as 1 min.
Spring Binaries Version
a) spring-rabbit-1.6.7.RELEASE.jar
b) spring-core-4.3.6.RELEASE.jar
c) spring-amqp-1.6.7.RELEASE.jar
We tried upgrading to Spring Rabbit and AMQP 2.0.2 version as well , But it didn’t helped.
You don't describe what your "RabbitMQ Client" is, but the java amqp-client uses classic Sockets by default. So you should get the same behavior with both (since Spring AMQP uses that client). Perhaps you are referring to some other language client.
With java Sockets, when the connection is blocked, the thread is "stuck" in socket write which is not interruptible, nor does it timeout.
To handle this condition, you have to use the 4.0 client or above and use NIO.
Here is an example application that demonstrates the technique.
#SpringBootApplication
public class So48699178Application {
private static Logger logger = LoggerFactory.getLogger(So48699178Application.class);
public static void main(String[] args) {
SpringApplication.run(So48699178Application.class, args);
}
#Bean
public ApplicationRunner runner(RabbitTemplate template, CachingConnectionFactory ccf) {
ConnectionFactory cf = ccf.getRabbitConnectionFactory();
NioParams nioParams = new NioParams();
nioParams.setWriteEnqueuingTimeoutInMs(20_000);
cf.setNioParams(nioParams);
cf.useNio();
return args -> {
Message message = MessageBuilder.withBody(new byte[100_000])
.andProperties(MessagePropertiesBuilder.newInstance()
.setDeliveryMode(MessageDeliveryMode.NON_PERSISTENT)
.build())
.build();
while (true) {
try {
template.send("foo", message);
}
catch (Exception e) {
logger.info(e.getMessage());
}
}
};
}
#Bean
public Queue foo() {
return new Queue("foo");
}
}
and
2018-02-09 12:00:29.803 INFO 9430 --- [ main] com.example.So48699178Application : java.io.IOException: Frame enqueuing failed
2018-02-09 12:00:49.803 INFO 9430 --- [ main] com.example.So48699178Application : java.io.IOException: Frame enqueuing failed
2018-02-09 12:01:09.807 INFO 9430 --- [ main] com.example.So48699178Application : java.io.IOException: Frame enqueuing failed
As I know ActiveMQ has a feature called AUTO Acknowledge that actually inform the broker that message has been received (not acknowledging the producer).
I want to know if it is possible to send acknowledgement from consumer to producer in ActiveMQ or RabbitMQ. then I want to handle the acknowledgment message in producer and if it wouldn't receive acknowledge then sending the message again to the consumer.
You want to perform a synchronous usecase over an asynchronous medium.
In RabbitMQ's case you can use RPC, as described here - https://www.rabbitmq.com/tutorials/tutorial-six-python.html
and
https://www.rabbitmq.com/direct-reply-to.html
Please notice that even authors advise to avoid it:
When in doubt avoid RPC. If you can, you should use an asynchronous pipeline - instead of RPC-like blocking, results are asynchronously pushed to a next computation stage.
RabbitMQ Java client provides auto-acking through com.rabbitmq.client.Channel.basicConsume.
At least for ActiveMQ - this is built in. You have to turn it on in activemq.xml
<policyEntry queue=">" advisoryForConsumed="true"/>
Simply listen the advisory topic for the queue you want to monitor consumed messages for. Then you can extract message id:s and what not to "tick off" outstanding requests.
For a complete end-to-end acknowledgement, I recommend something more custom. I.e. your producer-app should listen to some "response" queue that receives responses about the status of the produced message. I.e. if processing failed - you may want to know why etc..
Anyway, here is some code with a producer that also listens to acknowledgements from ActiveMQ.
public void run() throws Exception {
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("tcp://localhost:61616");
conn = cf.createConnection();
sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination dest = sess.createQueue("duck");
MessageConsumer mc = sess.createConsumer(AdvisorySupport.getMessageConsumedAdvisoryTopic(dest));
mc.setMessageListener(this);
conn.start();
MessageProducer mp = sess.createProducer(sess.createQueue("duck"));
mp.send(sess.createTextMessage("quack"));
}
public void onMessage(Message msg) {
try {
String msgId = msg.getStringProperty("orignalMessageId");
System.out.println("Msg: " + msgId + " consumed");
} catch ( Exception e) {
e.printStackTrace();
}
}
I am doing the following steps:
MessageListener receives the message from queue Q1
Validate the message
If validation fails, call channel.basicReject() and move it to dead letter queue
Else, lets say, email server fails. I call channel.basicReject() with requeue true and throw an exception. It goes to retry template and after maxAttempts, is recovered(RepublishMessageRecoverer) and goes to dead letter queue.
But it does not remove the message from Q1.
public void onMessage(Message message, Channel channel) throws Exception {
try {
validateMessage();
processMessage(message);
channel.basicAck(message.getMessageProperties().getDeliveryTag(),false);
}
catch (DataValidationException ex){
channel.basicReject(message.getMessageProperties().getDeliveryTag(),false);
}
catch(DownstreamAppException ex) {
channel.basicReject(message.getMessageProperties().getDeliveryTag(),true);
throw ex;
}
}
void validMessage() {
..
throw new DataValidationException();
}
void processMessage() {
...
throw new DownstreamAppException();
}
I do not want to requeue messages that failed validation, but want to requeue those that were not processed because of some downstream app failure for retries.
A couple of questions:
1. If I don't throw exception in catch of DownstreamAppException, message does not go throw retryTemplate and recoverer. Is it because requeuing a rejected message is a new message?
Why is the message not removed from Q1 ? and how can I fix it ?
Thanks
You are responsible for acking when using manual acks (regardless of retry). If your code never acks, the message will (eventually) be requeued; but you don't have access to the channel in the recoverer.
The real question is why are you using manual acks and a ChannelAwareMessageListener? Your use case is straightforward. Using AUTO ackmode, the container will ack the message on success and reject it on any exception.
Since the recoverer republishes the message, that is considered success and the message will be ack'd by the container.
To selectively retry/requeue, you will need a customized error handler see this answer for more information.
Currently I've set up my message listener container to use spring-retry for handling retries but when someone sends a message without giving a message-id, the message listener stops. Can this behaviour be changes so that it puts the message on the dead letter queue instead of stopping the listener ?
My configuration for retry is the following:
#Bean
public StatefulRetryOperationsInterceptor retryInterceptor() {
StatefulRetryOperationsInterceptorFactoryBean f = new
StatefulRetryOperationsInterceptorFactoryBean();
f.setRetryOperations(retryTemplate());
f.setMessageRecoverer(new RejectAndDontRequeueRecoverer());
return f.getObject();
}
private RetryOperations retryTemplate() {
RetryTemplate retryTemplate = new RetryTemplate();
ExponentialRandomBackOffPolicy backOffPolicy = new ExponentialRandomBackOffPolicy();
backOffPolicy.setInitialInterval(50);
backOffPolicy.setMultiplier(1.5);
backOffPolicy.setMaxInterval(1000);
retryTemplate.setBackOffPolicy(backOffPolicy);
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(10);
retryTemplate.setRetryPolicy(retryPolicy);
return retryTemplate;
}
and I get the following exception:
2014-08-01 08:50:27,858 [taskExecutor<OmittedForPrivacy>-2] WARN mqp.rabbit.listener.SimpleMessageListenerContainer - Execution of Rabbit message listener failed, and no ErrorHandler has been set.
org.springframework.amqp.rabbit.listener.FatalListenerExecutionException: Illegal null id in message. Failed to manage retry for message: (Body:'{
<Omitted for privacy>
}'; ID:null; Content:application/json; Headers:{__TypeId__=<OmittedForPrivacy>}; Exchange:; RoutingKey:<OmittedForPrivacy>; Reply:null; DeliveryMode:NON_PERSISTENT; DeliveryTag:1)
at org.springframework.amqp.rabbit.config.StatefulRetryOperationsInterceptorFactoryBean$3.getKey(StatefulRetryOperationsInterceptorFactoryBean.java:114) ~[spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.retry.interceptor.StatefulRetryOperationsInterceptor.invoke(StatefulRetryOperationsInterceptor.java:132) ~[spring-retry-1.1.0.RELEASE.jar:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.0.0.RELEASE.jar:4.0.0.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.0.0.RELEASE.jar:4.0.0.RELEASE]
at com.sun.proxy.$Proxy612.invokeListener(Unknown Source) ~[na:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:620) [spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:454) ~[spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:480) [spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:464) [spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$300(SimpleMessageListenerContainer.java:61) [spring-rabbit-1.2.1.RELEASE.jar:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:558) [spring-rabbit-1.2.1.RELEASE.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_17]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_17]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_17]
So I would like to change the fact that he stops the message listener container afterwards and instead just puts the message on the deadletter queue.
Krgds
What do you mean the listener "stops"; please show your config and post a DEBUG log someplace showing the behavior you describe.
EDIT:
Ah - the AmqpRejectAndDontRequeueRecover is to recover after listener exceptions; this exception occurs before we even get into the retry logic.
You can add a MissingMessageIdAdvice to the advice chain (before the retry advice).
It will let a message with a missing id be tried, but will throw a RejectAndDontRequeueException if it is redelivered (and fails again) - only one retry, regardless of the retry setting.
If you want to reject such messages without trying at all, you would have to create your own advice.