Right now I'm working with a JMS queue that has a redelivery policy such that messages that are rolled back more than five times are sent to a dead letter queue.
When a message is sent to a Dead Letter Queue in ActiveMQ, is there a way to see what exception caused that message rollback? Is there a message property I can set to specify what caused the message failure?
If you use the dead letter channel functionality
http://activemq.apache.org/message-redelivery-and-dlq-handling.html
of the AMQ broker then its as answered above, literally just the message that couldn't be delivered. The broker does not have any knowledge what went wrong, it just gets a 'rollback' signal on the message.
If you use Camel's dead letter channel functionality
http://camel.apache.org/dead-letter-channel.html
then Camel stores the caused exception as a property on the Exchange, which you can enrich on the failed message, and send it to a dead letter queue, in the AMQ broker. So instead of letting AMQ rollback the message, you handle the exception, and send the message to another queue, which happens to be a dead letter queue. You can then enrich the message by taking the stacktrace and/or exception message from the Exchange property and store somewhere on the message, in a header etc.
Camel stores the caught exception as a property on the Exchange with the key: Exchange.EXCEPTION_CAUGHT
You'll need to look in the logs on the box that's generating the exception. What's sent to the DLQ is literally just the message that couldn't be delivered.
Related
Question one: Can I subscribe to the event of a message being sent to the _skipped queue?
I am using masstransit together with rabbit mq. Some messages sometimes are sent to the _skipped queue for unclear reasons. The message type has a consumer, the ttl (time to life) is not small. It should not happen, and I am getting a log entry from masstransit, but I want to do more at the moment. Maybe log an error, in test maybe pop-up a window. Is there a way to achieve this? I am only getting these log messages below.
MassTransit.ReceiveTransport|SKIP rabbitmq://localhost/services_admin db270000-1fd6-00ff-3b83-08d9000ef97c
MassTransit.ReceiveTransport|Declare queue: name: services_admin_skipped, durable, consumer-count: 0 message-count: 3
Question two: What exactly happens to messages in the _skipped queue? Can they be resent?
Skipped messages either don't match the type (namespace included), don't have a consumer on the endpoint, or were a response to a request client that is no longer waiting for it. Since it's a receive endpoint queue, it's likely one of the first two reasons. Look at the message body/details in the RabbitMQ Management Console, that should give you some ideas.
You can use a shovel in RabbitMQ to move the messages back into the queue once you've resolved the issue.
I am using spring cloud stream with rabbitMQ.
I am trying to do the negative test by deleting the queue on runtime and post the message to the deleted queue.
source.queue2Source().send(MessageBuilder.withPayload(queue4DTO).build());
I am listening message from queue 1 and posting the message to queue 2( deleted one). I was expecting that above code will throw an exception but it is not. Even the message from reading from queue 1 has been acknowledged. I have a dead letter queue on queue 1 and queue 2 but message not went into dlq.
That is because the you are sending message to a channel defined by the Source. The channel actually does exist and is bridged via AMQPOutboundChannelAdapter to an exchange which by default will drop undeliverable messages.
i am using Spring JmsTemplate for sending/pushing message to ActiveMQ queue.
i know after listener start processing that message and any exception occur,that message will be considered as a poison pill and moved to default DLQ of acive mq.
is that same possible (i.e. that message will move to default Dead Letter Queue ) while sending/pushing that message to Queue if there is any connection error or any other error occured while sending?
for example -
public void push(){
jmsTemplate.send(test.Queue, "This message is from client");
//if any exception occur here,i want this message to be placed
in DLQ
}
please suggest is it possible?
If you can't connect to send it the main queue, you probably can't connect to send it to the DLQ either.
I am throwing an AmqpException inside of my consumer.
My expectation is that the message will return back to the queue in FIFO order and will be reprocessed sometime in the future.
It seems as if Spring AMQP does not release the message back to the queue. But instead tries to reprocess the failed messages over and over again.
This blocks the newly arrived messages from being processed. The ones that are stuck appear in the "unpacked" state forever inside of the AMQP console.
Any thoughts?
That's the way rabbitmq/Spring AMQP works; if a message is rejected (any exception is thrown) the message is requeued by default and is put back at the head of the queue so it is retried immediately.
... reprocessed sometime in the future.
You have to configure things appropriately to make that happen.
First, you have to tell the broker to NOT requeue the message. That is done by setting defaultRequeueRejected on the listener container to false (it's true by default). Or, you can throw an AmqpRejectAndDontRequeueException which instructs the container to reject (and not requeue) an individual message.
But that's not the end of it; just doing that will simply cause the rejected message to be discarded.
To avoid that, you have to set up a Dead Letter Exchange/Queue for the queue - rejected messages are then sent to the DLX/DLQ instead of being discarded. Using a policy rather than queue arguments is generally recommended.
Finally, you can set a message time to live on the the DLQ so, after that time, the message is removed from the queue. If you set up an another appropriate dead letter exchange on that queue (the DLQ), you can cause the message to be requeued back to the original queue after the time expires.
Note that this will only work for rejected deliveries from the original queue; it will not work when expiring messages in that queue.
See this answer and some of the links from its question for more details.
You can use the contents of the x-death header to decide if you should give up completely after some number of attempts (catch the exception and somehow dispose of the bad message; don't thrown an exception and the container will ack the message).
Here is a solution I used to solve this. I setup an Interceptor to retry the message x number of times while applying a backoff policy.
http://trippstech.blogspot.com/2016/03/rabbitmq-deadletter-queue-with.html
I understand the difference between Dead Letter Queue and Poison Queue. I will be speaking in the context of transactions and durability.
DLQ: Client sends message. MSMQ is not able to deliver the message. Message remains in queue, waiting to be sent to the destination queue.
PQ: Message is successfully delivered. WCF service processes the message. Some error occured during the WCF operations. Message is placed in the poison queue.
(please correct me if I am wrong above)
Now, if the WCF Service encounters an error, and with the transaction scope in place (attribute), then how does it place the message in the poison queue or does this have to be developed? if it needs to be developed then how is it done?
What about processing poison messages? Is there a way to place them back in the queue for processing again? How is it determined if the message is poison? For example, an operation in WCF may encounter an error but can recover from it.
You can read more about Poison Message Handling on MSDN More about the MSMQ on MSDN - How to: Exchange Messages with WCF Endpoints and Message Queuing Applications
"When the service reads messages from the target queue under a transaction, the service may fail to process the message for various reasons. The message is then put back into the queue to be read again. To deal with messages that fail repeatedly, a set of poison-message handling properties can be configured in the binding. There are four properties: ReceiveRetryCount, MaxRetryCycles, RetryCycleDelay, and ReceiveErrorHandling. "
Only Message Queuing places messages in dead-letter queues. Applications can only read and delete messages in dead-letter queues.
An answer from https://social.msdn.microsoft.com/Forums/vstudio/en-US/84a0a601-1e0a-4693-b5f7-868d3eacb43a/what-is-the-difference-between-dead-letter-queue-and-poison-queue-in-msmq-?forum=wcf
A message in the Poison Queue is a message that has exceeded the
maximum number of delivery attempts to the application. These poison
messages are placed in teh Poison Message Queue. Items in the queue
still need to be read and handled though.
The Dead Letter Queue is for messages that will not be handled at all
and no processing will be done on them.
It is common to move items from the Poison Message Queue to the Dead
Letter Queue.