I understand the difference between Dead Letter Queue and Poison Queue. I will be speaking in the context of transactions and durability.
DLQ: Client sends message. MSMQ is not able to deliver the message. Message remains in queue, waiting to be sent to the destination queue.
PQ: Message is successfully delivered. WCF service processes the message. Some error occured during the WCF operations. Message is placed in the poison queue.
(please correct me if I am wrong above)
Now, if the WCF Service encounters an error, and with the transaction scope in place (attribute), then how does it place the message in the poison queue or does this have to be developed? if it needs to be developed then how is it done?
What about processing poison messages? Is there a way to place them back in the queue for processing again? How is it determined if the message is poison? For example, an operation in WCF may encounter an error but can recover from it.
You can read more about Poison Message Handling on MSDN More about the MSMQ on MSDN - How to: Exchange Messages with WCF Endpoints and Message Queuing Applications
"When the service reads messages from the target queue under a transaction, the service may fail to process the message for various reasons. The message is then put back into the queue to be read again. To deal with messages that fail repeatedly, a set of poison-message handling properties can be configured in the binding. There are four properties: ReceiveRetryCount, MaxRetryCycles, RetryCycleDelay, and ReceiveErrorHandling. "
Only Message Queuing places messages in dead-letter queues. Applications can only read and delete messages in dead-letter queues.
An answer from https://social.msdn.microsoft.com/Forums/vstudio/en-US/84a0a601-1e0a-4693-b5f7-868d3eacb43a/what-is-the-difference-between-dead-letter-queue-and-poison-queue-in-msmq-?forum=wcf
A message in the Poison Queue is a message that has exceeded the
maximum number of delivery attempts to the application. These poison
messages are placed in teh Poison Message Queue. Items in the queue
still need to be read and handled though.
The Dead Letter Queue is for messages that will not be handled at all
and no processing will be done on them.
It is common to move items from the Poison Message Queue to the Dead
Letter Queue.
Related
Question one: Can I subscribe to the event of a message being sent to the _skipped queue?
I am using masstransit together with rabbit mq. Some messages sometimes are sent to the _skipped queue for unclear reasons. The message type has a consumer, the ttl (time to life) is not small. It should not happen, and I am getting a log entry from masstransit, but I want to do more at the moment. Maybe log an error, in test maybe pop-up a window. Is there a way to achieve this? I am only getting these log messages below.
MassTransit.ReceiveTransport|SKIP rabbitmq://localhost/services_admin db270000-1fd6-00ff-3b83-08d9000ef97c
MassTransit.ReceiveTransport|Declare queue: name: services_admin_skipped, durable, consumer-count: 0 message-count: 3
Question two: What exactly happens to messages in the _skipped queue? Can they be resent?
Skipped messages either don't match the type (namespace included), don't have a consumer on the endpoint, or were a response to a request client that is no longer waiting for it. Since it's a receive endpoint queue, it's likely one of the first two reasons. Look at the message body/details in the RabbitMQ Management Console, that should give you some ideas.
You can use a shovel in RabbitMQ to move the messages back into the queue once you've resolved the issue.
I'm in a phase of learning RabbitMQ/AMQP from the RabbitMQ documentation. Something that is not clear to me that I wanted to ask those who have hands-on experience.
I want to have multiple consumers listening to the same queue in order to balance the work load. What I need is pretty much close to the "Work Queues" example in the RabbitMQ tutorial.
I want the consumer to acknowledge message explicitly after it finishes handling it to preserve the message and delegate it to another consumer in case of crash. Handling a message may take a while.
My question is whether AMQP postpones next message processing until the previous message is ack'ed? If so how do I achieve load balancing between multiple workers and guarantee no messages get lost?
No, the other consumers don't get blocked. Other messages will get delivered even if they have unacknowledged but delivered predecessors. If a channel closes while holding unacknowledged messages, those messages get returned to the queue.
See RabbitMQ Broker Semantics
Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages.
EDIT In response to your comment:
Time to dive a little deeper into the AMQP specification then perhaps:
3.1.4 Message Queues
A message queue is a named FIFO buffer that holds message on behalf of a set of consumer applications.
Applications can freely create, share, use, and destroy message queues, within the limits of their authority.
Note that in the presence of multiple readers from a queue, or client transactions, or use of priority fields,
or use of message selectors, or implementation-specific delivery optimisations the queue MAY NOT
exhibit true FIFO characteristics. The only way to guarantee FIFO is to have just one consumer connected
to a queue. The queue may be described as “weak-FIFO” in these cases. [...]
3.1.8 Acknowledgements
An acknowledgement is a formal signal from the client application to a message queue that it has
successfully processed a message.[...]
So acknowledgement confirms processing, not receipt. The broker will hold on to the message until it's gotten acknowleged, so that it can redeliver them. But it is free to deliver more messages to consumers even if the prededing messages have not yet been acknowledged. The consumers will not be blocked.
Is there a good best practice about what kind of messages an application is allowed to reject?
My understanding is that all messages which can't be handled should be rejected to the dead letter queue - no matter if the problem is a syntax error or a semantic error in the message or if the application is temporarily not able to handle the message (for instance because the db just went down).
Of course - if the app already knows upfront that it will not be able to handle a message (DB down), it should stop accepting messages.
So what's the common understanding / best practice?
My response is with respect to WebSphere MQ:
A Dead Letter Queue (DLQ for short) is a place where messages that could not be delivered to their destination are put. Messages can be put on the DLQ by queue managers, message channel agents (MCAs), and applications. All messages on the DLQ must be prefixed with a dead-letter header structure, MQDLH. The MQDLH header is automatically fixed when queue manager or MCAs put messages whereas applications must prefix the MQDLH explicitly.
As far applications are concerned, if they are unable to handle the message, say for example the message format is not understood, they can put the message to a BACKOUT queue instead of a DLQ. A BACKOUT queue is just like any normal queue where messages rejected by applications can be put. The advantage of BACKOUT queue is that you can specify a BACKOUT queue on a per queue basis and the messages put there need not have MQDLH header prefixed.
An application can be written to read the messages from BACKOUT and route them back to the target queue as it is. However the messages in a DLQ require additional processing to remove the MQDLH before they are put onto a target queue.
Does anyone know if the pop operation on a RabbitMQ queue is atomic?
I have several processes reading from the same queue (the queue is marked as durable, running on version 2.0.0) and I am seeing some quite odd behaviour.
If your multiple processes are consuming messages from the same queue then they should never consume the same message.
Here are the caveats, though:
If a message has been delivered by the broker to one of your consumers and it rejects the message (or terminates before getting a chance to acknowledge it) then the broker will put it back on the same queue and it would be delivered to one of your remaining active consumers.
If your consumers are pulling from distinct queues -- each with a matching binding -- then the broker will put copies of the message on each queue and each consumer will get a copy of the same message.
I am running a WCF client that invokes a WCF service via an MsmqBinding.
Framework is .Net 4.0, client and server runs on Windows Server 2008 R2.
The channel queue is transactional.
The service is hosted with these binding parameters:
receiveErrorHandling="Move"
receiveRetryCount="3"
retryCycleDelay="00:00:20"
maxRetryCycles="5"
Given that ((ReceiveRetryCount+1) * (MaxRetryCycles + 1)) is in effect, this will result in
4*6 = 24 retries of any given message before it is moved to the poison subqueue.
Attaching an IErrorHandler to my service I notice that HandleError is called with a MsmqPoisonMessageException a total of 6 times (for a poison message), before the wcf subsystem finally moves the message to the ;poison subqueue.
I want to log the precise time when a message is done being retried and the message is moved to the poison queue. It seems to me the only option is to count the number of times a certain message faults and compare this count with the binding MaxRetryCycles. This is awkward and errorprone.
My question is:
Is there any way for me to
conclusively detect the event where
the wcf subsystem moves the message
to the poison queue?
My references are:
http://msdn.microsoft.com/en-us/library/aa395218.aspx
And: http://consultingblogs.emc.com/simonevans/archive/2007/09/17/A-comprehensive-guide-to-using-MsmqIntegrationBinding-with-MSMQ-3.0-in-WCF.aspx
The number of retries is of course an outcome of your parameters; however in your IErrorHandler you can explicitly move the message to the poison queue yourself. Otherwise, it will always move based on your binding parameters, and would be detected by listening to the poison queue like any other queue.
There are a number of good monitoring solutions that you can use to watch message queues for the arrival of a message. MonitorWang is an open source one that can detect when a message has arrived in a poison message or error queue. Detecting when a message has been received in the error queue is more reliable than trying to detect when a message has been sent to the error queue.