I am running a WCF client that invokes a WCF service via an MsmqBinding.
Framework is .Net 4.0, client and server runs on Windows Server 2008 R2.
The channel queue is transactional.
The service is hosted with these binding parameters:
receiveErrorHandling="Move"
receiveRetryCount="3"
retryCycleDelay="00:00:20"
maxRetryCycles="5"
Given that ((ReceiveRetryCount+1) * (MaxRetryCycles + 1)) is in effect, this will result in
4*6 = 24 retries of any given message before it is moved to the poison subqueue.
Attaching an IErrorHandler to my service I notice that HandleError is called with a MsmqPoisonMessageException a total of 6 times (for a poison message), before the wcf subsystem finally moves the message to the ;poison subqueue.
I want to log the precise time when a message is done being retried and the message is moved to the poison queue. It seems to me the only option is to count the number of times a certain message faults and compare this count with the binding MaxRetryCycles. This is awkward and errorprone.
My question is:
Is there any way for me to
conclusively detect the event where
the wcf subsystem moves the message
to the poison queue?
My references are:
http://msdn.microsoft.com/en-us/library/aa395218.aspx
And: http://consultingblogs.emc.com/simonevans/archive/2007/09/17/A-comprehensive-guide-to-using-MsmqIntegrationBinding-with-MSMQ-3.0-in-WCF.aspx
The number of retries is of course an outcome of your parameters; however in your IErrorHandler you can explicitly move the message to the poison queue yourself. Otherwise, it will always move based on your binding parameters, and would be detected by listening to the poison queue like any other queue.
There are a number of good monitoring solutions that you can use to watch message queues for the arrival of a message. MonitorWang is an open source one that can detect when a message has arrived in a poison message or error queue. Detecting when a message has been received in the error queue is more reliable than trying to detect when a message has been sent to the error queue.
Related
Question one: Can I subscribe to the event of a message being sent to the _skipped queue?
I am using masstransit together with rabbit mq. Some messages sometimes are sent to the _skipped queue for unclear reasons. The message type has a consumer, the ttl (time to life) is not small. It should not happen, and I am getting a log entry from masstransit, but I want to do more at the moment. Maybe log an error, in test maybe pop-up a window. Is there a way to achieve this? I am only getting these log messages below.
MassTransit.ReceiveTransport|SKIP rabbitmq://localhost/services_admin db270000-1fd6-00ff-3b83-08d9000ef97c
MassTransit.ReceiveTransport|Declare queue: name: services_admin_skipped, durable, consumer-count: 0 message-count: 3
Question two: What exactly happens to messages in the _skipped queue? Can they be resent?
Skipped messages either don't match the type (namespace included), don't have a consumer on the endpoint, or were a response to a request client that is no longer waiting for it. Since it's a receive endpoint queue, it's likely one of the first two reasons. Look at the message body/details in the RabbitMQ Management Console, that should give you some ideas.
You can use a shovel in RabbitMQ to move the messages back into the queue once you've resolved the issue.
I am using java client of
https://www.rabbitmq.com/tutorials/tutorial-six-java.html
. My setup is RPC. My server is creating queue and client is also creating same queue and sending the message. After receiving message server is performing some operation and sending result back to client.
Now if server created the queue and connect with it while queue get's deleted for some reason. The server is not throwing any exception and when the client is creating the same queue and putting messages server is not getting those messages either as it's not connected.
How do server knows that the queue get deleted?
Thanks so much
It sounds like the following situation is happening:
Queue A is created.
Consumer 1 subscribes to Queue A
Queue A is deleted while Consumer 1 is still active
Queue A is re-created (call it A')
Now, you're wondering why Consumer 1 is not getting any messages? You would have to re-subscribe your consumer. I don't usually delete queues, because there is no need to do so under any reasonable scenario (instead, use the queue.expires property to handle auto-deletion of queues).
According to the AMQP 0-9-1 Specification,
When a queue is deleted any pending messages are sent to a dead-letter
queue if this is defined in the server configuration, and all
consumers on the queue are cancelled.
So, based on the description of the behavior, this is a bug with the consumer. It should throw an exception or otherwise exit the consuming loop in this case. In any case, you'll have to re-subscribe to A' before you'll get any more messages.
We have a WCF service that listens for messages on a queue (MSMQ). It sends a request to our web server (REST API), which returns an HTTP status code.
If the status code falls within the 400 range, we are throwing away the message. The idea is a 400 range error can never succeed (unauthorized, bad request, not found, etc.) and so we don't want keep retrying.
For all other errors (e.g., 500 - Internal Server Error), we have WCF configured to put the message on a "retry" queue. Messages on the retry queue get retried after a certain amount of time. The idea is that the server is temporarily down, so wait and try again.
The way WCF is set up, if we throw a FaultException in the service contract, it will automatically put the message on the retry queue.
When a message causes a 400 range error, we are just swallowing the error (we just log it). This prevents the retry mechanism from firing; however, it would be better to move the message to a dead-letter queue. This way we can react to the error by sending an email to the user and/or a system administrator.
Is there a way to immediately move these bad messages to a dead-letter queue?
First, I kept referring to the dead-letter queue. At the time when I posted this question, I was unaware that WCF/MSMQ automatically creates what's known as a poison sub-queue. Any message that can't be delivered in the configured number of times is put in the poison sub-queue.
In my situation, I knew that some messages would never succeed, so I wanted to move the message out of the queue immediately.
The solution was to create a second queue that I called "poison" (not to be confused with the poison sub-queue). My catch block would create an instance of a WCF client and forward the message to this poison queue. I could reuse the same client to post to both the original queue and the poison queue; I just had to create a separate client end-point in the configuration file for each.
I had two separate ServiceHost instances running that read the queues. The ServiceHost for the original queue did the HTTP request and forwarded messages to the poison queue when unrecoverable errors occurred. The second ServiceHost would simply send out an email to record that a message was lost.
There was also the issue of temporary errors that exceeded the maximum number of tries. WCF/MSMQ automatically creates a sub-queue called <myqueuename>;poison. You cannot directly write to a sub-queue via WCF, but you can read from it using a ServiceHost. Whenever messages end up in the poison sub-queue, I simply forward the message to the poison queue, with the exact same client I use in the original handler's catch block.
I wanted the ability to include a stack trace in the error emails. Since I was reusing the same client and service contract for all of the handlers, I couldn't just pass along the stack trace as a string (unless I added it to all of my data contracts). Instead, I had the poison handler try to execute the code one more time, which would fail again and spit out the stack trace.
This is what my message queues ended up looking like:
MyQueue
- Queue messages
- Retry
- Poison
MyQueuePoison
- Queue messages
This approach is pretty convoluted. It was strange calling A WCF client from within a WCF service handler. It also meant setting up one more queue on the server and a ton of additional configuration sections for specifying which queue a client should forward messages to.
hopefully I have understood your question and if it is what i think you are saying then yes there is but you obviously need to program it to do this. But you DO need a retry amount set so the MSMQ can retry until it gives up. Or you can create your own custom queue for dead letters/messages
http://msdn.microsoft.com/en-us/library/ms789035(v=vs.110).aspx
http://msdn.microsoft.com/en-us/library/ms752268(v=vs.110).aspx
take a look here also:
http://www.michaelfcollins3.me/blog/2012/09/20/wcf-msmq-bad-message-handling.html
How do I handle message failure in MSMQ bindings for WCF
I hope these links help.
Is there a good best practice about what kind of messages an application is allowed to reject?
My understanding is that all messages which can't be handled should be rejected to the dead letter queue - no matter if the problem is a syntax error or a semantic error in the message or if the application is temporarily not able to handle the message (for instance because the db just went down).
Of course - if the app already knows upfront that it will not be able to handle a message (DB down), it should stop accepting messages.
So what's the common understanding / best practice?
My response is with respect to WebSphere MQ:
A Dead Letter Queue (DLQ for short) is a place where messages that could not be delivered to their destination are put. Messages can be put on the DLQ by queue managers, message channel agents (MCAs), and applications. All messages on the DLQ must be prefixed with a dead-letter header structure, MQDLH. The MQDLH header is automatically fixed when queue manager or MCAs put messages whereas applications must prefix the MQDLH explicitly.
As far applications are concerned, if they are unable to handle the message, say for example the message format is not understood, they can put the message to a BACKOUT queue instead of a DLQ. A BACKOUT queue is just like any normal queue where messages rejected by applications can be put. The advantage of BACKOUT queue is that you can specify a BACKOUT queue on a per queue basis and the messages put there need not have MQDLH header prefixed.
An application can be written to read the messages from BACKOUT and route them back to the target queue as it is. However the messages in a DLQ require additional processing to remove the MQDLH before they are put onto a target queue.
I understand the difference between Dead Letter Queue and Poison Queue. I will be speaking in the context of transactions and durability.
DLQ: Client sends message. MSMQ is not able to deliver the message. Message remains in queue, waiting to be sent to the destination queue.
PQ: Message is successfully delivered. WCF service processes the message. Some error occured during the WCF operations. Message is placed in the poison queue.
(please correct me if I am wrong above)
Now, if the WCF Service encounters an error, and with the transaction scope in place (attribute), then how does it place the message in the poison queue or does this have to be developed? if it needs to be developed then how is it done?
What about processing poison messages? Is there a way to place them back in the queue for processing again? How is it determined if the message is poison? For example, an operation in WCF may encounter an error but can recover from it.
You can read more about Poison Message Handling on MSDN More about the MSMQ on MSDN - How to: Exchange Messages with WCF Endpoints and Message Queuing Applications
"When the service reads messages from the target queue under a transaction, the service may fail to process the message for various reasons. The message is then put back into the queue to be read again. To deal with messages that fail repeatedly, a set of poison-message handling properties can be configured in the binding. There are four properties: ReceiveRetryCount, MaxRetryCycles, RetryCycleDelay, and ReceiveErrorHandling. "
Only Message Queuing places messages in dead-letter queues. Applications can only read and delete messages in dead-letter queues.
An answer from https://social.msdn.microsoft.com/Forums/vstudio/en-US/84a0a601-1e0a-4693-b5f7-868d3eacb43a/what-is-the-difference-between-dead-letter-queue-and-poison-queue-in-msmq-?forum=wcf
A message in the Poison Queue is a message that has exceeded the
maximum number of delivery attempts to the application. These poison
messages are placed in teh Poison Message Queue. Items in the queue
still need to be read and handled though.
The Dead Letter Queue is for messages that will not be handled at all
and no processing will be done on them.
It is common to move items from the Poison Message Queue to the Dead
Letter Queue.