Okay, so I've been trying to wrap my head around queue -> dead-letter queue -> poison subqueue of dead-letter queue -> ?
Now, as far as I understand, messages can be sent to the poison queue if they fail processing or if the Action on the message is not supported by the receiver. Most of the articles I've found show implementing the poison service with the same contract as the main service. Wouldn't that put a message poisonous to the poison message queue into the poison message queue if the error is that the contract doesn't support the message provided?
Wouldn't it make more sense to have a handler that can handle anything in the poison queue? The below assumes a WCF Message, which probably isn't safe either (and is completely untested), but is there a way to have fool-proof poison queue handling? The more I try to think through the edge cases with queueing, the more I feel that it's impossible to have a complete system that handles all possibilities.
[OperationContract(IsOneWay = true, Action = "*")]
[OperationBehavior(TransactionScopeRequired = true, TransactionAutoComplete = true)]
public void CatchAll(Message message)
{
// Log somewhere?
}
Another thought is to have a trigger that moves all messages from the poison queue back into the dead-letter queue to be handled again - which will probably poison again and loop infinitely.
I guess the specific question is.. how do people handle poison messages in a dead-letter queue? And the general question is - how the heck do you handle all situations in MSMQ?
As far as I was aware, the MsmqMessage<T> type can be used in concert with any of the WCF msmq bindings.
So if you specify an recieve type of MsmqMessage<String> this should cover all bases. You will receive the message as a string. How you handle what is in the string is another story...
Related
Can anyone guess what the problem can be because I'm clueless on how to solve this. MassTransit generates _skipped queues and I don't have a clue why it is generating those. It is being generated when doing a publish request response.
Request Client is created using following method in MassTransit.RequestClientExtensions
public static IRequestClient<TRequest, TResponse> CreatePublishRequestClient<TRequest, TResponse>(this IBus bus, TimeSpan timeout, TimeSpan? ttl = null, Action<SendContext<TRequest>> callback = null) where TRequest : class where TResponse : class
{
return (IRequestClient<TRequest, TResponse>) new PublishRequestClient<TRequest, TResponse>(bus, timeout, ttl, callback);
}
And Request is done as follows:
TResponse response = TaskUtil.Await(() => requestClient.Request(request));
As you can see this is Request Response scenario where Request is being sent to all consumers. But because at the moment we have only one consumer it only is being sent to that consumer. deadletters appear easily if a publishrequestresponse is done to multiple consumers, once a consumer responds, the other consumer doesn't know where to respond and a deadletter is generated. But because we have one consumer here, we can eliminate this possibility.
So what could be other reasons for these skipped queues? Huge thanks for any help on how I can troubleshoot this...
I have to say, in the Consume method, in some condition, we raise a RequestTimeoutException and catch it in the requesting application. This is tested and this doesn't generate skipped queues.
Skipped queue is a dead letter queue. It means that your endpoint queue has a binding to some message exchange but there is no consumer for that message any longer. Maybe you change the topology and moved the consumer. You can go to the RMQ management UI and check the bindings for your endpoint exchange. If you look at messages that ended up in the skipped queue, you will find out what message types to look for.
Exchanges are named after message types so it will be easy to find the obsolete binding.
Then, in the management UI, you can manually remove the binding that is obsolete and there will be no more messages coming to the skipped queue.
I have a producer and broker on the same machine. The producer sends messages like so:
channel = connection.createChannel();
//Create a durable queue (if not already present)
channel.queueDeclare(merchantId, true, false, false, null);
//Publish message onto the queue
channel.basicPublish("", consumerId, true, false,
MessageProperties.MINIMAL_PERSISTENT_BASIC, "myMessage");
The consumer sits on another machine and listens to messages. It uses explicit acknowledgement like so:
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
//Handle message here
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
From what I understand, the ack is meant for the broker to dequeue the message.
But how can my producer come to know about the ack that the consumer sent?
Producers and consumers normally don't interact. This is by AMQP protocol design. For example, consuming a specific message may be done a long time after it was published, and there is no sense in leaving the producer up and running for a long time. Another example is when a publisher sends one message to a broker, and due to routing logic that message gets duplicated to more than one queue, leading to ambiguity (because multiple consumers can acknowledge the same message). AMQP protocol is asynchronous (mostly), and letting the publisher know about its message being consumed just doesn't fit the AMQP async model.
There are exceptions from that, notably, RPC calls. Then the producer becomes a producer-consumer. It sends a message and then immediately waits for a reply (there is a good RabbitMQ manual - Direct reply-to related to RPC with RabbtiMQ).
In general, you can ensure that a message is delivered to a broker with Confirms (aka Publisher Acknowledgements) alongside with Dead Letter Exchanges and Alternate Exchanges. Those cover most cases under which a message can be lost from its normal flow.
We have a WCF service that listens for messages on a queue (MSMQ). It sends a request to our web server (REST API), which returns an HTTP status code.
If the status code falls within the 400 range, we are throwing away the message. The idea is a 400 range error can never succeed (unauthorized, bad request, not found, etc.) and so we don't want keep retrying.
For all other errors (e.g., 500 - Internal Server Error), we have WCF configured to put the message on a "retry" queue. Messages on the retry queue get retried after a certain amount of time. The idea is that the server is temporarily down, so wait and try again.
The way WCF is set up, if we throw a FaultException in the service contract, it will automatically put the message on the retry queue.
When a message causes a 400 range error, we are just swallowing the error (we just log it). This prevents the retry mechanism from firing; however, it would be better to move the message to a dead-letter queue. This way we can react to the error by sending an email to the user and/or a system administrator.
Is there a way to immediately move these bad messages to a dead-letter queue?
First, I kept referring to the dead-letter queue. At the time when I posted this question, I was unaware that WCF/MSMQ automatically creates what's known as a poison sub-queue. Any message that can't be delivered in the configured number of times is put in the poison sub-queue.
In my situation, I knew that some messages would never succeed, so I wanted to move the message out of the queue immediately.
The solution was to create a second queue that I called "poison" (not to be confused with the poison sub-queue). My catch block would create an instance of a WCF client and forward the message to this poison queue. I could reuse the same client to post to both the original queue and the poison queue; I just had to create a separate client end-point in the configuration file for each.
I had two separate ServiceHost instances running that read the queues. The ServiceHost for the original queue did the HTTP request and forwarded messages to the poison queue when unrecoverable errors occurred. The second ServiceHost would simply send out an email to record that a message was lost.
There was also the issue of temporary errors that exceeded the maximum number of tries. WCF/MSMQ automatically creates a sub-queue called <myqueuename>;poison. You cannot directly write to a sub-queue via WCF, but you can read from it using a ServiceHost. Whenever messages end up in the poison sub-queue, I simply forward the message to the poison queue, with the exact same client I use in the original handler's catch block.
I wanted the ability to include a stack trace in the error emails. Since I was reusing the same client and service contract for all of the handlers, I couldn't just pass along the stack trace as a string (unless I added it to all of my data contracts). Instead, I had the poison handler try to execute the code one more time, which would fail again and spit out the stack trace.
This is what my message queues ended up looking like:
MyQueue
- Queue messages
- Retry
- Poison
MyQueuePoison
- Queue messages
This approach is pretty convoluted. It was strange calling A WCF client from within a WCF service handler. It also meant setting up one more queue on the server and a ton of additional configuration sections for specifying which queue a client should forward messages to.
hopefully I have understood your question and if it is what i think you are saying then yes there is but you obviously need to program it to do this. But you DO need a retry amount set so the MSMQ can retry until it gives up. Or you can create your own custom queue for dead letters/messages
http://msdn.microsoft.com/en-us/library/ms789035(v=vs.110).aspx
http://msdn.microsoft.com/en-us/library/ms752268(v=vs.110).aspx
take a look here also:
http://www.michaelfcollins3.me/blog/2012/09/20/wcf-msmq-bad-message-handling.html
How do I handle message failure in MSMQ bindings for WCF
I hope these links help.
I understand the difference between Dead Letter Queue and Poison Queue. I will be speaking in the context of transactions and durability.
DLQ: Client sends message. MSMQ is not able to deliver the message. Message remains in queue, waiting to be sent to the destination queue.
PQ: Message is successfully delivered. WCF service processes the message. Some error occured during the WCF operations. Message is placed in the poison queue.
(please correct me if I am wrong above)
Now, if the WCF Service encounters an error, and with the transaction scope in place (attribute), then how does it place the message in the poison queue or does this have to be developed? if it needs to be developed then how is it done?
What about processing poison messages? Is there a way to place them back in the queue for processing again? How is it determined if the message is poison? For example, an operation in WCF may encounter an error but can recover from it.
You can read more about Poison Message Handling on MSDN More about the MSMQ on MSDN - How to: Exchange Messages with WCF Endpoints and Message Queuing Applications
"When the service reads messages from the target queue under a transaction, the service may fail to process the message for various reasons. The message is then put back into the queue to be read again. To deal with messages that fail repeatedly, a set of poison-message handling properties can be configured in the binding. There are four properties: ReceiveRetryCount, MaxRetryCycles, RetryCycleDelay, and ReceiveErrorHandling. "
Only Message Queuing places messages in dead-letter queues. Applications can only read and delete messages in dead-letter queues.
An answer from https://social.msdn.microsoft.com/Forums/vstudio/en-US/84a0a601-1e0a-4693-b5f7-868d3eacb43a/what-is-the-difference-between-dead-letter-queue-and-poison-queue-in-msmq-?forum=wcf
A message in the Poison Queue is a message that has exceeded the
maximum number of delivery attempts to the application. These poison
messages are placed in teh Poison Message Queue. Items in the queue
still need to be read and handled though.
The Dead Letter Queue is for messages that will not be handled at all
and no processing will be done on them.
It is common to move items from the Poison Message Queue to the Dead
Letter Queue.
I am running a WCF client that invokes a WCF service via an MsmqBinding.
Framework is .Net 4.0, client and server runs on Windows Server 2008 R2.
The channel queue is transactional.
The service is hosted with these binding parameters:
receiveErrorHandling="Move"
receiveRetryCount="3"
retryCycleDelay="00:00:20"
maxRetryCycles="5"
Given that ((ReceiveRetryCount+1) * (MaxRetryCycles + 1)) is in effect, this will result in
4*6 = 24 retries of any given message before it is moved to the poison subqueue.
Attaching an IErrorHandler to my service I notice that HandleError is called with a MsmqPoisonMessageException a total of 6 times (for a poison message), before the wcf subsystem finally moves the message to the ;poison subqueue.
I want to log the precise time when a message is done being retried and the message is moved to the poison queue. It seems to me the only option is to count the number of times a certain message faults and compare this count with the binding MaxRetryCycles. This is awkward and errorprone.
My question is:
Is there any way for me to
conclusively detect the event where
the wcf subsystem moves the message
to the poison queue?
My references are:
http://msdn.microsoft.com/en-us/library/aa395218.aspx
And: http://consultingblogs.emc.com/simonevans/archive/2007/09/17/A-comprehensive-guide-to-using-MsmqIntegrationBinding-with-MSMQ-3.0-in-WCF.aspx
The number of retries is of course an outcome of your parameters; however in your IErrorHandler you can explicitly move the message to the poison queue yourself. Otherwise, it will always move based on your binding parameters, and would be detected by listening to the poison queue like any other queue.
There are a number of good monitoring solutions that you can use to watch message queues for the arrival of a message. MonitorWang is an open source one that can detect when a message has arrived in a poison message or error queue. Detecting when a message has been received in the error queue is more reliable than trying to detect when a message has been sent to the error queue.