I have a singleton orchestration which concatenates the incoming messages. When the number of incoming messages is large, it throws an error after around 2000 messages that "The instance completed without consuming all of its messages. The instance and its unconsumed messages have been suspended."
Normally this is a zombie message error associated with singleton orchestrations, but we are dropping all the XMLs instantly which are published to message box using a FILE adapter.
Since messages are already present in Message box, why are we still getting a zombie message error.
Also, why does this always breaks at around 2000 messages? Is there any threshold associated with this?
Thanks
Pulkit
Related
I am using ActiveMQ with web console (activemq-web-console-5.16.4) in TomEE. The ActiveMQ-web-console-5.16.4.war was added to the TomEE webapps folder. Afterwards, I could access the web console. Currently, I want to view/monitor the content of enqueued/processed messages in the web console "Messages Enqueued". How can I manage that in my case? Should I bind the KahaDB message store or other databases?
In my application I use Apache Camel and send messages from one route to another by ActiveMQ.
I would appreciate any help.
Screenshots:
You can use the web console itself to view the content of the message assuming it fits into the narrow constraints of what the console can decode into human readable format.
First, click the "Browse" link.
Second, click the link for the actual message.
Third, see the "Message Details."
To be clear, you can only inspect the content of messages which are in the queue. This is represented by the "Number of Pending Messages." The "Messages Enqueued" is the number of messages sent to the queue (but not necessarily in the queue currently) since the broker was started. The "Messages Dequeued" is the number of messages consumed from the queue. In your case you have 66 messages which have been enqueued and dequeued (i.e. consumed) and therefore 0 pending messages.
If you want to keep a copy of every message sent to your queue for auditing purposes you can use a mirrored queue. As noted previously, you can only inspect messages which are in the queue and a mirrored queue will hold a copy of every message sent to the source queue allowing you to inspect those messages at your convenience.
We have a WCF service that listens for messages on a queue (MSMQ). It sends a request to our web server (REST API), which returns an HTTP status code.
If the status code falls within the 400 range, we are throwing away the message. The idea is a 400 range error can never succeed (unauthorized, bad request, not found, etc.) and so we don't want keep retrying.
For all other errors (e.g., 500 - Internal Server Error), we have WCF configured to put the message on a "retry" queue. Messages on the retry queue get retried after a certain amount of time. The idea is that the server is temporarily down, so wait and try again.
The way WCF is set up, if we throw a FaultException in the service contract, it will automatically put the message on the retry queue.
When a message causes a 400 range error, we are just swallowing the error (we just log it). This prevents the retry mechanism from firing; however, it would be better to move the message to a dead-letter queue. This way we can react to the error by sending an email to the user and/or a system administrator.
Is there a way to immediately move these bad messages to a dead-letter queue?
First, I kept referring to the dead-letter queue. At the time when I posted this question, I was unaware that WCF/MSMQ automatically creates what's known as a poison sub-queue. Any message that can't be delivered in the configured number of times is put in the poison sub-queue.
In my situation, I knew that some messages would never succeed, so I wanted to move the message out of the queue immediately.
The solution was to create a second queue that I called "poison" (not to be confused with the poison sub-queue). My catch block would create an instance of a WCF client and forward the message to this poison queue. I could reuse the same client to post to both the original queue and the poison queue; I just had to create a separate client end-point in the configuration file for each.
I had two separate ServiceHost instances running that read the queues. The ServiceHost for the original queue did the HTTP request and forwarded messages to the poison queue when unrecoverable errors occurred. The second ServiceHost would simply send out an email to record that a message was lost.
There was also the issue of temporary errors that exceeded the maximum number of tries. WCF/MSMQ automatically creates a sub-queue called <myqueuename>;poison. You cannot directly write to a sub-queue via WCF, but you can read from it using a ServiceHost. Whenever messages end up in the poison sub-queue, I simply forward the message to the poison queue, with the exact same client I use in the original handler's catch block.
I wanted the ability to include a stack trace in the error emails. Since I was reusing the same client and service contract for all of the handlers, I couldn't just pass along the stack trace as a string (unless I added it to all of my data contracts). Instead, I had the poison handler try to execute the code one more time, which would fail again and spit out the stack trace.
This is what my message queues ended up looking like:
MyQueue
- Queue messages
- Retry
- Poison
MyQueuePoison
- Queue messages
This approach is pretty convoluted. It was strange calling A WCF client from within a WCF service handler. It also meant setting up one more queue on the server and a ton of additional configuration sections for specifying which queue a client should forward messages to.
hopefully I have understood your question and if it is what i think you are saying then yes there is but you obviously need to program it to do this. But you DO need a retry amount set so the MSMQ can retry until it gives up. Or you can create your own custom queue for dead letters/messages
http://msdn.microsoft.com/en-us/library/ms789035(v=vs.110).aspx
http://msdn.microsoft.com/en-us/library/ms752268(v=vs.110).aspx
take a look here also:
http://www.michaelfcollins3.me/blog/2012/09/20/wcf-msmq-bad-message-handling.html
How do I handle message failure in MSMQ bindings for WCF
I hope these links help.
I am using RabbitMQ version 3.0.2 & I see close to 1000 message in Error queue. I want to know
At what point messages are moved to the error queues?
Is there a way to know why a certain message is being moved to an error queue?
Is there any way to move message from error queue to normal queue?
Thank you
a) they fail to deserialize or b) the consumer throws an exception processing that message five times
Not really... If you peek at the message in the queue, the payload headers might contain a note but I don't think we did that. If you turn logging on (NLog, log4net, etc) you should be able to see the exceptions in your log. You'll have to correlate message ids at that point to figure out exactly why.
There is no built in way via MassTransit. Mostly because there doesn't seem to be a great, generic way to handle this. Everyone wants some process around this. Dru did create a BusDriver app (in the main MT source repo) that could be used to move messages back to the exchange in question. This default behaviour is there so you at least know things have been failing if you don't put in the infrastructure to handle it.
To add to Travis' answer, During my development I found some other reasons for messages going onto the error queue:
The published message type has no consumer
A SAGA and a consumer are expecting the same concrete message type. Even if you try and differentiate using "Accepts" and ".Selected", both a SAGA and a Consumer should not be programmed to receive the same message type.
Suppose I have an application fed by a MQ queue. When the application receives a message that contains errors, the application itself pushes the received message to a certain invalid message queue.
My question is: what is the recommended way to have the receiving application append the failure/rejection reason to the message pushed on the invalid message queue? Some solutions come to mind, but I'm unsure which one is considered "best-practice":
(ab)using a standard header field
adding a custom header
encapsualting the message in another message
If all that you need is to place a reason code in the message, use the MQMD.Feedback field with one of the standard reason codes. In WMQ v7.0 or later, the application can set any number of message properties which are then readable both with JMS semantics and native WMQ API calls. It is up to you to define the taxonomy for naming the application-defined properties.
If the message is requeued to the Dead Letter Queue instead of an application-owned backout queue, it is customary to prepend a Dead Letter Header to it. The MQDLH structure contains a field for the reason code describing why that the message was requeued. As a rule, applications should avoid using the DLQ in favor of an application-owned queue. When applications do use the DLQ, it is normal that they should have access to put messages there but not to retrieve messages from that queue. This is because it is a system-wide resource and messages from different applications may land there. Normally, an admin application or person with elevated access is responsible for adjudicating and disposing of messages on the system DLQ.
I am running a WCF client that invokes a WCF service via an MsmqBinding.
Framework is .Net 4.0, client and server runs on Windows Server 2008 R2.
The channel queue is transactional.
The service is hosted with these binding parameters:
receiveErrorHandling="Move"
receiveRetryCount="3"
retryCycleDelay="00:00:20"
maxRetryCycles="5"
Given that ((ReceiveRetryCount+1) * (MaxRetryCycles + 1)) is in effect, this will result in
4*6 = 24 retries of any given message before it is moved to the poison subqueue.
Attaching an IErrorHandler to my service I notice that HandleError is called with a MsmqPoisonMessageException a total of 6 times (for a poison message), before the wcf subsystem finally moves the message to the ;poison subqueue.
I want to log the precise time when a message is done being retried and the message is moved to the poison queue. It seems to me the only option is to count the number of times a certain message faults and compare this count with the binding MaxRetryCycles. This is awkward and errorprone.
My question is:
Is there any way for me to
conclusively detect the event where
the wcf subsystem moves the message
to the poison queue?
My references are:
http://msdn.microsoft.com/en-us/library/aa395218.aspx
And: http://consultingblogs.emc.com/simonevans/archive/2007/09/17/A-comprehensive-guide-to-using-MsmqIntegrationBinding-with-MSMQ-3.0-in-WCF.aspx
The number of retries is of course an outcome of your parameters; however in your IErrorHandler you can explicitly move the message to the poison queue yourself. Otherwise, it will always move based on your binding parameters, and would be detected by listening to the poison queue like any other queue.
There are a number of good monitoring solutions that you can use to watch message queues for the arrival of a message. MonitorWang is an open source one that can detect when a message has arrived in a poison message or error queue. Detecting when a message has been received in the error queue is more reliable than trying to detect when a message has been sent to the error queue.