Background:
I'm troubleshooting a problem where messages sent by WCF over transactional MSMQ (with netMsmqBinding) seem to disappear. The code that uses WCF is in a third-party assembly which I cannot change. I have few clues to what the problem is, but plan to enable various tracing capabilities in order to pin-point where the problem relies.
Context:
I have enabled MSMQ End-to-End Tracing. It logs two events for every message that gets sent.
One event when a message is written to the outgoing queue. This message contain the MSMQ message id (which is composed by a guid and an integer, ie 7B476ADF-DEFD-49F2-AF5A-0CF27C5152C0\6481271).
Another event when that message is sent across the network.
I have enabled verbose WCF Tracing.
I also have application level logging that logs a message IDs defined by the application code (let's call this the "application message id").
I have enabled positive and negative source journaling on the MSMQ messages that get sent.
I have enabled journaling on the receiving queue.
Problem:
When messages go missing, I know the missing message's application id (it's logged by the sending side). I would now like to look at the End-to-End trace to see
whether the message was written to the outgoing queue or not.
How can I correlate the events in the End-to-End trace with the application level logs and WCF traces?
Ideas:
When sending a MSMQ message using the managed MSMQ API in System.Messaging, the message's MSMQ id is available after the message is sent. However, I have not found a way to log this when WCF is performing the send operation. The WCF trace logs a MSMQMessageId guid, but this value is, surprisingly, not the actual MSMQ id as I guessed it would be. Is it possible to access the actual MSMQ message id and log it?
Log the native thread id in the application log along with the application level id and a time stamp. The native thread id is logged to the End-to-End trace by MSMQ, so this might actually be sufficient to correlate. This is plan B for me if I don't find a more elegant solution.
You sounds like you're on the right track. However you could bump up a bit with this:
Using SvcConfigEditor.exe
Configure WCF Verbose Tracing for Propagate ACtiveity and Activity tracing
Configure WCF MessageLogging for "Malformed Messages, Service Messages, Transport Messages"
Use LogEntireMessage
In End to End, trace it All
Make sure you enable these *.config on BOTH sides, yours and the 3rd party executable.
Collect your logs files, and add them ALL to SvcTraceViewer.exe
You can configure windows MSMQ to sense subjects of messages and if subjects contains a key word fire an application. This application can logs incoming messages. In sender side you can write actual message id into subject of message and add your key word to subject. In receiver side fired application can access to actual message id near added key word in subject.
It looks like your message is being discarded by WCF because it is malformed in some way (i.e. contract mismatch, one of the WCF message size limits exceeded).
To trap this error you could write an ErrorHanlder that audits these errors.
Here a link to a sample of doing that.
Another option ,if you are using Win 2008 R2 and up, is to use the built in poison message handling. here`s a link to the the docs.
To the question, to trace end to end with an application trace identifier:
I would pass the application trace id in the message header (look here for an example).
To audit the message header on the service side i would use WCF's IOperationInvoker to intercept each call, and audit the id in the messaged header.
This can be configured in the config file for the process without altering the third party code.here`s an example of how to implement an invoker and how to set it in config.
Related
I've got 2 services set up to receive the same message, e.g. ConsumerA : IConsumer<IMessageA> and ConsumerB : IConsumer<IMessageA>. Each service sets up a unique endpoint, e.g. queue_a and queue_b and registers its consumer. I see in RabbitMQ an exchange for IMessageA that is of type fanout and bound to queue_a and queue_b. So far, so good.
I run both services and publish a message, but only service A is gets it.
I stop service A and manually publish a message to B in RabbitMQ (service A is a web service that publishes IMessageA in response to a POST using IRequestClient<IMessageA, IMessageAResponse>, which I why I need to manually post) and now service B gets the message and consumes it as expected.
To be clear, with service A stopped, the message is routed by RabbitMQ to both queue_a and queue_b. If service A is running, the message only goes to queue_b in spite of exchange bindings existing showing that queue_b is bound to the IMessageA exchange and absolutely should be getting them. Or at least by the time I'm able to examine RabbitMQ through the management web UI, there is no evidence there was ever a message delivered to queue_b (i.e. nothing in queue_b_error or queue_b_skipped, the latter not even existing).
I've added IReceiveObserver to both service A and B but nothing is triggering ReceiveFault or ConsumeFault.
The consumer in service A is basically doing:
var result = await MethodThatReturnsIMessageAResponse(messageA);
context.Respond(result);
Why is service A interfering with delivery of messages to service B? Where do I even begin looking?
The problem was that I was "publishing" the messages using:
c.Resolve<IBus>().CreateRequestClient<IToDoMessage, IToDoMessageResponse>(new Uri(QueueAddress + QueueName),TimeSpan.FromSeconds(5));
which requires a specific enpoint (e.g. queue name). Instead, I needed to use CreatePublishRequestClient:
c.Resolve<IBus>().CreatePublishRequestClient<IToDoMessage, IToDoMessageResponse>(TimeSpan.FromSeconds(5));
which uses the bus to publish, and goes through the exchange, not to a specific queue. It doesn't help that the GitHub sample project shows the former...
I have an NServiceBus endpoint that handles saving documents to a document management system. After the document is saved, I call Bus.Reply(new DocumentSaved{}).
This works fine when I am sending SaveDocument from a Saga (which cares deeply about the reply), but it fails when I am sending it from my web client endpoint (i.e. an MVC project, which doesn't care at all about the reply). The failure is because my web client endpoint doesn't have a queue to process the reply.
What am I doing wrong here? (I really don't want to have to create a queue for my MVC project to hold a bunch of replies that will never ever get processed.)
Replies are just normal messages. The only thing that links original messages and relies is correlation id, which is stored in the message header and the originator address, where a reply is sent to.
This means that all rules that apply to normal messages are also applicable to replies. There are no special "reply queues". Replies go to normal queues as any other message.
I suspect that you have no message-endpoint mapping configuration in your web endpoint. I am not sure if SendOnly endpoint has any effect here, since I assume you already received a message there, which you want to send a reply to.
I would start by checking the message assembly to endpoint mapping and enabling debug level logging.
We have a WCF service that listens for messages on a queue (MSMQ). It sends a request to our web server (REST API), which returns an HTTP status code.
If the status code falls within the 400 range, we are throwing away the message. The idea is a 400 range error can never succeed (unauthorized, bad request, not found, etc.) and so we don't want keep retrying.
For all other errors (e.g., 500 - Internal Server Error), we have WCF configured to put the message on a "retry" queue. Messages on the retry queue get retried after a certain amount of time. The idea is that the server is temporarily down, so wait and try again.
The way WCF is set up, if we throw a FaultException in the service contract, it will automatically put the message on the retry queue.
When a message causes a 400 range error, we are just swallowing the error (we just log it). This prevents the retry mechanism from firing; however, it would be better to move the message to a dead-letter queue. This way we can react to the error by sending an email to the user and/or a system administrator.
Is there a way to immediately move these bad messages to a dead-letter queue?
First, I kept referring to the dead-letter queue. At the time when I posted this question, I was unaware that WCF/MSMQ automatically creates what's known as a poison sub-queue. Any message that can't be delivered in the configured number of times is put in the poison sub-queue.
In my situation, I knew that some messages would never succeed, so I wanted to move the message out of the queue immediately.
The solution was to create a second queue that I called "poison" (not to be confused with the poison sub-queue). My catch block would create an instance of a WCF client and forward the message to this poison queue. I could reuse the same client to post to both the original queue and the poison queue; I just had to create a separate client end-point in the configuration file for each.
I had two separate ServiceHost instances running that read the queues. The ServiceHost for the original queue did the HTTP request and forwarded messages to the poison queue when unrecoverable errors occurred. The second ServiceHost would simply send out an email to record that a message was lost.
There was also the issue of temporary errors that exceeded the maximum number of tries. WCF/MSMQ automatically creates a sub-queue called <myqueuename>;poison. You cannot directly write to a sub-queue via WCF, but you can read from it using a ServiceHost. Whenever messages end up in the poison sub-queue, I simply forward the message to the poison queue, with the exact same client I use in the original handler's catch block.
I wanted the ability to include a stack trace in the error emails. Since I was reusing the same client and service contract for all of the handlers, I couldn't just pass along the stack trace as a string (unless I added it to all of my data contracts). Instead, I had the poison handler try to execute the code one more time, which would fail again and spit out the stack trace.
This is what my message queues ended up looking like:
MyQueue
- Queue messages
- Retry
- Poison
MyQueuePoison
- Queue messages
This approach is pretty convoluted. It was strange calling A WCF client from within a WCF service handler. It also meant setting up one more queue on the server and a ton of additional configuration sections for specifying which queue a client should forward messages to.
hopefully I have understood your question and if it is what i think you are saying then yes there is but you obviously need to program it to do this. But you DO need a retry amount set so the MSMQ can retry until it gives up. Or you can create your own custom queue for dead letters/messages
http://msdn.microsoft.com/en-us/library/ms789035(v=vs.110).aspx
http://msdn.microsoft.com/en-us/library/ms752268(v=vs.110).aspx
take a look here also:
http://www.michaelfcollins3.me/blog/2012/09/20/wcf-msmq-bad-message-handling.html
How do I handle message failure in MSMQ bindings for WCF
I hope these links help.
Suppose I have an application fed by a MQ queue. When the application receives a message that contains errors, the application itself pushes the received message to a certain invalid message queue.
My question is: what is the recommended way to have the receiving application append the failure/rejection reason to the message pushed on the invalid message queue? Some solutions come to mind, but I'm unsure which one is considered "best-practice":
(ab)using a standard header field
adding a custom header
encapsualting the message in another message
If all that you need is to place a reason code in the message, use the MQMD.Feedback field with one of the standard reason codes. In WMQ v7.0 or later, the application can set any number of message properties which are then readable both with JMS semantics and native WMQ API calls. It is up to you to define the taxonomy for naming the application-defined properties.
If the message is requeued to the Dead Letter Queue instead of an application-owned backout queue, it is customary to prepend a Dead Letter Header to it. The MQDLH structure contains a field for the reason code describing why that the message was requeued. As a rule, applications should avoid using the DLQ in favor of an application-owned queue. When applications do use the DLQ, it is normal that they should have access to put messages there but not to retrieve messages from that queue. This is because it is a system-wide resource and messages from different applications may land there. Normally, an admin application or person with elevated access is responsible for adjudicating and disposing of messages on the system DLQ.
I'm writing a sample console service host and I want to plug into WCF stack to be able to print a message to console when new message arrives, even if it won't get processed by the service at the moment (because service is working on previous calls).
This is based on my assumption that messages arriving get queued by the WCF, is that correct?
Additionally, I'm using netTcpBinding if this is important.
You could write a custom channel and insert it into the channel stack at whatever point you want. Queuing happens in ChannelListeners, which sit between each channel:
ChannelListener_1 -> Channel_1 -> ChannelListener_2 -> Channel_2 -> etc.
So, if you were to insert your own channel listener / channel you could hook into any part of the process you need to. Note however that the message contents might not be readable until it gets farther down the stack and gets decrypted.
You probably have to write a custom channel for that. Check out WCF Channels Mini Book