I have a service that is configured to handle a command object. In the Handle method I am throwing an exception in specific business cases. When this occurs it send the error to the Error queue and I think the Particular Management service is processing the Error queue. It is my understanding is that the Particular Management service will monitor the Error queue and process the messages, that is to say, mine the metadata and persist it to RavenDB. Lastly, it should forward the message to the Error.Log queue. Regrettably, every message seems to get routed to the Service.Management.Errors queue. Presumably, because the Particular Management service handler is failing.
If I look the Extension Headers of the messages in the Service.Management.Errors queue, I see the following information:
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.ExceptionType</Key>
<Value>System.NullReferenceException</Value>
</HeaderInfo>
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.Message</Key>
<Value>Object reference not set to an instance of an object.</Value>
</HeaderInfo>
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.Source</Key>
<Value>NServiceBus.Core</Value>
</HeaderInfo>
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.StackTrace</Key>
<Value>System.NullReferenceException: Object reference not set to an instance of an object.
at NServiceBus.Unicast.Transport.TransportReceiver.ProcessMessage (TransportMessage message) in c:\TeamCity\buildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 357
at NServiceBus.Unicast.Transport.TransportReceiver.TryProcess(TransportMessage message) in c:\TeamCity\buildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 235
at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.Action() in c:\TeamCity\buildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 170
Since the ServiceInsight Particular Management service is presumably supposed to be generic in nature, I would assume that it does not need to know about the actual types of messages. Meaning, I don't have to throw my Message dll's into the directory of the service (correct?). Thus implying that it simply mines all of it's data to persist to RavenDB by simply reading the Extension Headers.
So with that ground work in place. Can someone please validate or invalidate my understanding of how ServiceInsight works and is configured. Secondly, my error looks like it is an error in NServiceBus.Core. Am I supposed to add some DLL's or something. Of is there another likely candidate for causing this issue that I am simply overlooking.
We were having a similar problem, and although our problem was probably pretty specific to us, I'll share it just in case.
Most of our services use NServiceBus 4.0.3 with XML serialization, but we have one older service that is using NSB 3.3.1 and binary serialization. That service was logging to the audit queue, and when the Particular Management service tried to read those messages, it would fail. I'm not sure what the source of the error was, due to NSB re-throwing the exception and losing the original stack trace. I suspect that it is trying to read a header that isn't there. It may also be due to using binary serialization.
Our solution was to turn off auditing for that one older service.
Related
Background:
I'm troubleshooting a problem where messages sent by WCF over transactional MSMQ (with netMsmqBinding) seem to disappear. The code that uses WCF is in a third-party assembly which I cannot change. I have few clues to what the problem is, but plan to enable various tracing capabilities in order to pin-point where the problem relies.
Context:
I have enabled MSMQ End-to-End Tracing. It logs two events for every message that gets sent.
One event when a message is written to the outgoing queue. This message contain the MSMQ message id (which is composed by a guid and an integer, ie 7B476ADF-DEFD-49F2-AF5A-0CF27C5152C0\6481271).
Another event when that message is sent across the network.
I have enabled verbose WCF Tracing.
I also have application level logging that logs a message IDs defined by the application code (let's call this the "application message id").
I have enabled positive and negative source journaling on the MSMQ messages that get sent.
I have enabled journaling on the receiving queue.
Problem:
When messages go missing, I know the missing message's application id (it's logged by the sending side). I would now like to look at the End-to-End trace to see
whether the message was written to the outgoing queue or not.
How can I correlate the events in the End-to-End trace with the application level logs and WCF traces?
Ideas:
When sending a MSMQ message using the managed MSMQ API in System.Messaging, the message's MSMQ id is available after the message is sent. However, I have not found a way to log this when WCF is performing the send operation. The WCF trace logs a MSMQMessageId guid, but this value is, surprisingly, not the actual MSMQ id as I guessed it would be. Is it possible to access the actual MSMQ message id and log it?
Log the native thread id in the application log along with the application level id and a time stamp. The native thread id is logged to the End-to-End trace by MSMQ, so this might actually be sufficient to correlate. This is plan B for me if I don't find a more elegant solution.
You sounds like you're on the right track. However you could bump up a bit with this:
Using SvcConfigEditor.exe
Configure WCF Verbose Tracing for Propagate ACtiveity and Activity tracing
Configure WCF MessageLogging for "Malformed Messages, Service Messages, Transport Messages"
Use LogEntireMessage
In End to End, trace it All
Make sure you enable these *.config on BOTH sides, yours and the 3rd party executable.
Collect your logs files, and add them ALL to SvcTraceViewer.exe
You can configure windows MSMQ to sense subjects of messages and if subjects contains a key word fire an application. This application can logs incoming messages. In sender side you can write actual message id into subject of message and add your key word to subject. In receiver side fired application can access to actual message id near added key word in subject.
It looks like your message is being discarded by WCF because it is malformed in some way (i.e. contract mismatch, one of the WCF message size limits exceeded).
To trap this error you could write an ErrorHanlder that audits these errors.
Here a link to a sample of doing that.
Another option ,if you are using Win 2008 R2 and up, is to use the built in poison message handling. here`s a link to the the docs.
To the question, to trace end to end with an application trace identifier:
I would pass the application trace id in the message header (look here for an example).
To audit the message header on the service side i would use WCF's IOperationInvoker to intercept each call, and audit the id in the messaged header.
This can be configured in the config file for the process without altering the third party code.here`s an example of how to implement an invoker and how to set it in config.
If I try to send a message (using bus.Send<SomeCommand>(...) for example), and the endpoint is not available for some reason (perhaps it has not been set up yet), I get an exception similar to the following
The destination queue 'myqueue#mycomputer' could not be found. You may
have misconfigured the destination for this kind of message
(MyCommands.SomeCommand) in the MessageEndpointMappings of the
UnicastBusConfig section in your configuration file. It may also be
the case that the given queue just hasn't been created yet, or has
been deleted.
I would like to avoid this by checking ahead of time (on app startup) that the bus is ready and the endpoint is reachable.
Short of catching exceptions on every bus.Send, is there some other way to determine the health of the bus? I'm looking for something like bus.IsReady() but I can't find it.
What are the essential differences between publishing a message using Bus.Publish and sending a message using Bus.Send? I am looking to understand how they differ and also when I should choose to use one over the other.
Publishing is used to notify multiple Subscribers of a particular event. A Publishing endpoint will have subscription storage to identify where to send messages to. Sending is typically used to issue a command to an endpoint. A command is telling the endpoint to do something and should not expect a reply(although you sometimes do want a reply and NSB supports this).
The reason you do not see a destination for Send() is that you specify the destination via configuration. In your app.config you will map message types(a whole assembly or a class) to a destination. When you do so, you do not have to provide the destination.
Bus.Publish: used when you don't know where the message is going (0 to many subscribers).
Bus.Send: when you are sending a message to a specific handler (client to server).
ususally Context.Publish() is for publishing Event Type and Context.Send() is for Command Type
It's a really common pattern and I'm finding it a nightmare to implement!
--
The WcfIntegration sample is almost what I'm looking for in that it receives messages via a WCF endpoint. However, it receives messages back on itself. I want a separate subscriber.
So what I'm trying to do is merge the WcfIntegration and pub/sub samples.
The real need is for a website to call a class library, which then calls the WCF endpoint of the publisher.
A subscriber then picks receives a message that the publisher publishes, and does whatever with it.
--
Rob
You shouldn't need the interface IEventMessageService; NSB handles that for you.
In my local example, I mapped the messages to myself. For you this would be:
<UnicastBusConfig><MessageEndpointMappings>
<add Messages="MyMessages" Endpoint="MyPublisherInputQueue"/>
</MessageEndpointMappings></UnicastBusConfig>
You also need to move your Publish to the actual handler. Right now, it is only happening once (at startup):
public void Handle(EventMessage message)
{
bus.Publish(message);
bus.Return((int)ErrorCodes.None);
}
Make sure your infrastructure is primed, i.e. the queue is transactional, and MSDTC is running.
I'm willing to bet that since there is no mapping, once the service is called, it doesn't know where to put the messages (it calls Bus.Send() internally). I didn't look at the subscribers since that didn't seem to be the issue.
I've finally got it to work here:-
http://code.google.com/p/nservicebus-wcf-pubsub/downloads/list
Anyone is welcome to improve the code.
I have the standard error handing in place in my service:
I have an IErrorHandler hooked to the service to handle unexpected errors during service execution.
I have try/catch blocks in all my service methods to handle expected cases.
However, there are cases where exceptions are thrown on the server and neither is called.
Here is a case where the server exception is not sent to the IErrorHandler:
Set the receiveTimout on the server binding to 5 seconds.
On the client do this:
.
Service1Client sc = new Service1Client();
ICommunicationObject o = sc as ICommunicationObject;
o.Open(); // open channel
sc.GetData(10); // do a first call
Thread.Sleep(10000); // wait longer than the server receiveTimeout
sc.GetData(10); // Attempt another call: server throws a FaulException
In that case, the error is thrown on the server but I cannot find a way to handle it (and log it). I know an error is raised because if I attach a debugger on the server process and break on all exceptions, the debugger breaks.
I have found other similar cases where low level errors are not passed to my program.
Where can I hook my code to ensure that I can handle ALL exceptions that occur on the server before they are returned to the client app? Should I implement my own IChannel or some other low level interface?
Thanks
UPDATE Sep 21 2009: See this thread on the Microsoft WCF Forum. I'll probably have to implement my own Channel if I want to handle this type of exception. I'll update this post again when I have more info.
After much research and experimentation, the answer is:
At this time (.Net 3.5) there is no mechanism that allows one to handle all possible exceptions that may occur in the context of a WCF call.
Exceptions that happen during the service method execution can easily be handled with:
Try/catch blocks in all service methods to handle expected cases.
IErrorHandler hooked to the services to handle unexpected errors during service execution.
However, for low level WCF infrastructure errors, there is no perfect solution. The best solution that exists seems to be to implement a custom channel to catch more exceptions.
In this Microsoft Connect Bug Report, Microsoft confirms that there is no way to handle all types WCF infrastructure errors.
In this thread on the Microsoft WCF forums, there is a sample on how to implement a custom channel. That solution only works for HTTP, not for HTTPS. Also some WCF infrastructure errors are not caught by the custom channel either (see more details in that specific thread).
Use FaultContracts. Then the fault can be handled at the client end.
http://msdn.microsoft.com/en-us/library/ms732013.aspx
http://www.c-sharpcorner.com/UploadFile/ankithakur/ExceptionHandlingWCF12282007072617AM/ExceptionHandlingWCF.aspx
This is also much better for debugging, since often you will be developing a client and don't want to bring down the server for debugging purposes.
On the client end, use try/catch blocks to catch all exceptions/faults. There are definitely errors that can't be detected on the server end, such as a communication problem, so you need to handle errors on the client end anyways.
If you want centralized error handling, you can create a service that takes messages about all errors, send the error to that server, and have it log that. This can be useful if you want to create a centralized message tracing/performance analysis/logging tool and have a large number of application processors, servers, clients etc.
The point is - if the server is not reachable or can't handle the message, there won't be an error on the server - the error will pop up on the client ("TimeoutException" or others).
So in those cases, having the IErrorHandler on the server really isn't gonna help - since the error really happens on the client (no connection can be made, due to network down, or typo in server's address or sstuff like that).
So on the client side, you definitely also have to use try....catch around all your server calls.
Marc
Set up diagnostic tracing and check the logs with Service Trace Viewer Tool. Link contains information about configuring tracing as well.