I have a service that is configured to handle a command object. In the Handle method I am throwing an exception in specific business cases. When this occurs it send the error to the Error queue and I think the Particular Management service is processing the Error queue. It is my understanding is that the Particular Management service will monitor the Error queue and process the messages, that is to say, mine the metadata and persist it to RavenDB. Lastly, it should forward the message to the Error.Log queue. Regrettably, every message seems to get routed to the Service.Management.Errors queue. Presumably, because the Particular Management service handler is failing.
If I look the Extension Headers of the messages in the Service.Management.Errors queue, I see the following information:
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.ExceptionType</Key>
<Value>System.NullReferenceException</Value>
</HeaderInfo>
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.Message</Key>
<Value>Object reference not set to an instance of an object.</Value>
</HeaderInfo>
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.Source</Key>
<Value>NServiceBus.Core</Value>
</HeaderInfo>
<HeaderInfo>
<Key>NServiceBus.ExceptionInfo.StackTrace</Key>
<Value>System.NullReferenceException: Object reference not set to an instance of an object.
at NServiceBus.Unicast.Transport.TransportReceiver.ProcessMessage (TransportMessage message) in c:\TeamCity\buildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 357
at NServiceBus.Unicast.Transport.TransportReceiver.TryProcess(TransportMessage message) in c:\TeamCity\buildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 235
at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.Action() in c:\TeamCity\buildAgent\work\d4de8921a0aabf04\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 170
Since the ServiceInsight Particular Management service is presumably supposed to be generic in nature, I would assume that it does not need to know about the actual types of messages. Meaning, I don't have to throw my Message dll's into the directory of the service (correct?). Thus implying that it simply mines all of it's data to persist to RavenDB by simply reading the Extension Headers.
So with that ground work in place. Can someone please validate or invalidate my understanding of how ServiceInsight works and is configured. Secondly, my error looks like it is an error in NServiceBus.Core. Am I supposed to add some DLL's or something. Of is there another likely candidate for causing this issue that I am simply overlooking.
We were having a similar problem, and although our problem was probably pretty specific to us, I'll share it just in case.
Most of our services use NServiceBus 4.0.3 with XML serialization, but we have one older service that is using NSB 3.3.1 and binary serialization. That service was logging to the audit queue, and when the Particular Management service tried to read those messages, it would fail. I'm not sure what the source of the error was, due to NSB re-throwing the exception and losing the original stack trace. I suspect that it is trying to read a header that isn't there. It may also be due to using binary serialization.
Our solution was to turn off auditing for that one older service.
Is it possible to disable retries in NServiceBus, version 3.2.2?
Using the following configuration, retries can be disabled:
<MsmqTransportConfig NumberOfWorkerThreads="1"
MaxRetries="0" />
<SecondLevelRetriesConfig Enabled="false"
TimeIncrease="00:00:10"
NumberOfRetries="0" />
But not when the thread count is set to 20. In this case, the message is retried twice:
<MsmqTransportConfig NumberOfWorkerThreads="20"
MaxRetries="0" />
<SecondLevelRetriesConfig Enabled="false"
TimeIncrease="00:00:10"
NumberOfRetries="0" />
This does look a lot like a bug. The retry behaviour should not depend on the number of threads.
The semantics of the MaxRetries is "At least X times". The reason for this is performance since we can't be a little more relaxed when it comes to syncronising our threads. You could also make your transport non transactional this will effectively give you one try for each message but you'll lose the error queue as well so failed messages are gone forever.
I see MessageEndpointMappings section in my App.config and it just keeps confusing me.
Sometimes it looks like it sets of a client to subscribe to another queue (like this one does):
<UnicastBusConfig>
<MessageEndpointMappings>
<add Messages="MyServiceBus.MessageHub.Contracts" Endpoint="MessageHub"/>
</MessageEndpointMappings>
</UnicastBusConfig>
But other times it seems to be used to configure outgoing messages to other queues:
<UnicastBusConfig>
<MessageEndpointMappings>
<add Messages="ServiceBus.MessageHub.InternalMessages" Endpoint="MessageHub"/>
</MessageEndpointMappings>
</UnicastBusConfig>
But really, there is no difference between the two of them. So how does it know when to setup a subscription and when to setup for sending messages?
Or is it really doing both all the time and I just don't use both?
Essentially, what you're declaring with the MessageEndpointMappings element, is "who is the owner of the specified messages?"
Another way to view it, is that this is the direction of the service dependency - not the message flow, because that is determined by whether messages get Sendt or Publishd.
Therefore, in order to be able to send messages and subscribe to messages from a given service, you declare an endpoint mapping in the sender/subscriber end.
And then, when the sender/subscriber has an endpoint mapping that points to another service and a handler for a message included in that endpoint mapping, NServiceBus is nice enough to automatically subscribe to that message (unless you disable the auto-subscription with DoNotAutoSubscribe()) - because, logically, that must mean that the sender/subscriber wishes to subscribe to the given message.
When you think about it this way, I think it makes sense. Hope you feel so too :)
Today, I saw many errors on my ActiveMQ 5.3.2 console:
INFO | Usage Manager Memory Limit
reached. Stopping producer (ID:...) to
prevent flooding topic://mytopic. See
http://activemq.apache.org/producer-flow-control.html
for more info (blocking for: 3422ms)
I did a little bit of poking around, and determined that the subscriber had gone out to lunch:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp6 0 130320 10.208.87.178:61613 66.31.31.216:37951 ESTABLISHED
In this situation, I don't want the producer to block; I would prefer to drop the client completely. http://activemq.apache.org/slow-consumer-handling.html explains how to limit the number of messages queued, which is a good start, but isn't really what I want. http://activemq.apache.org/slow-consumers.html alludes to being able to drop a slow consumer, but doesn't explain how one might do this.
So, this is my question: is it possible to set up ApacheMQ to drop slow consumers completely, and how do I do so?
I would turn off producerFlowControl for the topics you want or all of them like so:
<policyEntry topic=">" producerFlowControl="false">
This has the ability to cause you to run out of memory or disk space now though because the message queue could keep growing. So make sure you set up an eviction strategy and pending message limit strategy like so:
<messageEvictionStrategy>
<oldestMessageEvictionStrategy/>
</messageEvictionStrategy>
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="10"/>
</pendingMessageLimitStrategy>
This will start throwing away messages after a limit of 10 has been reached. The oldest messages will be thrown out first.
AbortSlowConsumerStrategy which has been improved in Active MQ 5.9.0 should help in this aspect. I haven't had the opportunity to test it thoroughly for various configurations. But I think this is what you are looking for, especially ignoreIdleConsumers=true.
http://java.dzone.com/articles/coming-activemq-59-new-way
You can configure below property - "sendFailIfNoSpaceAfterTimeout" for throwing error to Producer, instead of blocking.
<systemUsage>
<systemUsage sendFailIfNoSpaceAfterTimeout="3000">
<memoryUsage>
<memoryUsage limit="256 mb"/>
</memoryUsage>
</systemUsage>
</systemUsage>
I cannot give you an answer on how to drop the client ( in fact: i am not so sure it is possible all together) but what you can do is add a policy to not block the producer:
" memoryLimit="10mb" producerFlowControl="false"/>
This way, your producer will not suffer from slow consumers.
In my WCF service, when trying transfer large data I constantly get an error: The underlying connection was closed: The connection was closed unexpectedly
I want to know what particular reason invokes this error, so I set up WCF Tracing and can read traces.svclog file.
The problem is, that I can see in this file a lot of information about flow of processes, I can see exact time when exception is appeared, but I can't see the exact reason for that. Is it due to MaxReceivedMessageSize or something like that.
Is it so that traces.svclog can not contain such information or am I doing something wrong?
How such information could be obtained?
Edited (added):
From my server-side app.config:
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="NAVBinding_ICustomer_Service"
closeTimeout="01:50:00"
openTimeout="01:50:00" receiveTimeout="01:50:00" sendTimeout="01:50:00"
allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
maxBufferSize="2147483647" maxBufferPoolSize="2147483647"
maxReceivedMessageSize="2147483647" messageEncoding="Text"
textEncoding="utf-8" transferMode="Buffered" useDefaultWebProxy="true">
<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647"
maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
</binding>
</basicHttpBinding>
</bindings>
<services>
<service name = "Customer_Service" behaviorConfiguration="returnFaults">
<endpoint name="NAVBinding_ICustomer_Service"
address = "http://localhost:8000/nav/customer"
binding = "basicHttpBinding"
bindingConfiguration= "NAVBinding_ICustomer_Service"
contract = "NAVServiceReference.ICustomer_Service"/>
</service>
</services>
<behaviors>
<serviceBehaviors>
<behavior name="returnFaults" >
<serviceDebug includeExceptionDetailInFaults="true" />
<serviceMetadata httpGetEnabled="true" />
</behavior>
</serviceBehaviors>
</behaviors>
</system.serviceModel>
Edited (added):
What is the right and best way to turn WCF service from a "black box" to an easily troubleshooted service, which tells the reason why something goes not the expected way?
What tools, techniques you use to troubleshoot WCF service?
Ignoring the problems with the maxRequestLength (which have been answered by others),
I will have a go at answering your original question about how to troubleshoot WCF.
If you are already using the Service Trace Viewer (I couldn't tell from the question
if you were just viewing them by hand) - it is possible that all the details aren't
going into the file.
When I want to get really hardcore, I enable all the logging parameters for
message logging. (This will generate some big service logs though so don't leave
it on)
<system.serviceModel>
<diagnostics>
<messageLogging logEntireMessage="true" logMalformedMessages="true" logMessagesAtServiceLevel="true" logMessagesAtTransportLevel="true" maxMessagesToLog="-1" />
</diagnostics>
</system.serviceModel>
If you are not using the Microsoft Service Trace Viewer I recommend that. It
gives all the information I need to track down those tricky message handshake, message
size exceptions etc. Here is an MSDN reference to get you started
http://msdn.microsoft.com/en-us/library/aa751795.aspx
Trace interactions that have potential problems are hilighted in yellow on the
left, and the detailed pane on the top right will normally hilight the exceptional
service event in red. Sometimes you will get multiple problems as the inner
error cascades through the service stack - but you can see it all in the
trace viewer.
If you get nothing in your server 'service log', then it is possible your exceptions are entirely at the client end - theoretically you could exceed some of the client
side security parameters (message size etc) before any message has actually reached the
web service end - but client problems are generally easier to track down because you know you only have to worry about editing the config file at the client end (i.e. it's not because of any interaction between the client and server settings).
I've spent the last 2+ days trying to find why I'm getting "The underlying connection was closed: The connection was closed unexpectedly" with a method call returning with more data vs when not so much data (i.e., it works ok with smaller sets of data being returned only).
My error messsages are slightly different (perhaps due to framework differences) but wanted to share the cause I found. First, I would like to state that while tracing and increasing the sizes of certain things in config files given as answers above may be helpful for tracking down WCF errors, these things did nothing to help me determine the real cause of the error.
By just looking at the exception thrown and up the chain, I could see the following root, error:
"An existing connection was forcibly closed by the remote host" - this was a System.Net.Sockets.SocketException
Going up the call chain then was:
"Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host." - a System.IO.IOException, then
"The underlying connection was closed: An unexpected error occurred on a receive." - a System.Net.WebException, then finally what was the caught exception's message,
"An error occurred while receiving the HTTP response to . This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details." - a System.ServiceModel.CommunicationException
The enabling tracing then viewing the trace logs with the TraceViewer did make this easier to see, but never told me the real cause for my "An existing connection was forcibly closed by the remote host" error.
In my case, my WCF service is hosted on IIS6 and only when I contacted our institutional support in charge of these servers and asked them to look in the system event logs, did I immediately see the answer - a System.OutOfMemoryException!
My WCF service runs in an allocated 200MB of RAM and my method was consuming more than this. I looked in my method and found eventually a block of code that should have been outside/below the block (loop) it was in. . ..so I had an exponential type of collection being generated in my method.
Hope this may help others.
To answer your question how to create an easily trouble shooted WCF service. One way is to minimise the number of potential errors, so that you have fewer things to look at while troubleshooting.
There are two main sources of errors:
An error due to the configuration
An exception thrown by the WCF service
Errors in configuration are often due to a mismatch between the client and the service. To avoid this place all configuration possible in a BindingConfiguration and copy and use this on both the client and the server. This I think is actually where your problem is, you are updating the service web.config, with things that also need to be in the client config. For eaxmple the max size, or having Buffered in one and Streamed in the other.
Errors throw by the service should be thrown as a FaultException and defined in the Contract as a FaultContract.
For remaining errors you need to look at the trace.svclog file as described in other posts. You need also to look at the event log and IIS log, the calls may be blocked before they reach the WCF service.
Try setting the maxRequestLength property:
<system.web>
<httpRuntime maxRequestLength="2147483647" />
</system.web>
For anyone still getting this problem - as usual there are a couple absolutely crucial things left out of the above discussion, without which there's no hope of finding an answer. Here's what took me 3 hours of rooting around online to find out.
To recap:
First, I got the dreaded Not Found error using WCF from a Silverlight service. No, it's not because the service isn't found. I was able to step thru the called service method clear to the end, including the return. Then the client side got an exception in the async End part of the call. No explanation. It's got NOTHING to do with the bindings etc.
Then I found forum messages like this one about using the trace viewer. Turns out I already had that configured, but was not getting any trace (so I figured my service must be ok, esp. since I could trace thru). Wrong, bongo boy. Then I found another message saying a little known fact is if you set up a trace listener to write to "C:\logs\mylog", you MUST first create C:\logs by hand. It will not do it for you.
Ok, now I get the log and bring it up in TraceViewer. Whereupon I get an "error message" about an unterminated string. Thirty minutes later I find another message saying, oh, everybody knows you must first end your local dev server in order to flush out the last of the messages. You know, the ones that actually tell you what went wrong?
Now I get to the real errors and look at each one of them: Throwing an Exception, RequestContext aborted, and Failed To Send Response message over http. Only the first one matters. Except of course when looking at the lower pane, it gave me no useful information at all except to say there was a Serialization error. Um, "where" would be nice.
At my wit's end, I suddenly notice there's a little XML tab in the lower pane, right next to the "Formatted" tab. When I click on that, for my ThrowingAnException message, there it is - a ginormous dump with a highly specific message that led me right to the problem:
System.ServiceModel.CommunicationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
There was an error while trying to serialize parameter :GetTimecardsWithAlertsResult. The InnerException message was 'Enum value '0' is invalid for type 'Timeclock.Web.ShiftManager.AlertType' and cannot be serialized. Ensure that the necessary enum values are present and are marked with EnumMemberAttribute attribute if the type has DataContractAttribute attribute.'. Please see InnerException for more details.
The problem was I had not initialized an enum-based member of a class, so it was 0, which was NOT one of my allowed enum values. Very simple to fix.
And clearly, very simple for Microsoft to detect, given that large amount of information they successfully hid from prying eyes for 3 hours.
Here's an idea Microsoft - how about you just offer a way to catch these errors and the all-important exception-message server side? Or let them fully pass through to the Silverlight client? You known, to make it easy to see what's going on, so I could fix this simple problem in the 3 seconds it deserves, instead of the 3 hours I have to charge my client for doing nothing useful?
Oh, I know. It's really hard because it's an async call over http, and the mean ol' internet makes it hurt your brain. But guess what? You're Microsoft. You have unlimited time and money. And you impact millions of people. When you screw around with this kind of s**t, like you do with thousands of scenarios where you just couldn't be bothered, you affect hundreds of thousands of developers across the planet.
Look around on StackOverflow. See how many people across the globe, smart people trying to write software to do useful important things, who are just not immersed in the incredible minutia like the above because, you know, they have real work to do.
Multiply my 3 hours on this stupid problem times tens of thousands of developers time 30-40 episodes of this sort of crap in a typical year, and you see what catastrophes you cause. That's fine to say "that's why we get paid the big bucks", but think about what actual good work we could all be accomplishing in the world if every time we turn around we didn't have to dive down a 3-hour s**t-hole you dug for us?
Microsoft, you are bad for programming, bad for business, and bad for humanity. I don't care how many computers run your software. You need to do better. Please start acting like you understand just how much you abuse minions of hard-working people in the world, in every country, every single day. And how much better a place you could make it if you just acted like it mattered to do things right.
Tim Johnson
You should get a specific communication exception on the client side.
I think that this exception that you're describing is an exception that is thrown after trying to reuse the client after it has faulted.
Try this:
On the server side config file set includeExceptionDetailInFaults="true"
When you're using the client side don't use the 'using pattern'. Check out this article.
I don't think you need tracing. Try the above and you'll be able to see the exact communication error.
Oh, and BTW is your client is Silverlight application?
If so then it's a bit more complicated... Check out this article.