I have been tracing messages using SvcTraceViewer.exe which were transmitted over MSMQ using a c# application, and have come across an informational event that I don't understand.
The raw XML looks like this:
<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
<EventID>262242</EventID>
<Type>3</Type>
<SubType Name="Information">0</SubType>
<Level>8</Level>
<TimeCreated SystemTime="2015-03-02T14:54:57.3176368Z" />
<Source Name="System.ServiceModel" />
<Correlation ActivityID="{f6fa4c52-6372-45ce-9171-1c5d789c3bf0}" />
<Execution ProcessName="MYPROCESS.EXE" ProcessID="12492" ThreadID="5" />
<Channel />
<Computer>MYCOMPUTER</Computer>
</System>
<ApplicationData>
<TraceData>
<DataItem>
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Information">
<TraceIdentifier>http://msdn.microsoft.com/en-GB/library/System.ServiceModel.Channels.MsmqPoolFull.aspx</TraceIdentifier>
<Description>Pool of the native MSMQ messages is full. This may affect performance.</Description>
<AppDomain>ASM.Sequoia.Reporting.ReportGenerator.exe</AppDomain>
</TraceRecord>
</DataItem>
</TraceData>
</ApplicationData>
</E2ETraceEvent>
The message 'Pool of the native MSMQ messages is full. This may affect performance.' is evident, but I can't find any meaningful information about what it might mean.
After a quick google, I have come across the following links - none of which really help me understand what this log is telling me, though the first one says that it can be ignored. As the blog is from 8 years ago, I thought I'd ask because I have no idea if the information is still up-to-date.
http://blogs.msdn.com/b/drnick/archive/2007/03/06/the-pool-is-full.aspx (says to ignore the info - but from 8 years ago)
https://msdn.microsoft.com/en-us/library/aa738731(v=vs.110).aspx (not very helpful MSDN post?!)
For info, I am using non-transactional queues, there are no queued messages (stuck in the queues or processing), nothing in the dead letter queues... The information I am sending over the queue is largish - can be up to a couple of MB in size as it contains a bunch of serialized images.
Does anyone know what this means?
I think that message is nothing to worry about. I took a look at WCF source code, and found it's just diagnostic message when some internal object pool is full. They use pool in order to reuse MsmqInputMessage objects. And when pool is full, they just leave that object to be garbage collected. It also seems they trace that same message when pool is empty, in which case they just create new object when that happens. So everything works correctly after these log messages, although a bit less optimally, i.e. some additional objects will be created and garbage collected.
Btw. you can browse .Net source online or download it and open some of subprojects in VS. It's here:
http://referencesource.microsoft.com/
And here are some of interesting files for your issue:
http://referencesource.microsoft.com/#System.ServiceModel/System/ServiceModel/Channels/MsmqInputMessagePool.cs
http://referencesource.microsoft.com/#System.ServiceModel/System/ServiceModel/Channels/MsmqOutputChannel.cs
You can search where MsmqDiagnostics.PoolFull method is invoked.
I see MessageEndpointMappings section in my App.config and it just keeps confusing me.
Sometimes it looks like it sets of a client to subscribe to another queue (like this one does):
<UnicastBusConfig>
<MessageEndpointMappings>
<add Messages="MyServiceBus.MessageHub.Contracts" Endpoint="MessageHub"/>
</MessageEndpointMappings>
</UnicastBusConfig>
But other times it seems to be used to configure outgoing messages to other queues:
<UnicastBusConfig>
<MessageEndpointMappings>
<add Messages="ServiceBus.MessageHub.InternalMessages" Endpoint="MessageHub"/>
</MessageEndpointMappings>
</UnicastBusConfig>
But really, there is no difference between the two of them. So how does it know when to setup a subscription and when to setup for sending messages?
Or is it really doing both all the time and I just don't use both?
Essentially, what you're declaring with the MessageEndpointMappings element, is "who is the owner of the specified messages?"
Another way to view it, is that this is the direction of the service dependency - not the message flow, because that is determined by whether messages get Sendt or Publishd.
Therefore, in order to be able to send messages and subscribe to messages from a given service, you declare an endpoint mapping in the sender/subscriber end.
And then, when the sender/subscriber has an endpoint mapping that points to another service and a handler for a message included in that endpoint mapping, NServiceBus is nice enough to automatically subscribe to that message (unless you disable the auto-subscription with DoNotAutoSubscribe()) - because, logically, that must mean that the sender/subscriber wishes to subscribe to the given message.
When you think about it this way, I think it makes sense. Hope you feel so too :)
I am using apache servicemix and apache activeMQ in my product. Here in case of HttpConnector i found a performance issue.
If i increase number of total request at a time then as the number will increase exchange queue gets stuck on any of the component. After all the requests start processing, most of the exchange gets stuck at the end component or at the dispatcher* component. Container's heap size reaches to very high and after some time it crashes and restarts automatically.
I have used transactionmanager also. flowname is also mentioned as jms.
need some solution urgently.
You might try using KahaDB instead of the "amqPersistenceAdapter". We saw a huge throughput increase just by switching to this.
here is the config we used (which is highly dependent on your application, but make sure "enableJournalDiskSyncs" is set to false)
<persistenceAdapter>
<kahaDB directory="../data/kaha"
enableJournalDiskSyncs="false"
indexWriteBatchSize="10000"
indexCacheSize="1000" />
</persistenceAdapter>
If i increase number of total request at a time then as the number will increase exchange queue gets stuck on any of the component.
ActiveMQ has a mechanism to stop producer writing messages, it is called "flow controll", seems that you producer is faster than consumer (or consumer is not stable it its speed), so first check the memoryLimit config for your AMQ (also defining it or special queue possible). Try to increase it.
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic="FOO.>" producerFlowControl="false" memoryLimit="1mb">
<dispatchPolicy>
<strictOrderDispatchPolicy/>
</dispatchPolicy>
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
</policyEntry>
</policyEntries>
</policyMap>
Also, you can disable that stopping of processing incoming messages with producerFlowControl="false" option. So, in case when all buffer will be used, AMQ will slow down and all messages will be stored to HD. More info Producer Flow Control and Message Cursors
Container's heap size reaches to very high and after some time it crashes and restarts automatically.
But anyway, it is just way of tunning of your app, it is not solution, because always you will have the case when some resources run out :)
You should limit the incoming requests or balance them f.e. using Pound
I'm doing testing against some software I've written. The test enqueues messages into MSMQ via WCF at a rate faster than than my software can dequeue and process them. This shouldn't be a problem, since that is MSMQ's intended purpose, but if I enqueue enough messages to where it's taking my software more than 24 hours to process, those messages will get moved to the "Transactional dead-letter messages" queue and have their Class set to "The time-to-be-received has elapsed".
The only configurable that I can find is on the binding itself:
<bindings>
<netMsmqBinding>
<binding timeToLive="7.00:00:00" /> <!-- 7 days -->
...
I use this binding both when enqueuing and dequeuing and it doesn't seem to do the trick. Setting the value 2 seconds does have an effect, but setting it longer than 1 day, including to its max value (24 days) does not.
Is there another way to lengthen this time-to-be-received window? I can't find anything else to configure (when sending the message or creating the queue).
The timeToLive attribute on the binding itself is, in fact, the only configurable necessary. I went back through all my configurations and apparently missed a spot. From "Programming WCF Services":
The TimeToLive property is only
relevant to the posting client, and
has no affect on the service side, nor
can the service change it. TimeToLive
defaults to one day.
I've had the test running all weekend now, progressing 1,000,000 messages. Nothing has ended up in the dead-letter queue yet.
I'm not 100% sure, but I believe that the TimeToLive property only sets the Time-To-Reach-Queue msmq property, but I don't know of a built-in way right now of setting the Time-To-Be-Received property...
In my WCF service, when trying transfer large data I constantly get an error: The underlying connection was closed: The connection was closed unexpectedly
I want to know what particular reason invokes this error, so I set up WCF Tracing and can read traces.svclog file.
The problem is, that I can see in this file a lot of information about flow of processes, I can see exact time when exception is appeared, but I can't see the exact reason for that. Is it due to MaxReceivedMessageSize or something like that.
Is it so that traces.svclog can not contain such information or am I doing something wrong?
How such information could be obtained?
Edited (added):
From my server-side app.config:
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="NAVBinding_ICustomer_Service"
closeTimeout="01:50:00"
openTimeout="01:50:00" receiveTimeout="01:50:00" sendTimeout="01:50:00"
allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
maxBufferSize="2147483647" maxBufferPoolSize="2147483647"
maxReceivedMessageSize="2147483647" messageEncoding="Text"
textEncoding="utf-8" transferMode="Buffered" useDefaultWebProxy="true">
<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647"
maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
</binding>
</basicHttpBinding>
</bindings>
<services>
<service name = "Customer_Service" behaviorConfiguration="returnFaults">
<endpoint name="NAVBinding_ICustomer_Service"
address = "http://localhost:8000/nav/customer"
binding = "basicHttpBinding"
bindingConfiguration= "NAVBinding_ICustomer_Service"
contract = "NAVServiceReference.ICustomer_Service"/>
</service>
</services>
<behaviors>
<serviceBehaviors>
<behavior name="returnFaults" >
<serviceDebug includeExceptionDetailInFaults="true" />
<serviceMetadata httpGetEnabled="true" />
</behavior>
</serviceBehaviors>
</behaviors>
</system.serviceModel>
Edited (added):
What is the right and best way to turn WCF service from a "black box" to an easily troubleshooted service, which tells the reason why something goes not the expected way?
What tools, techniques you use to troubleshoot WCF service?
Ignoring the problems with the maxRequestLength (which have been answered by others),
I will have a go at answering your original question about how to troubleshoot WCF.
If you are already using the Service Trace Viewer (I couldn't tell from the question
if you were just viewing them by hand) - it is possible that all the details aren't
going into the file.
When I want to get really hardcore, I enable all the logging parameters for
message logging. (This will generate some big service logs though so don't leave
it on)
<system.serviceModel>
<diagnostics>
<messageLogging logEntireMessage="true" logMalformedMessages="true" logMessagesAtServiceLevel="true" logMessagesAtTransportLevel="true" maxMessagesToLog="-1" />
</diagnostics>
</system.serviceModel>
If you are not using the Microsoft Service Trace Viewer I recommend that. It
gives all the information I need to track down those tricky message handshake, message
size exceptions etc. Here is an MSDN reference to get you started
http://msdn.microsoft.com/en-us/library/aa751795.aspx
Trace interactions that have potential problems are hilighted in yellow on the
left, and the detailed pane on the top right will normally hilight the exceptional
service event in red. Sometimes you will get multiple problems as the inner
error cascades through the service stack - but you can see it all in the
trace viewer.
If you get nothing in your server 'service log', then it is possible your exceptions are entirely at the client end - theoretically you could exceed some of the client
side security parameters (message size etc) before any message has actually reached the
web service end - but client problems are generally easier to track down because you know you only have to worry about editing the config file at the client end (i.e. it's not because of any interaction between the client and server settings).
I've spent the last 2+ days trying to find why I'm getting "The underlying connection was closed: The connection was closed unexpectedly" with a method call returning with more data vs when not so much data (i.e., it works ok with smaller sets of data being returned only).
My error messsages are slightly different (perhaps due to framework differences) but wanted to share the cause I found. First, I would like to state that while tracing and increasing the sizes of certain things in config files given as answers above may be helpful for tracking down WCF errors, these things did nothing to help me determine the real cause of the error.
By just looking at the exception thrown and up the chain, I could see the following root, error:
"An existing connection was forcibly closed by the remote host" - this was a System.Net.Sockets.SocketException
Going up the call chain then was:
"Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host." - a System.IO.IOException, then
"The underlying connection was closed: An unexpected error occurred on a receive." - a System.Net.WebException, then finally what was the caught exception's message,
"An error occurred while receiving the HTTP response to . This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details." - a System.ServiceModel.CommunicationException
The enabling tracing then viewing the trace logs with the TraceViewer did make this easier to see, but never told me the real cause for my "An existing connection was forcibly closed by the remote host" error.
In my case, my WCF service is hosted on IIS6 and only when I contacted our institutional support in charge of these servers and asked them to look in the system event logs, did I immediately see the answer - a System.OutOfMemoryException!
My WCF service runs in an allocated 200MB of RAM and my method was consuming more than this. I looked in my method and found eventually a block of code that should have been outside/below the block (loop) it was in. . ..so I had an exponential type of collection being generated in my method.
Hope this may help others.
To answer your question how to create an easily trouble shooted WCF service. One way is to minimise the number of potential errors, so that you have fewer things to look at while troubleshooting.
There are two main sources of errors:
An error due to the configuration
An exception thrown by the WCF service
Errors in configuration are often due to a mismatch between the client and the service. To avoid this place all configuration possible in a BindingConfiguration and copy and use this on both the client and the server. This I think is actually where your problem is, you are updating the service web.config, with things that also need to be in the client config. For eaxmple the max size, or having Buffered in one and Streamed in the other.
Errors throw by the service should be thrown as a FaultException and defined in the Contract as a FaultContract.
For remaining errors you need to look at the trace.svclog file as described in other posts. You need also to look at the event log and IIS log, the calls may be blocked before they reach the WCF service.
Try setting the maxRequestLength property:
<system.web>
<httpRuntime maxRequestLength="2147483647" />
</system.web>
For anyone still getting this problem - as usual there are a couple absolutely crucial things left out of the above discussion, without which there's no hope of finding an answer. Here's what took me 3 hours of rooting around online to find out.
To recap:
First, I got the dreaded Not Found error using WCF from a Silverlight service. No, it's not because the service isn't found. I was able to step thru the called service method clear to the end, including the return. Then the client side got an exception in the async End part of the call. No explanation. It's got NOTHING to do with the bindings etc.
Then I found forum messages like this one about using the trace viewer. Turns out I already had that configured, but was not getting any trace (so I figured my service must be ok, esp. since I could trace thru). Wrong, bongo boy. Then I found another message saying a little known fact is if you set up a trace listener to write to "C:\logs\mylog", you MUST first create C:\logs by hand. It will not do it for you.
Ok, now I get the log and bring it up in TraceViewer. Whereupon I get an "error message" about an unterminated string. Thirty minutes later I find another message saying, oh, everybody knows you must first end your local dev server in order to flush out the last of the messages. You know, the ones that actually tell you what went wrong?
Now I get to the real errors and look at each one of them: Throwing an Exception, RequestContext aborted, and Failed To Send Response message over http. Only the first one matters. Except of course when looking at the lower pane, it gave me no useful information at all except to say there was a Serialization error. Um, "where" would be nice.
At my wit's end, I suddenly notice there's a little XML tab in the lower pane, right next to the "Formatted" tab. When I click on that, for my ThrowingAnException message, there it is - a ginormous dump with a highly specific message that led me right to the problem:
System.ServiceModel.CommunicationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
There was an error while trying to serialize parameter :GetTimecardsWithAlertsResult. The InnerException message was 'Enum value '0' is invalid for type 'Timeclock.Web.ShiftManager.AlertType' and cannot be serialized. Ensure that the necessary enum values are present and are marked with EnumMemberAttribute attribute if the type has DataContractAttribute attribute.'. Please see InnerException for more details.
The problem was I had not initialized an enum-based member of a class, so it was 0, which was NOT one of my allowed enum values. Very simple to fix.
And clearly, very simple for Microsoft to detect, given that large amount of information they successfully hid from prying eyes for 3 hours.
Here's an idea Microsoft - how about you just offer a way to catch these errors and the all-important exception-message server side? Or let them fully pass through to the Silverlight client? You known, to make it easy to see what's going on, so I could fix this simple problem in the 3 seconds it deserves, instead of the 3 hours I have to charge my client for doing nothing useful?
Oh, I know. It's really hard because it's an async call over http, and the mean ol' internet makes it hurt your brain. But guess what? You're Microsoft. You have unlimited time and money. And you impact millions of people. When you screw around with this kind of s**t, like you do with thousands of scenarios where you just couldn't be bothered, you affect hundreds of thousands of developers across the planet.
Look around on StackOverflow. See how many people across the globe, smart people trying to write software to do useful important things, who are just not immersed in the incredible minutia like the above because, you know, they have real work to do.
Multiply my 3 hours on this stupid problem times tens of thousands of developers time 30-40 episodes of this sort of crap in a typical year, and you see what catastrophes you cause. That's fine to say "that's why we get paid the big bucks", but think about what actual good work we could all be accomplishing in the world if every time we turn around we didn't have to dive down a 3-hour s**t-hole you dug for us?
Microsoft, you are bad for programming, bad for business, and bad for humanity. I don't care how many computers run your software. You need to do better. Please start acting like you understand just how much you abuse minions of hard-working people in the world, in every country, every single day. And how much better a place you could make it if you just acted like it mattered to do things right.
Tim Johnson
You should get a specific communication exception on the client side.
I think that this exception that you're describing is an exception that is thrown after trying to reuse the client after it has faulted.
Try this:
On the server side config file set includeExceptionDetailInFaults="true"
When you're using the client side don't use the 'using pattern'. Check out this article.
I don't think you need tracing. Try the above and you'll be able to see the exact communication error.
Oh, and BTW is your client is Silverlight application?
If so then it's a bit more complicated... Check out this article.