We have an application that has a WCF service (*.svc) running on IIS7 and various clients querying the service. The server is running Win 2008 Server. The clients are running either Windows 2008 Server or Windows 2003 server. I am getting the following exception, which I have seen can in fact be related to a large number of potential WCF issues.
System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:59.9320000. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://www.domain.com/WebServices/myservice.svc/gzip' has exceeded the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout.
I have increased the timeout to 30min and the error still occurred. This tells me that something else is at play, because the quantity of data could never take 30min to upload or download.
The error comes and goes. At the moment, it is more frequent. It does not seem to matter if I have 3 clients running simultaneously or 100, it still occurs once in a while. Most of the time, there are no timeouts but I still get a few per hour. The error comes from any of the methods that are invoked. One of these methods does not have parameters and returns a bit of data. Another takes in lots of data as a parameter but executes asynchronously. The errors always originate from the client and never reference any code on the server in the stack trace. It always ends with:
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
On the server:
I've tried (and currently have) the following binding settings:
maxBufferSize="2147483647" maxReceivedMessageSize="2147483647" maxBufferPoolSize="2147483647"
It does not seem to have an impact.
I've tried (and currently have) the following throttling settings:
<serviceThrottling maxConcurrentCalls="1500" maxConcurrentInstances="1500" maxConcurrentSessions="1500"/>
It does not seem to have an impact.
I currently have the following settings for the WCF service.
[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Single)]
I ran with ConcurrencyMode.Multiple for a while, and the error still occurred.
I've tried restarting IIS, restarting my underlying SQL Server, restarting the machine. All of these don't seem to have an impact.
I've tried disabling the Windows firewall. It does not seem to have an impact.
On the client, I have these settings:
maxReceivedMessageSize="2147483647"
<system.net>
<connectionManagement>
<add address="*" maxconnection="16"/>
</connectionManagement>
</system.net>
My client closes its connections:
var client = new MyClient();
try
{
return client.GetConfigurationOptions();
}
finally
{
client.Close();
}
I have changed the registry settings to allow more outgoing connections:
MaxConnectionsPerServer=24, MaxConnectionsPer1_0Server=32.
I have now just recently tried SvcTraceViewer.exe. I managed to catch one exception on the client end. I see that its duration is 1 minute. Looking at the server side trace, I can see that the server is not aware of this exception. The maximum duration I can see is 10 seconds.
I have looked at active database connections using exec sp_who on the server. I only have a few (2-3). I have looked at TCP connections from one client using TCPview. It usually is around 2-3 and I have seen up to 5 or 6.
Simply put, I am stumped. I have tried everything I could find, and must be missing something very simple that a WCF expert would be able to see. It is my gut feeling that something is blocking my clients at the low-level (TCP), before the server actually receives the message and/or that something is queuing the messages at the server level and never letting them process.
If you have any performance counters I should look at, please let me know. (please indicate what values are bad, as some of these counters are hard to decypher). Also, how could I log the WCF message size? Finally, are there any tools our there that would allow me to test how many connections I can establish between my client and server (independently from my application)
Thanks for your time!
Extra information added June 20th:
My WCF application does something similar to the following.
while (true)
{
Step1GetConfigurationSettingsFromServerViaWCF(); // can change between calls
Step2GetWorkUnitFromServerViaWCF();
DoWorkLocally(); // takes 5-15minutes.
Step3SendBackResultsToServerViaWCF();
}
Using WireShark, I did see that when the error occurs, I have a five TCP retransmissions followed by a TCP reset later on. My guess is the RST is coming from WCF killing the connection. The exception report I get is from Step3 timing out.
I discovered this by looking at the tcp stream "tcp.stream eq 192". I then expanded my filter to "tcp.stream eq 192 and http and http.request.method eq POST" and saw 6 POSTs during this stream. This seemed odd, so I checked with another stream such as tcp.stream eq 100. I had three POSTs, which seems a bit more normal because I am doing three calls. However, I do close my connection after every WCF call, so I would have expected one call per stream (but I don't know much about TCP).
Investigating a bit more, I dumped the http packet load to disk to look at what these six calls where.
1) Step3
2) Step1
3) Step2
4) Step3 - corrupted
5) Step1
6) Step2
My guess is two concurrent clients are using the same connection, that is why I saw duplicates. However, I still have a few more issues that I can't comprehend:
a) Why is the packet corrupted? Random network fluke - maybe? The load is gzipped using this sample code: http://msdn.microsoft.com/en-us/library/ms751458.aspx - Could the code be buggy once in a while when used concurrently? I should test without the gzip library.
b) Why would I see step 1 & step 2 running AFTER the corrupted operation timed out? It seems to me as if these operations should not have occurred. Maybe I am not looking at the right stream because my understanding of TCP is flawed. I have other streams that occur at the same time. I should investigate other streams - a quick glance at streams 190-194 show that the Step3 POST have proper payload data (not corrupted). Pushing me to look at the gzip library again.
If you are using .Net client then you may not have set
//This says how many outgoing connection you can make to a single endpoint. Default Value is 2
System.Net.ServicePointManager.DefaultConnectionLimit = 200;
here is the original question and answer WCF Service Throttling
Update:
This config goes in .Net client application may be on start up or whenever but before starting your tests.
Moreover you can have it in app.config file as well like following
<system.net>
<connectionManagement>
<add maxconnection = "200" address ="*" />
</connectionManagement>
</system.net>
If you havn't tried it already - encapsulate your Server-side WCF Operations in try/finally blocks, and add logging to ensure they are actually returning.
If those show that the Operations are completing, then my next step would be to go to a lower level, and look at the actual transport layer.
Wireshark or another similar packet capturing tool can be quite helpful at this point. I'm assuming this is running over HTTP on standard port 80.
Run Wireshark on the client. In the Options when you start the capture, set the capture filter to tcp http and host service.example.com - this will reduce the amount of irrelevant traffic.
If you can, modify your client to notify you the exact start time of the call, and the time when the timeout occurred. Or just monitor it closely.
When you get an error, then you can trawl through the Wireshark logs to find the start of the call. Right click on the first packet that has your client calling out on it (Should be something like GET /service.svc or POST /service.svc) and select Follow TCP Stream.
Wireshark will decode the entire HTTP Conversation, so you can ensure that WCF is actually sending back responses.
from: http://www.codeproject.com/KB/WCF/WCF_Operation_Timeout_.aspx
To avoid this timeout error, we need
to configure the OperationTimeout
property for Proxy in the WCF client
code. This configuration is something
new unlike other configurations such
as Send Timeout, Receive Timeout etc.,
which I discussed early in the
article. To set this operation timeout
property configuration, we have to
cast our proxy to IContextChannel in
WCF client application before calling
the operation contract methods.
I'm having a very similar problem. In the past, this has been related to serialization problems. If you are still having this problem, can you verify that you can correctly serialize the objects you are returning. Specifically, if you are using Linq-To-Sql objects that have relationships, there are known serialization problems if you put a back reference on a child object to the parent object and mark that back reference as a DataMember.
You can verify serialization by writing a console app that serializes and deserializes your objects using the DataContractSerializer on the server side and whatever serialization methods your client uses. For example, in our current application, we have both WPF and Compact Framework clients. I wrote a console app to verify that I can serialize using a DataContractSerializer and deserialize using an XmlDesserializer. You might try that.
Also, if you are returning Linq-To-Sql objects that have child collections, you might try to ensure that you have eagerly loaded them on the server side. Sometimes, because of lazy loading, the objects being returned are not populated and may cause the behavior you are seeing where the request is sent to the service method multiple times.
If you have solved this problem, I'd love to hear how because I'm stuck with it too. I have verified that my issue is not serialization so I'm at a loss.
UPDATE: I'm not sure if it will help you any but the Service Trace Viewer Tool just solved my problem after 5 days of very similar experience to yours. By setting up tracing and then looking at the raw XML, I found the exceptions that were causing my serialization problems. It was related to Linq-to-SQL objects that occasionally had more child objects than could be successfully serialized. Adding the following to your web.config file should enable tracing:
<sharedListeners>
<add name="sharedListener"
type="System.Diagnostics.XmlWriterTraceListener"
initializeData="c:\Temp\servicetrace.svclog" />
</sharedListeners>
<sources>
<source name="System.ServiceModel" switchValue="Verbose, ActivityTracing" >
<listeners>
<add name="sharedListener" />
</listeners>
</source>
<source name="System.ServiceModel.MessageLogging" switchValue="Verbose">
<listeners>
<add name="sharedListener" />
</listeners>
</source>
</sources>
The resulting file can be opened with the Service Trace Viewer Tool or just in IE to examine the results.
Are you closing the connection to the WCF service in between requests? If you don't, you'll see this exact timeout (eventually).
Did you try using clientVia to see the message sent, using SOAP toolkit or something like that? This could help to see if the error is coming from the client itself or from somewhere else.
Did you check the WCF traces? WCF has a tendency to swallow exceptions and only return the last exception, which is the timeout that you're getting, since the end point didn't return anything meaningful.
I've just solved the problem.I found that the nodes in the App.config file have configed wrong.
<client>
<endpoint name="WCF_QtrwiseSalesService" binding="wsHttpBinding" bindingConfiguration="ws" address="http://cntgbs1131:9005/MyService/TGE.ISupplierClientManager" contract="*">
</endpoint>
</client>
<bindings>
<wsHttpBinding>
<binding name="ws" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647" messageEncoding="Text">
<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647"/>
<**security mode="None">**
<transport clientCredentialType="None"></transport>
</security>
</binding>
</wsHttpBinding>
</bindings>
Confirm your config in the node <security>,the attribute "mode" value is "None". If your value is "Transport",the error occurs.
You will also receive this error if you are passing an object back to the client that contains a property of type enum that is not set by default and that enum does not have a value that maps to 0. i.e enum MyEnum{ a=1, b=2};
Looks like this exception message is quite generic and can be received due to a variety of reasons. We ran into this while deploying the client on Windows 8.1 machines. Our WCF client runs inside of a windows service and continuously polls the WCF service. The windows service runs under a non-admin user. The issue was fixed by setting the clientCredentialType to "Windows" in the WCF configuration to allow the authentication to pass-through, as in the following:
<security mode="None">
<transport clientCredentialType="Windows" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
I'm not a WCF expert but I'm wondering if you aren't running into a DDOS protection on IIS.
I know from experience that if you run a bunch of simultaneous connections from a single client to a server at some point the server stops responding to the calls as it suspects a DDOS attack.
It will also hold the connections open until they time-out in order to slow the client down in his attacks.
Multiple connection coming from different machines/IP's should not be a problem however.
There's more info in this MSDN post:
http://msdn.microsoft.com/en-us/library/bb463275.aspx
Check out the MaxConcurrentSession sproperty.
Related
I am getting the following error in WCF how can I repro on my test environment and what is the fix for the problem ? How could I set timeout in milliseconds to reproduce in test environment .
The request channel timed out while waiting for a reply after 00:00:59.9843740.
Increase the timeout value passed to the call to Request or
increase the Send Timeout value on the Binding.
The time allotted to this operation may have been a portion of a longer timeout
The issue is unrelated to the SendTimeOut property, I don’t think the following code segments could solve the problem.
closeTimeout="00:10:00" openTimeout="00:10:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"
It more relates to your actual scenery.
For example, the client used multiple Stream objects and did not close the channel appropriately. there is something wrong with the Nettcpbinding configuration,authentication, ReadQuotas, and so on.
I would like you could post your complete configuration, try to reproduce your issue.
Besides, try to apply the below service behavior in you binding property.
<serviceThrottling maxConcurrentCalls="20" maxConcurrentSessions="20" maxConcurrentInstances="30" />
Feel free to let me know if the problem still exists.
I have a WCF Web Service hosted in a Windows Service.
The service is configured using the following attributes (VB)
<ServiceBehavior(ConcurrencyMode:=ConcurrencyMode.Multiple,
InstanceContextMode:=InstanceContextMode.PerCall,
UseSynchronizationContext:=False)>
I created a test client, which creates 50 threads at once and and I am starting them straight away, all querying an operation on the WCF Service. The first 30 or so openings (client.open()) work, but afterwards I get an error message on my client that the opening couldn't be done because there was a timeout while trying to open the service and thus I get an EndPointNotFoundException. (The operationss I am calling are all synchronous if that matters.)
Can I expect that sort of behaviour and thus simply have to live with these exceptions or do I have to look at some of my settings, either the above, or any other settings? (I am using a NetTcpBinding.)
What I see is that the client works on the request of the at once created threads simultaneously, so the above settings have the desired effect of making the service multi-user-at-the-same-time-enabled.
But ideally I wouldn't get timeout exceptions. Rather I would want the clients to wait longer to get their response than to time out.
Any ideas?
Thanks all!
Check settings such as these:
<behavior name="CommonServices_Behavior">
<serviceThrottling maxConcurrentCalls="100" maxConcurrentSessions="50"
maxConcurrentInstances="50" />
</behavior>
I seem to recall that some of these values have defaults set in your machine.config file. There is also a MaxConnections value to check for NetTcpBinding.
I have a WCF 4.0 service deployed on a 2K8R2 dev server and hosted under IIS 7.5. I'm calling it locally from a test app (WPF). I'm running into a problem sending largish files (via a byte array) where I get the following error when attempting to send a 23MB (or larger) file.
There was no endpoint listening at
http:///FileStorageClone/FileStorage.svc that
could accept the message. This is often caused by an incorrect address
or SOAP action. See InnerException, if present, for more details.
Up to a 19MB works fine. I've upped the timeout settings and set max buffer, received message and array sizes all at 2GB in the web.config file. The exception is thrown almost immediately (not long enough to have timed out).
It's running using basicHttpBinding, though as that's solidified next steps will be other binding types.
I first worked out the connections with small files, then moved to larger. At 64KB I ran into max buffer and received message sizes. At 4MB I learned I needed to bump up the httpRuntime under system.Web from the default 4GB value.
It feels like an IIS or site-level issue to me. Any idea where I may be missing?
It looks like this is a setting more at the IIS level. Setting requestFiltering in the web.config or via appcmd.exe are two viable options. In the web.config, the following example sets the limit to 2.2 billion bytes:
<system.webServer>
<security>
<requestFiltering>
<requestLimits maxAllowedContentLength="2200000000" />
</requestFiltering>
</security>
</system.webServer>
Thanks to: http://www.dantor.com/support/misc/web-config-requestFiltering-user-agent.aspx
Also, via appcmd.exe, see: http://forums.iis.net/t/1066272.aspx
I've got a duplex WCF service which hangs after the magic 10 proxy instantiations. The specific error on the client is:
"System.TimeoutException: This request operation sent to net.tcp://localhost:8080/RoomService/netTcp did not receive a reply within the configured timeout (00:00:59.9960000)".
There aren't any apparent error messages on the server.
Note that this isn't the standard, obvious problem, i.e., failing to close my proxy connections, as I'm closing every instance of my proxy connection appropriately before opening the next one:
try
{
client.Close();
}
catch (CommunicationException)
{
client.Abort();
}
catch (TimeoutException)
{
client.Abort();
}
catch (Exception)
{
client.Abort();
throw;
}
And I've set my throttling behavior to 500 simultaneous everything:
ServiceThrottlingBehavior throttlingBehavior = new ServiceThrottlingBehavior()
{
MaxConcurrentCalls = 500,
MaxConcurrentSessions = 500,
MaxConcurrentInstances = 500
};
I've set the ConcurrencyMode of my service to Multiple, and I've tried all three possible values for InstanceContextMode.
[ServiceBehavior(InstanceContextMode=InstanceContextMode.PerSession, ConcurrencyMode = ConcurrencyMode.Multiple)]
I've tried self-hosting the service, and hosting it within IIS, and I get the same results on each.
I've tried the NetTcpBinding, the WSDualHttpBinding, and the PollingDuplexBinding (on Silverlight), with the same results on each. I can't try the BasicHttpBinding or the WSHttpBinding, as this is a duplex service.
There was one place in my code where I was launching multiple threads (to execute multiple callbacks simultaneously), but for troubleshooting purposes I've commented that bit out, and it hasn't made a difference.
On the client, I've tried using new proxies for each test, and reusing the same proxy across all tests, but without any luck. I've tried creating a new InstanceContext for each proxy, and reusing the same InstanceContext across all proxies, and again, no luck.
Whatever I do, after the 10th test executed in my test harness, the next call to the service hangs.
Any thoughts on what I might be doing wrong?
OK, so I made at least one stupid mistake: I was creating the throttling behavior, but was neglecting to add it to the service proper. It's now added correctly:
ServiceThrottlingBehavior throttlingBehavior = new ServiceThrottlingBehavior()
{
MaxConcurrentCalls = 500,
MaxConcurrentSessions = 500,
MaxConcurrentInstances = 500
};
base.Description.Behaviors.Add(throttlingBehavior);
And now I can run more than 10 tests, and hence my immediate problem is solved.
But I'm still puzzled as to why I'm running into this problem at all, since I'm specifically closing one proxy before moving on to the next. A MaxConcurrentXXX of 2 ought to be working in this scenario; I shouldn't need a MaxConcurrentXXX of 500. I'm a tad worried about scalability if every client that connects to the server continues to chew up a connection beyond the time that it's actually connected.
Maybe I'm making a stupid mistake somewhere else -- it wouldn't be the first time -- but I've specifically stepped through the code that's closing the proxy, and it's definitely getting called.
I've had this happen when I had a semaphore on the server-side which wasn't being released after a client was done being serviced.
Are any server-side resources or locks not being released properly? Since your service instance is per session, I'd suspect that the server object is hanging around and holding a lock. What if you change the behavior to per call?
I don't have an answer but I do have some debugging advice.
1) Attach the visual studio debugger to the service process and see if it can catch whatever is going on.
2) Configure the service behavior to pass exception information back to the client and see if the service is throwing an exception that isn't being reported:
<behaviors>
<serviceBehaviors>
<behavior name="ServiceBehavior">
<serviceDebug includeExceptionDetailInFaults="True"/>
</behavior>
</serviceBehaviors>
</behaviors>
3) Turn on service logging with ActivityTracing enabled and use the Service Trace Viewer (from Windows SDK) to analyze the log and see if anything pops up
<system.diagnostics>
<trace autoflush="true" />
<sources>
<source name="System.ServiceModel" switchValue="Information, ActivityTracing" propagateActivity="true">
<listeners>
<add name="sdt" type="System.Diagnostics.XmlWriterTraceListener" initializeData="Service.svclog" />
</listeners>
</source>
</sources>
</system.diagnostics>
4) Isolate the service wrapper from the functional code and see if the service still hangs. If it doesn't then incrementally add functionality back until you figure out what is making it hang
5) If you are using an HTTP binding, proxy the service with fiddler and log the http traffic.
6) Try Hosting the WCF service in a Managed Windows Service and attach a debugger to the service process after you start it up.
Take a look at ServiceBehavior.AutomaticSessionShutdown.
At the begining of the Callback function in the client use
OperationContext.Current.Channel.Close();
This will solev the problem.
We are getting some weird behavior from a service we have deployed on a remote system which we are using to access and post data to.
Retrieving information from the service doesn't seem to be an issue. However, whenever we try to execute insert methods on the service we get an System.ServiceModel.ProtocolException. The weird thing is, it will successfully execute approximate 40 records before faulting.
The client is communicating over SSL and TransportWithMessageSecurity is enabled on the wsHttpBinding.
The error message is being generated at the "Set up Secure Session activity":
System.ServiceModel.ProtocolException,
System.ServiceModel, Version=3.0.0.0,
Culture=neutral,
PublicKeyToken=b77a5c561934e089
The content type text/html of the
response message does not match the
content type of the binding
(application/soap+xml; charset=utf-8).
If using a custom encoder, be sure
that the IsContentTypeSupported method
is implemented properly.
We have tried a number of tests all of which consistently fault as soon as we reach a particular number of inserts.
Any suggestions would be appreciated.
I wonder if the trace viewer might shed some light here? No idea what the problem is, I'm afraid...
If a piece of code sends you HTML, then you should probably read it to see what it says.
You probably have reached the configured message size and the WCF infrastructure is truncating the SOAP message, making the serializer (Contract or XML Serializer) to not be able to deserialize the message.
Try increasing the WCF request size see < readerQuotas >
ASP.NET might not be configured properly for .NET 4.0
navigate where .NET 4.0 is installed and try running the following command to reinstall it:
C:\Windows\Microsoft.NET\Framework64\v4.0.30319>aspnet_regiis.exe -iru
This could help somebody. My problem was: I have a web site consuming a WCF service.
After deploying the web site, I forgot to deploy the WCF service, too.
After deploying the WCF service, all worked fine.
I ran into the same problem. Finally, I got a solution by adding some code in the web.config file.
I added these under the <basicHttpBinding> tag:
<binding name="xyz">
<security mode= "Transport" />
</binding>