We have a Silverlight client which runs "out-of-browser" on a Mac. This client consumes a WCF service through a polling duplex binding.
In the client I am listening to the "Faulted" event exposed by the "InnerChannel" property of the System.ServiceModel.DuplexClientBase derivative which represents the service in the client side.
After exactly one minute this "Faulted" event is triggered and after that the channel is not working anymore, i.e. when the server tries to send a message through the callback channel it gets a timeout exception.
Here is a theory I have: I suspect that the underlying polling operation in the client has a timeout of 1 minute. In the server side the serverPollTimeout property of the pollingDuplexHttpBinding section is set to more than one minute. This means that the server holds a poll request for more than one minute if it has nothing to tell to the client during that time. I suspected that this revealed to the timeout in the client poll message. To test my theory, I've reduced the serverPollTimeout setting to less than a minute and indeed – the problem is not shown.
In the client-side, there is the PollingDuplexBindingElement.ClientPollTimeout property which is according to this blog exactly the setting which should tell the client to wait for more than a minute. The default for this setting is 5 minutes and I've even set it explicitly – but the problem still remains (without the workaround as described above).
Please note that this problem happens only on Mac out of browser client.
To sum-up, here are my questions:
How/where can I see a descriptive error message which tells exactly what is the problem here?
Why does it happen only in Mac out of browser client?
Can someone confirm my theory?
If my theory is true – how can I really set the timeout for the polling request in the client?
After a long thread regarding this with Microsoft support here are the conclusions regarding this issue:
The problem is relevant also for "regular" WCF calls. If we call a regular WCF operation from SL Mac OOB, it will timeout after exactly one minute, even-though the timeout is set to be higher
Microsoft confirmed that 60 seconds is the default timeout on Mac OS and that the SL runtime doesn't call the required Mac OS API to set it to higher value even-though the SL client code uses the documented SL API for changing the timeout. They say it’s "by design" since they are not aware of any scenarios where the server might take that long even in a long poll
Related
I am using Visual Studio 2012 to generate a web service to be used by a winforms client. I created the client side by using "add service reference". This winforms client is a .net c# replacement of an old VB 6 app. Previously, in the VB app there were external settings for timeout values including the following:
DNS timeout
Connect timeout
Request timeout
The DNS timeout would work when the endpoint host address is a FQDN forcing a DNS lookup. The timeout value here would place a limit on the amount of time to wait for DNS resolution.
The connect timeout would place a limit on the amount of time the winforms client would wait to establish an http connection to the server. DNS lookup would have been successful.
The request timeout would place a limit on the amount of time to wait for the request to return after an http connection was successful. This would come into play if a long running query took too long after the web service call was initiated.
Is there something similar to the above in .net 4.0. I would like to be able to configure this in the app.config. I do know about the below.
<bindings>
<basicHttpBinding>
<binding name="IncreasedTimeout"
openTimeout="12:00:00"
receiveTimeout="12:00:00" closeTimeout="12:00:00"
sendTimeout="12:00:00">
</binding>
</basicHttpBinding>
Could these map to the ones I need or does it really not matter?
thanks
The OpenTimeout setting for the WCF binding is for the length of time to wait when opening the channel, so I believe this will be analogous to your old Connect timeout. This should be fast so you normally would only want to specify a few seconds to wait (30 or less), not 12 hours.
The WCF CloseTimeout is for when a Close Channel message is sent, and this is how long to wait for an acknowledgement. This may not have an equivalent in your old architecture. Again, this should be fast and should only need a few seconds.
The WCF SendTimeout (for the client config) essentially covers the time for the Client to send the message to the service, and to receive back the response (if any). This would correspond to your old Request timeout. This may need to be for several minutes if your server takes a while to process things.
The WCF SendTimeout (for the server config) is for when you want callbacks, so that the Server knows how long to wait for acknowledgement that its callback was received.
The WCF ReceiveTimeout does not apply to client-side configuration. For Server-side config the ReceiveTimeout is used by ServiceFramework layer to initialize the session-idle timeout (to be honest I don't really know what that is)
This MSDN discussion may be helpful http://social.msdn.microsoft.com/Forums/vstudio/en-US/84551e45-19a2-4d0d-bcc0-516a4041943d/explaination-of-different-timeout-types?forum=wcf
As a final note, having really big timeout values isn't a good idea unless you definitely have long running requests. This is because you can run out of available resources on your server if the client isn't closing the connections properly.
We observed the following behavior on one of the servers hosting a WCF service on IIS 6.0:
The IIS log shows a high value for time-taken (> 100000)
The HTTP status code is 200
sc-win32-status code shows a value of 64
I found out that sc-win32-status code of 64 indicates "The specified network is no longer available"
Initially I suspected that it could be because of limits set on MinFileBytesPerSecond, which sets the minimum throughput rate that HTTP.sys enforces when sending data from the client to the server, and back from the server to the client.
But the value for sc-bytes and cs-bytes indicate that the amount of data is sent is within the range generally observed for the service.
Also note that the WCF service is hosted on four boxes and is load-balanced, but the problem occurs only one of the servers. (but not essentially on the same server). The problem is also intermittent.
Has anybody else encountered this error? Any clues about what could be wrong?
Update
Note: Observation on IIS 7.5 (IIS version does not really matter)
I was able to replicate the issue. The issue occurs if:
1. The WCF service takes a long time to respond
2. The client proxy times out before it receives a response from the server. In this case it leads to TimeoutException on the client.
3. The server keeps waiting for TCP ACK for the client, which it would never receive.
Hence a long timeout (TCP socket timeout (default value: 4 minutes) and sc-win32-status of 64
So essentially it appears that WCF code is taking a long time to respond and the client is timing out, what I observe in IIS log is just a symptom and not a problem.
The behavior you are describing will also occur if you exceed a WCF service's max sessions, calls or instances (depending on how you have your service instancecontext mode configured). If you observe the System.ServiceModel performance counters for %max concurrent sessions and/or %max concurrent calls (again depending on your service's instance context), you may see a correlation with the IIS log entries.
Note that these maxes can be configured in the service throttling behavior.
https://msdn.microsoft.com/en-us/library/vstudio/system.servicemodel.description.servicethrottlingbehavior(v=vs.100).aspx
I saw your question again and wanted to point out that I found a solution for this. It turned out to be this piece of code in the web.config:
<pages smartNavigation="true">
After turning this off I stopped receiving the same time-out errors. See also the answer here
IIS put the services into sleep to save recources.
Copied from here (WCF REST Service goes to sleep after inactivity)
The application pool hosting your service defines Idle Time-out property (advanced settings of app pool in IIS management console) which defaults to 20 minutes. If no request is received by the app pool within idle timeout the worker processes serving the pool is terminated. After receiving a new request the IIS must start the process again, the process must load application domain and all related assemblies, compile .svc file, run the service host and process the request.The solution can be increasing idle time-out but the meaning of this time-out is correct handling of server resources. If the process is not needed it should be stopped. Another ugly workaround is using some ping process (for example cron job or scheduled task on the server) which will regularly ping call some method on the service or page in the same application.
In a Silverlight 4 app, I would like to increase the timeout for a specific RIA service load operation (not for all loads, just in a specific case). At Configuring the timeout for a WCF RIA Services call from a Silverlight 3 client I followed a link to instructions that purportedly would allow me to set the timeout.
It appeared to work fine (no compiler error, warning, exception, etc) except that the load operation still timed out early. It appears that with or without the code that modifies the endpoint the load operation is timing out after 2 minutes. There is an Opening event on the ChannelFactory which I subscribed to, and my handler was called at the start of the load operation, so that seems to confirm that the ChannelFactory is being used. Also I set all 4 timeout values (Receive, Open, Close, Send) to 10 minutes just to be sure that I wasn't setting the wrong one.
Why I am unable to actually change the timeout for the RIA load?
I discovered the problem was that multiple timeouts were in play. I was using an EntityFramework domain service for RIA, and I was getting an EntityFramework timeout. I was misinterpreting the source of the timeout as being from the RIA load until I noticed in the stack trace that the timeout was server side. I extended the allotted EntityFramework command timeout to fit my desired load behavior. I was able to confirm that after getting a reference to the channel factory for the domain context I could set the RIA client-side timeout.
NOTE TO SELF: a good way to troubleshoot a timeout is to start by setting it really short to confirm it is working as intended
I have already asked a similar question here: WCF Service calling an external web service results in timeouts in heavy load environment but I've got a better idea now as to what's happening so posting a new question.
This is what is happening:
.NET client sends multiple requests at the same time to a WCF service (if it helps - I'm replicating this scneario by using Visual Studio Load Tests)
The client has got a "sendTimeout" set to 5 seconds
The WCF service receives it and start processing it. The processing involves sending a request to an external service which could take about 1 second to come back with a response
This is where I think the problem is: the client has sent many requests to the service and since the service is still busy processing the concurrent requests, some of the reqeusts from the client are timing out after 5 seconds
I have tried the following:
Changed the InstanceContextMode to PerCall
Increased the values of maxConcurrentCalls & maxConcurrentInstances
Increased the value of connectionManagement.maxconnection in machine.config
But none of that seems to be making any difference. Does anyone has any idea how can I ensure that I don't run into this timeout issue?
OK, you say WCF and that is not enough. What binding are you using and where are you hosting it? If you are using IIS, the could be different underlying problem than self-hosting.
The likely reason is the small number of ThreadPool size. You can use ThreadPool.SetMaxThreads() to change this but beware this is a sensitive value. Have a look here.
Check out the following link:
http://weblogs.asp.net/paolopia/archive/2008/03/23/wcf-configuration-default-limits-concurrency-and-scalability.aspx
I'm not sure what you're trying to achieve. Since the WCF service is doing a time consuming operation, you can't overload it and expect it to function. You can do the following (check the link about to set the following):
Increase the receiving capacity of the wcf service
Increase the send timeout of the service
Increase the send timeout of the client
Increase the receive timeout of the client
Limit the outgoing connections to the wcf service
The best and most robust option would be to configure and use MSMQ with the WCF service.
I have 2 wcf services, 1 which polls the other service at regular interval.The service2 is hosted in no. of machines with the same configuration.
My problem is that whenever the poller service gets restarted, even though the service2 on other machines runs fine, i am not getting the response from those services (basically it gets timed out - getting SYSTEM.TimeOutException ). If I try to access the same service (service2) from some temp application (without restarting the service2) it gives response.
If I restart the service2, than it works fine, the service1 (poller service) gets the responses from all hosted services (Service2).
Dont know what is causing problem.
Regards,
Chirag
Attach VS to your wcf service which hangs. And find out if your connection is successful.
Do it with both services, so that you can debug the services at runtime.
If you're using a sessionful binding (netTcpBinding, wsHttpBinding), it's more than likely that you're not explicitly closing your client channel when you're done with it. This would cause the behavior you see, because the session takes a minute or so to time out if you don't explicitly close it, and the default max number of sessions is low (10)- the server will let new sessions stack up until old ones close. You can also adjust the service throttle on the server side binding to increase the max number of open sessions allowed, but you really should make sure your clients are getting cleaned up properly first.