I have hosted WCF service in a console application which is expected to run 24/7 and respond to requests from several applications. It works great if it gets requests continuously or even after 4 hrs of idle time. The code written to host the service is given below,
static void Main(string[] args)
{
using (ServiceHost host = new ServiceHost(typeof(abcdService)))
{
host.Open();
Console.ReadLine();
host.Close();
}
}
In the WCF service, I was writing the output to console. When it throws the timeout error, I could see that the method in WCF service never received a request.
Is there any settings I need to do to make the self hosted WCF service respond to simple calls even after several hours of idle time?
The error I received is given below,
The request channel timed out while waiting for a reply after 00:00:59.9989999. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to '' has exceeded the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out. Aborting test execution.
Note: When I restart the console application hosting WCF service, it starts to work fine again.
Your requests might be queuing up on server before they are handled and then times out.
1) Try throttling in your WCF services.
2) Try using PerCall Use the following to remove the Session in your interface
[ServiceContract(Namespace="namespace", SessionMode=SessionMode.NotAllowed)]
Contract class implementing interface
ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
You should be able to enable tracing to see exactly what is occurring.
Related
In our production environment, we have a WCF serivce that is very frequently called.
We noticed that sometimes, calls to this service (only this one) fail on timeout for a period of time, after everything falls into place and the service responds correctly again.
I used Dynatrace to try to understand what's happening, I noticed that for the calls resulting on a timeout, the method of the service is never called ! And at the same time the server throw this error
A blocking operation was interrupted by a call to
WSACancelBlockingCall
and the client throws a Timeout Exception.
I want to understand the cause of this errors. Is the server error caused by the client's TimeoutException (when the client close its connection) ? Otherwise why do the server throw this error ?
Can you attach a screenshot of that PurePath?
The TimeoutException is simply thrown by the caller of a service when the called web services doesnt return within the default timeout - typically something like 60s. And - once the client aborts its network connection it will cause the exception in the server who has accepted that connection.
There can be multiple reasons for this slow behavior, e.g: you are maxing out the number of connections you have in your client - or the server implementation is overloaded and cant handle incoming requests. Definitely look at the number of worker threads/connections configured on both sides
If you want specific help on dynatrace freel free to send over the PurePaths - check out http://bit.ly/sharepurepath
hope this helps
We observed the following behavior on one of the servers hosting a WCF service on IIS 6.0:
The IIS log shows a high value for time-taken (> 100000)
The HTTP status code is 200
sc-win32-status code shows a value of 64
I found out that sc-win32-status code of 64 indicates "The specified network is no longer available"
Initially I suspected that it could be because of limits set on MinFileBytesPerSecond, which sets the minimum throughput rate that HTTP.sys enforces when sending data from the client to the server, and back from the server to the client.
But the value for sc-bytes and cs-bytes indicate that the amount of data is sent is within the range generally observed for the service.
Also note that the WCF service is hosted on four boxes and is load-balanced, but the problem occurs only one of the servers. (but not essentially on the same server). The problem is also intermittent.
Has anybody else encountered this error? Any clues about what could be wrong?
Update
Note: Observation on IIS 7.5 (IIS version does not really matter)
I was able to replicate the issue. The issue occurs if:
1. The WCF service takes a long time to respond
2. The client proxy times out before it receives a response from the server. In this case it leads to TimeoutException on the client.
3. The server keeps waiting for TCP ACK for the client, which it would never receive.
Hence a long timeout (TCP socket timeout (default value: 4 minutes) and sc-win32-status of 64
So essentially it appears that WCF code is taking a long time to respond and the client is timing out, what I observe in IIS log is just a symptom and not a problem.
The behavior you are describing will also occur if you exceed a WCF service's max sessions, calls or instances (depending on how you have your service instancecontext mode configured). If you observe the System.ServiceModel performance counters for %max concurrent sessions and/or %max concurrent calls (again depending on your service's instance context), you may see a correlation with the IIS log entries.
Note that these maxes can be configured in the service throttling behavior.
https://msdn.microsoft.com/en-us/library/vstudio/system.servicemodel.description.servicethrottlingbehavior(v=vs.100).aspx
I saw your question again and wanted to point out that I found a solution for this. It turned out to be this piece of code in the web.config:
<pages smartNavigation="true">
After turning this off I stopped receiving the same time-out errors. See also the answer here
IIS put the services into sleep to save recources.
Copied from here (WCF REST Service goes to sleep after inactivity)
The application pool hosting your service defines Idle Time-out property (advanced settings of app pool in IIS management console) which defaults to 20 minutes. If no request is received by the app pool within idle timeout the worker processes serving the pool is terminated. After receiving a new request the IIS must start the process again, the process must load application domain and all related assemblies, compile .svc file, run the service host and process the request.The solution can be increasing idle time-out but the meaning of this time-out is correct handling of server resources. If the process is not needed it should be stopped. Another ugly workaround is using some ping process (for example cron job or scheduled task on the server) which will regularly ping call some method on the service or page in the same application.
[ServiceContract]
public interface Service
{
[OperationContract(IsOneWay = true)]
void ServiceMethod();
}
I set server's code with IsOneWay = true, because the client does not care about the server's result and the server's method may need run a long time (e.g.30 mins) in some cases.
But I found the client still waits for the server's method to be finished. After the server finished in 30 mins, client requests again, report the CommunicationException:
"The socket connection was aborted. This could be caused by an error
processing your message or a receive timeout being exceeded by the
remote host, or an underlying network resource issue. Local socket
timeout was '00:01:00'".
I think the client still wait the result (default receiveTimeOut is 10 mins), then lead to timeout. I use WCF 3.0.
Can you help me? Thank you!
A one-way call in WCF is not the same thing as an asynchronous call.
Even though the client making a one-way call will not receive a response from the service, if the service does not have a thread available to dispatch or queue the incoming client request then the client will hang and eventually timeout if no dispatcher thread becomes available within the timeout period.
The number of available threads and the size of the request queue are managed by WCF and are determined by the service concurrency mode, session mode, and whether the service was configured with reliable messaging, amongst other factors.
MSDN ServiceBehviorAttribute.concurrencyMode states:
Setting ConcurrencyMode to Single instructs the system to restrict
instances of the service to one thread of execution at a time, which
frees you from dealing with threading issues.
That means that server side all calls on the service will come in on a unique thread. Which is great as you don't have to worry about multithreading but also not so great in that if you block that one thread with a long operation then other calls from your client that happen while its processing will not get through. Hence the exception.
ConcurrancyMode = Single is the default. You could try setting the concurrancy mode to Multiple - which will mean that calls will now come in on random threadpool threads and if one of those is busy processing a request another one is available for another request. But because the enviroment is now multithreading you will have to protect server objects from access by multiple thread with locks or other syncronisation mechanisms.
Have you tried re-generating the service client? It may be that the client still has a reference to a synchronous operation, whereas the server has been re-defined as one way / async.
In a Silverlight 4 app, I would like to increase the timeout for a specific RIA service load operation (not for all loads, just in a specific case). At Configuring the timeout for a WCF RIA Services call from a Silverlight 3 client I followed a link to instructions that purportedly would allow me to set the timeout.
It appeared to work fine (no compiler error, warning, exception, etc) except that the load operation still timed out early. It appears that with or without the code that modifies the endpoint the load operation is timing out after 2 minutes. There is an Opening event on the ChannelFactory which I subscribed to, and my handler was called at the start of the load operation, so that seems to confirm that the ChannelFactory is being used. Also I set all 4 timeout values (Receive, Open, Close, Send) to 10 minutes just to be sure that I wasn't setting the wrong one.
Why I am unable to actually change the timeout for the RIA load?
I discovered the problem was that multiple timeouts were in play. I was using an EntityFramework domain service for RIA, and I was getting an EntityFramework timeout. I was misinterpreting the source of the timeout as being from the RIA load until I noticed in the stack trace that the timeout was server side. I extended the allotted EntityFramework command timeout to fit my desired load behavior. I was able to confirm that after getting a reference to the channel factory for the domain context I could set the RIA client-side timeout.
NOTE TO SELF: a good way to troubleshoot a timeout is to start by setting it really short to confirm it is working as intended
I'm making a call to a WCF service but I get a CommunicationException on the client while receiving the response from the service.
System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://localhost:8080/Service. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
The client that makes the call is a WCF proxy client.
The service method executes without
any exceptions.
The WCF call works fine in those cases where it does not
take a long time for the serivce
method to finish.
The WCF call fails with the above exception message when the service method is taking long time to finish.
The sendTimeout property of the client's binding has been increased to 30 minutes to accommodate the time it can take for the service method to finish.
Try to set the receiveTimeout equal or greater than the time it takes for the service method to complete. The default value for the receiveTimeout property is 10 minutes. So if the service method takes longer time to complete the connection will be closed (if no other activity takes place before the receiveTimeoutoccurs). The receiveTimeout property is described here.
A very long operation like this should most likely be called asynchronously - in other words, the client asks the server to prepare the data, then gets on with something else while the server does the work. When the server's finished it calls the client back.
Asynchronous WCF operations are discussed here.