In our production environment, we have a WCF serivce that is very frequently called.
We noticed that sometimes, calls to this service (only this one) fail on timeout for a period of time, after everything falls into place and the service responds correctly again.
I used Dynatrace to try to understand what's happening, I noticed that for the calls resulting on a timeout, the method of the service is never called ! And at the same time the server throw this error
A blocking operation was interrupted by a call to
WSACancelBlockingCall
and the client throws a Timeout Exception.
I want to understand the cause of this errors. Is the server error caused by the client's TimeoutException (when the client close its connection) ? Otherwise why do the server throw this error ?
Can you attach a screenshot of that PurePath?
The TimeoutException is simply thrown by the caller of a service when the called web services doesnt return within the default timeout - typically something like 60s. And - once the client aborts its network connection it will cause the exception in the server who has accepted that connection.
There can be multiple reasons for this slow behavior, e.g: you are maxing out the number of connections you have in your client - or the server implementation is overloaded and cant handle incoming requests. Definitely look at the number of worker threads/connections configured on both sides
If you want specific help on dynatrace freel free to send over the PurePaths - check out http://bit.ly/sharepurepath
hope this helps
Related
Good time of day. I would like to know how to properly connect in a WCF application. In other words, it should be created when the app is launched and be active throughout the entire operation? Or do you need to create a connection every time a service function is called? Now I have the first option, but somewhere everything is fine, and sometimes for unknown reasons I get an error: it is Impossible to use the object for communication, since it is in the failed state. There are no visible reasons for this - the code runs without errors. NetTcpBinding is used as the binding
The wcf service needs to be hosted in the process so that the client can connect to the server. As long as you are using the wcf service, you need to enable it. Faulted state means there has been an unexpected exception on the server side, so you need to use a try…catch block. Another possibility is that the channel has expired. The default timeout period of the WCF service is 10 minutes. If the client does not communicate with the server within 10 minutes, the channel will be closed. You need to recreate the channel to call the service.
[ServiceContract]
public interface Service
{
[OperationContract(IsOneWay = true)]
void ServiceMethod();
}
I set server's code with IsOneWay = true, because the client does not care about the server's result and the server's method may need run a long time (e.g.30 mins) in some cases.
But I found the client still waits for the server's method to be finished. After the server finished in 30 mins, client requests again, report the CommunicationException:
"The socket connection was aborted. This could be caused by an error
processing your message or a receive timeout being exceeded by the
remote host, or an underlying network resource issue. Local socket
timeout was '00:01:00'".
I think the client still wait the result (default receiveTimeOut is 10 mins), then lead to timeout. I use WCF 3.0.
Can you help me? Thank you!
A one-way call in WCF is not the same thing as an asynchronous call.
Even though the client making a one-way call will not receive a response from the service, if the service does not have a thread available to dispatch or queue the incoming client request then the client will hang and eventually timeout if no dispatcher thread becomes available within the timeout period.
The number of available threads and the size of the request queue are managed by WCF and are determined by the service concurrency mode, session mode, and whether the service was configured with reliable messaging, amongst other factors.
MSDN ServiceBehviorAttribute.concurrencyMode states:
Setting ConcurrencyMode to Single instructs the system to restrict
instances of the service to one thread of execution at a time, which
frees you from dealing with threading issues.
That means that server side all calls on the service will come in on a unique thread. Which is great as you don't have to worry about multithreading but also not so great in that if you block that one thread with a long operation then other calls from your client that happen while its processing will not get through. Hence the exception.
ConcurrancyMode = Single is the default. You could try setting the concurrancy mode to Multiple - which will mean that calls will now come in on random threadpool threads and if one of those is busy processing a request another one is available for another request. But because the enviroment is now multithreading you will have to protect server objects from access by multiple thread with locks or other syncronisation mechanisms.
Have you tried re-generating the service client? It may be that the client still has a reference to a synchronous operation, whereas the server has been re-defined as one way / async.
In a Silverlight 4 app, I would like to increase the timeout for a specific RIA service load operation (not for all loads, just in a specific case). At Configuring the timeout for a WCF RIA Services call from a Silverlight 3 client I followed a link to instructions that purportedly would allow me to set the timeout.
It appeared to work fine (no compiler error, warning, exception, etc) except that the load operation still timed out early. It appears that with or without the code that modifies the endpoint the load operation is timing out after 2 minutes. There is an Opening event on the ChannelFactory which I subscribed to, and my handler was called at the start of the load operation, so that seems to confirm that the ChannelFactory is being used. Also I set all 4 timeout values (Receive, Open, Close, Send) to 10 minutes just to be sure that I wasn't setting the wrong one.
Why I am unable to actually change the timeout for the RIA load?
I discovered the problem was that multiple timeouts were in play. I was using an EntityFramework domain service for RIA, and I was getting an EntityFramework timeout. I was misinterpreting the source of the timeout as being from the RIA load until I noticed in the stack trace that the timeout was server side. I extended the allotted EntityFramework command timeout to fit my desired load behavior. I was able to confirm that after getting a reference to the channel factory for the domain context I could set the RIA client-side timeout.
NOTE TO SELF: a good way to troubleshoot a timeout is to start by setting it really short to confirm it is working as intended
This may be a shot in the dark (I don't know much about the internals of WCF), but here goes...
I'm currently working with a legacy application at a client site and we're experiencing a persistent issue with a WCF service. The application is using the Microsoft Sync Framework 2.0 and syncing through the aforementioned service. The server-side implementation of the service has a lot of custom code in various states of "a mess."
Anyway, we're seeing an error on the client application most of the time and the pattern we're narrowing down centers around different users using the application on the same machine hitting the same service. It seems that the service and the client are getting out of sync in some way on an authentication level.
The error is discussed in an article here, and we're currently investigating the approach of switching from message layer security to transport layer security, which will hopefully solve the problem. However, we may be able to solve it in a less invasive manner if this question makes sense.
In the linked article, one of the suggestions was to forcibly terminate the connection if the specific exception is caught, try again, and if it fails again it wasn't because of this particular theory. Sounds good, and easy to implement. However, I find myself unable to say with confidence if the connection is being properly terminated.
The service operates through a custom interface, which is implemented on the server-side. The only thing that interface can do to end the connection is call EndSession() on the proxy itself, which calls EndSession() on the server which is a custom method.
So...
From a WCF service method, is there a way to properly and gracefully terminate the connection with the client in a way the client will like?
That is, in this custom EndSession() is there some last step I can take to cause the server to completely forget that this connection was open? Because it seems like when another user on the same machine tries to hit the service within the application, that's when it fails with the error in the linked article.
The idea is that, at the client side of things, code which calls EndSession() is followed by nulling out the proxy object, then a factory method is called to supply another one the next time it's needed. So I wonder if something additional needs to happen on the server side (and does by default in WCF were it not for all this custom implementation code) to terminate the connection on that end?
Like I said, a shot in the dark. But maybe in answers/discussions here I can at least further diagnose the problem, so any help is much appreciated. Thanks.
Unfortunately there are only really three ways in which a session can terminated
The client closes the proxy
The service's receiveTimeout is exceeded
before the client sends another
request
the service throws a
non-fault exception which will fault
the channel and so terminate the
session
if you don't want the client involved then you only have 2 and 3 neither of which end well for the client - they will get an exception in both situation on the next attempt to talk to the service.
You could use Duplex messaging and get the service to notify the client that its requires session termination - the client then gets an opportunity to close down the proxy gracefully but this is a cooperative strategy
My application has 50 service endpoints (such as /mysite/myService.svc). It's hosted in IIS. Intermittently (once every two or three days) a service stops responding. It's never the same service that hangs. While a service is hung, some of the other services work fine and some other are also hung.
All clients (from different computers) get this error:
ServiceModel.CommunicationException
Message: An error occurred while receiving the HTTP response to
https://server/mysite/myservice1.svc.
This could be due to the service endpoint binding not using the HTTP
protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down).
See server logs for more details.
No exceptions are raised by the server when the client attempts to call the service that is hung. All I have is that error on the client side.
I have to manually recycle the application pool to fix the problem.
Do you know what could be the cause? How can I investigate this issue? I'm willing to take a memory dump of the worker process when a service is hung but I would not know what to search for in the dump.
Update (Aug 13 2009): I have almost ruled out the idea that the server runs out of connections (see comment in Shiraz Bhaiji's answer). I might have a new lead: I log all server-side exceptions in a log file. So in theory, when this occurs on the client, no exceptions are raised on the server; otherwise I'd have proof of that in my logs. But what if an error does occur on the server but is happening at a low level where exceptions are not routed to my exception handling code? I have posted this question about scenarios where low level exceptions cannot be handled. I'll keep you informed of the progress of my investigation.
Sounds like you are running out of connections.
By default WCF has a timeout and therefore holds a connection open for 10 mins.
When you recycle the app pool all connections are closed, and therefore things work again.
To fix it check your code to make sure that you close connections / dispose of proxies.
To resolve this, we set establishSecurityContext to False on the binding.
I have not come across this particular issue but would suggest to turn on tracing/message logging for the WCF service in the config for the service and/or the client app (if you have control over that). I've done this in the last few days for a service that I needed to troubleshoot.
The MSDN link here is a good starting point.
Also see the table in this post for the varying levels of trace detail you can configure. There are several levels which can go from exception only logging to full message details. It is quite quick to set this up in the app.config file.
To parse the log file output use the SvcTraceViewer.exe that comes with the Windows SDK, which if you have it installed should be located in this folder: C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin