consider a WCF service, which is heavily used and behaves normally. But then is stopps responding. In the sevice level message trace you can see the outgoing message on the client, but no incoming message on the server. On transport level theres a incoming message and then nothing. After 60 seconds the client throws a TimeoutException.
What can cause a behavior like this?
What would you do to debug this behavior?
Is it possible that this behavior is caused by too many concurrent connections/sessions?
EDIT:
Client and Server are on the same machine. Both are .NET apps. When the client is restarted the problem sometimes does not happen. Also the problem does only appear on a single machine. I was not able to reproduce the behavior on any other machine.
Regards
Michael
I understand you have no problem on network level as you have mentioned that you can see incoming request on transport level.
So the first thing to check does the service is up and does it works if the client is on the same machine.
Also you can analyze incoming messages may be the problem can be there.
Here WireShark will be your friend.
Also check can you view the wsdl from the client machine. By the way are your clients also .NET apps?
You can configure tracing using the application’s configuration file—either Web.config for Web-hosted applications, or Appname.config for self-hosted applications by Service Trace Viewer
or use from debug tools such Debug Diag
Related
I built a application with 5 wcf services and hosted them in IIS 7.5. I used the default configuration for the net.Tcp port (808*).
I am not used to host WCF services in IIS (I always hosted in Windows Services) and I found interesting that when I call the service (using tcp binding) I got two different process on the server.
One is SMSvcHost.exe (the one that is actually using the 808 port) and the other is w3wp.exe wich I think is handling a instance of the server I am calling.
I have a lot of questions so I will enumerate them:
Should I use IIS instead of Windows Service to host WCF Services (tcp binding) ?
The fact that I have two processes to answer my request means that I may have cpu impacts ?
Sometimes my services stop answering using the tcp binding. I got a timeout error on my clients but the mex is correctly answering if I go to the http://myServer/Service1.svc. I suspect that this problem is caused by fault connections but I am throwing exceptions correctly (using fault exceptions) and catching them correctly on my clients. Besides, I am also implementing a partial class for every service to dispose corretly (either using Close or Abort methods). Is there any way to figure out what's going on when the services stop answering ?
Shouldn't the w3wp.exe processes be closed after the client ends the request ? They remain on the Task Manager even when no one is using the services. I guess this is the reason of why my Entity Library logging locks the file after my request is completed.
Ideally it would be better hosted in Windows Activation Services (WAS) which is close to what you think of as IIS, but not quite. Here's a good introductory article on MSDN magazine:
http://msdn.microsoft.com/en-us/magazine/cc163357.aspx
Ok, can I vent?? I am so sick and tired of this. I'm working away most of the day and the WCF services are working great. Next time I run my app and make a WCF call, bam! the tcp socket is no longer available. I have searched high and low to solve this and there is no real solution. The only solution I can find is to reboot the machine which is a huge time-waste and burden. Restarting WPA service, net.tcp service, IIS, etc. does not do a thing. Logging off and back on does not fix it. Only a reboot fixes this issue. I do nothing except run my app again making a WCF call, and this crap happens. There are no configuration issues with anything. I have been dealing with this for months and cannot find any specific reason or solution as to why this happens. It happens with my firewall on or off, does not matter.
Any insight from anyone? I think there is truly a bug in the WCF / net.tcp layer that is causing this. I even get it on a production 2008 R2 server when sometimes making a Web.config change, so I have learned to stop the IIS, WPA, net.tcp, etc. services prior to the change then restart them. What a pain.
I'm using .NET4 all around, VS2010, all service packs, etc. applied. Everything is the most current.
Excuse me while I reboot.....
Can anyone help with this?
Open a command prompt
Navigate to c:\windows\microsoft.net\framework64\v4.0.30319
Register the service model using the command "ServiceModelReg.exe -r"
Credits go there http://kumaranbose.blogspot.be/2010/08/cryptic-wcf-nettcp-errors.html
This issue hunts me for almost 3 years now but only happens sporadically. TCPView helped.
I have killed SMSSvcHost.exe process and then restarted Net.Tcp Listener Adapter service. That cleared the issue. Not really a solution but at least, I don't have to resort to rebooting the server anymore.
I had this issue. It would happen after each IIS reset (which happens as part of our deployment). The issue was resolved after restarting NetTcpPortSharing service (which also restarts Net.Tcp Listener Adapter service)
I am not sure I have an answer but, you could identify the process that has the port open and that can help narrow the scope of the problem. I have used Sysinternals suite which has a TCPView. This proggy was helpful to me.
TCPView - http://technet.microsoft.com/en-us/sysinternals/bb897437
Sounds Net.Tcp Listener Adapter service is being killed by some process or exception being throw by the web service putting the channel in a faulted state.
Have you tried setting the startup type of the service to automatic and recovery to restart service on first and second failure?
I doubt it very much that there is a bug in wcf net.tcp channel layer. If the listener is running and tcp socket no longer available i would suggest you look into the code especially around the exception handling strategy and have a peek into the iis request logs.
I have two windows services. One ('server') acts as a WCF host to which the other ('client') connects. So I have configured a dependency from client to server. Both are also set up to start automatically.
When I start these services by hand, everything works fine. When I stop both services and tell client to start, then server will be started before client and all is fine.
However, when I reboot the machine only server is started.
When I add a diagnostic listener I see it got a TimeoutException with the helpful message:
The HTTP request to 'http://[server address]' has exceeded the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.
At some other SO question there was an answer that claims WCF is probably confused about what went wrong and therefore starts lying about the timeout.
Did I perhaps miss a dependency for either service? Does WCF require something that hasn't or is being started when client is trying to contact server?
I think you should check your client service. On startup windows services are starting while network devices are still being initialized. Services should be ready to start without network and without any network device. Usual approach is to keep periodic retries to establish connection. You can do little experiment on your machine by uninstalling all network adapters and trying to start up your services.
Additional quick workaround you can do is to setup recovery options on your service -- for example you can configure it to restart service on crash after some timeout -- you can do this through UI in services.msc or in command line using 'sc config' command.
Configuring the dependency between the two Windows Services is not necessarily sufficient to avoid there being a race condition: i.e. to avoid the client service calling the WCF service before the server's WCF channel stack is fully initialised.
The service dependency just ensures that the Windows Service Control Manager won't start the client service process before the server Windows Service has notified the SCM that it has started. Whether this is sufficient depends on how you write the server.
If the server service starts a new thread on which to initialize the WCF stack, your OnStart method is probably returning before the WCF stack is ready for clients. There is then a race condition as to whether the client's first call will succeed.
On the other hand, if the server service does not return from OnStart (and thus doesn't notify the SCM that it has started) until the channel stack is fully open, the dependency removes the race condition, but there is a different pitfall: you need to beware that the SCM's own timeout for starting the Windows service is not triggered while waiting for the WCF stack to initialise, as might well happen on a reboot if the WCF service depends on the network stack, for example. If the server's OnStart does not return within the SCM's timeout, the SCM will not try to start the dependent client service at all, because it does not receive the server's start notification. (There will be a message in the Windows event log from the SCM saying that the server service didn't start within the expected time.) You can extend the SCM timeout by calling ServiceBase.RequestAdditionalTime while the WCF service is being initialised.
Either way, the client service really ought to be written so that it doesn't fail completely if the first WCF call doesn't succeed.
You don't actually say what binding you are using. If client and server services are always running on the same machine, as you seem to indicate, then consider using the NetNamedPipeBinding: then your service won't be dependent on initialization of networking resources and startup should be quicker.
I have a fairly straightforward WCF service that performs one-way file synchronization for a bunch of smart clients. I've noticed that when there's a network or service interruption during a call, the client stops being able to communicate with the server until the entire application is restarted.
The service runs with BasicHttpBinding and is hosted with IIS6 (a .svc page), using transferMode="Streamed" and messageEncoding="Mtom". The service is configured to use the default InstanceContextMode (I think it's Per Call?) and ConcurrencyMode=Single. It's using the default throttling behavior, but I'm in an isolated test environment that nobody else is hitting.
Clients are Windows Services. I'm using this ServiceProxyHelper to ensure connections are Close()'d or Abort()'d correctly when Dispose()'d, though there are no sessions so I don't think that even matters. When an error occurs, the Client object is disposed and then goes out of scope. After the exception is detected, the service waits a bit, then creates a new client object and tries again. So it should recover from the failure, but for some reason all subsequent calls to the service fail.
I can reproduce this reliably by starting a client, allowing it to transfer a few files, then iisresetting the server. First the client generally displays a "Service is Too Busy" error (which maps to the IIS 503 error that you get during an app restart). After that, all subsequent calls to the service time out. As far as I can tell the calls are not even being attempted by the client. I have tracing enabled and what I see is: Timeout error, followed by a "Failed to send request message over HTTP" warning, followed by another Timeout error.
The crazy thing is that when I configure the client to use Fiddler (port 8888) as a proxy in app.config, everything works as desired. So somehow Fiddler as the proxy is closing or finalizing some kind of connection that WCF on its own is not.
Thoughts?
Edit 2009-10-30 8:54PM: Changed service attributes to: InstanceContextMode=Single and ConcurrencyMode=Multiple. No difference.
Well that was painful. It took me forever but finally I zeroed in on the difference between running with a proxy vs without and started poking around the <system.net> settings. It turns out that adding this configuration bit to the client fixes the problem:
<system.net>
<settings>
<servicePointManager expect100Continue="false" />
</settings>
</system.net>
Can somebody explain what's going on? Why should this setting cause WCF clients to hang irreparably when there's a service interruption?
Are you sure this isn't a client side issue. If your Windows service is making the WCF calls on a seperate thread from the main, and you have an un-handled exception happening on the child thread...the calling thread may or may not sit there and wait forever becaues it's waiting for that thread to return.
That would explain why there's an Exception inside the service and then it looks like the service makes no more calls to the service...it's hung.
Used to be a huge issue when using Timers to spawn processes in .NET 2.0 Windows Services.
My application has 50 service endpoints (such as /mysite/myService.svc). It's hosted in IIS. Intermittently (once every two or three days) a service stops responding. It's never the same service that hangs. While a service is hung, some of the other services work fine and some other are also hung.
All clients (from different computers) get this error:
ServiceModel.CommunicationException
Message: An error occurred while receiving the HTTP response to
https://server/mysite/myservice1.svc.
This could be due to the service endpoint binding not using the HTTP
protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down).
See server logs for more details.
No exceptions are raised by the server when the client attempts to call the service that is hung. All I have is that error on the client side.
I have to manually recycle the application pool to fix the problem.
Do you know what could be the cause? How can I investigate this issue? I'm willing to take a memory dump of the worker process when a service is hung but I would not know what to search for in the dump.
Update (Aug 13 2009): I have almost ruled out the idea that the server runs out of connections (see comment in Shiraz Bhaiji's answer). I might have a new lead: I log all server-side exceptions in a log file. So in theory, when this occurs on the client, no exceptions are raised on the server; otherwise I'd have proof of that in my logs. But what if an error does occur on the server but is happening at a low level where exceptions are not routed to my exception handling code? I have posted this question about scenarios where low level exceptions cannot be handled. I'll keep you informed of the progress of my investigation.
Sounds like you are running out of connections.
By default WCF has a timeout and therefore holds a connection open for 10 mins.
When you recycle the app pool all connections are closed, and therefore things work again.
To fix it check your code to make sure that you close connections / dispose of proxies.
To resolve this, we set establishSecurityContext to False on the binding.
I have not come across this particular issue but would suggest to turn on tracing/message logging for the WCF service in the config for the service and/or the client app (if you have control over that). I've done this in the last few days for a service that I needed to troubleshoot.
The MSDN link here is a good starting point.
Also see the table in this post for the varying levels of trace detail you can configure. There are several levels which can go from exception only logging to full message details. It is quite quick to set this up in the app.config file.
To parse the log file output use the SvcTraceViewer.exe that comes with the Windows SDK, which if you have it installed should be located in this folder: C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin