I have a WCF service run on Windows Server 2008 RC2 IIS 7 with no firewall. When I trying to call it with netTcpBinding binding, I get this exception:
System.TimeoutException: The open
operation did not complete within the
allotted timeout of 00:00:30. The time
allotted to this operation may have
been a portion of a longer timeout.
---> System.TimeoutException: The socket transfer timed out after
00:00:30. You have exceeded the
timeout set on your binding. The time
allotted to this operation may have
been a portion of a longer timeout.
---> System.Net.Sockets.SocketException: A
connection attempt failed because the
connected party did not properly
respond after a period of time, or
established connection failed because
connected host has failed to respond
at
System.Net.Sockets.Socket.Receive(Byte[]
buffer, Int32 offset, Int32 size,
SocketFlags socketFlags) at
System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[]
buffer, Int32 offset, Int32 size,
TimeSpan timeout, Boolean closing)...
The method I call just returns a numeric value, nothing else, so the problem not in timeout. If I use wsHttpBinding - it works without problems. Also I added logging to method I call, so I know that it even not executed.
I made all steps to configure IIS from here. The questions are:
Anybody know what the problem may be?
How can I troubleshoot/debug this
problem?
I am as frustrated as you are with this misleading fault message. If I am not wrong you get this TimeoutException within the first second. I can assure you your issue either related to security or serialization problems. I took this issue to Microsoft's WCF product group, they seemed to be surprised but didn't hear back anything.
First suggestion is to look at NetTcpBinding security settings. Make sure both client and service have identical bare bones security settings to start with (such as not message encryption nor transport layer security). If you can make it work without security, step by step increase security settings.
Second suggestion some serialization problem may be crashing the service: An empty nullable field, overloaded methods mixing with operations, ambiguous contract names, invalid casts.To debug, setup tracing in your service's config settings. You can do this easily via WCF Service Config utility(SvcConfigEditor.exe). Run your service get the exception, stop it and open the generated trace logs with WCF Service Trace Viewer Tool. Both tools comes with .NET (not Visual Studio) and can be found in Program Files/Windows SDKs folder.
Related
I am running a python script that controls an ADI software named ACE capturing data remotely through port 8080. The ACE will attempt to create an IPC server on the port number. It takes around 12 minutes to capture 1 data set. The script runs normally for capture under 10 minutes. I suspects that the exception occurs because the default ReceiveTimeout is 10 minutes.
I adjusts the ReceiveTimeout in the SMSvcHost.exe.config file but it still not fix the issue. Other threads suggest that I need to adjust the ReceiveTimeout in app.config/ web.config. I am not sure how to locate these file. I am worried I might make changes to the wrong config file. Please advise!
Tracing Information:
[1]: https://i.stack.imgur.com/AKeHA.png
Here is the error message
This request operation sent to net.tcp://localhost:8080/ did not receive a reply within the configured timeout (00:10:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.
First of all, sorry, i'm not fluent.
I'm trying to figure out why my WCF services stop working when we have an environment with high calls/second rate. I'm not sure that just increasing timeout will solve the issue.
We have 2 webservices:
The first is hosted on IIS 7.5, Windows Server 2008 R2 Enterprise SP1 x64, with AppFabric (and WAS)
Second, hosted on Windows Service, Windows 2003 R2 SP1 x86
Both webservices have minimum configuration: No authentication, No trasaction, Without special treating of message.. check the binding:
<netTcpBinding>
<binding transactionFlow="false">
<security mode="None">
<message clientCredentialType="None" />
<transport clientCredentialType="None"></transport>
</security>
<reliableSession enabled="false"/>
</binding>
</netTcpBinding>
We are trying to use Net.Tcp binding because of its realibility and velocity.
FACT 1 - Net.Tcp Binding is primary reason
When the load is high, the channel Net.Tcp stop working. That's it! But the BasicHttp still working like a charm.
The WindowsService: the channel net.tcp last down for some minutes (3m - 10m) before get working back (BY ITSELF, without we change anything. Goblins are working hard).
The AppFabric/IIS/WAS: the channel net.tcp keep down. Need manual restart.
The BasicHttpBinding configuration is similar to net.tcp: without any treating of the message, whitout security concerns or something like that.
FACT 2 - Without any kind of logging
We couldn't find any kind, tip, trick to figure out what's happening. I have tried Dump the memory, event logs, System.Diagnostics and nothing relevant. The most relevant tip is an Error from SMSvcHost 4.0.0.0:
An error occurred while dispatching a duplicated socket: this handle
is now leaked in the process. ID: 2272 Source:
System.ServiceModel.Activation.TcpWorkerProcess/62875109 Exception:
System.TimeoutException: This request operation sent to
http://schemas.microsoft.com/2005/12/ServiceModel/Addressing/Anonymous
did not receive a reply within the configured timeout (00:01:00). The
time allotted to this operation may have been a portion of a longer
timeout. This may be because the service is still processing the
operation or because the service was unable to send a reply message.
Please consider increasing the operation timeout (by casting the
channel/proxy to IContextChannel and setting the OperationTimeout
property) and ensure that the service is able to connect to the
client.
Server stack trace: at
System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at
System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult
result) at
System.ServiceModel.Channels.ServiceChannel.EndCall(String action,
Object[] outs, IAsyncResult result) at
System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage
methodCall, ProxyOperationRuntime operation) at
System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage
message)
Exception rethrown at [0]: at
System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at
System.ServiceModel.Activation.WorkerProcess.EndDispatchSession(IAsyncResult
result) Process Name: SMSvcHost Process ID: 1532
Do you have any tip or configuration trick to help me solve this issue?
Whats the best configuration for high load scenarios?
If you generated a service reference in Visual Studio, or with the svcutil tool, make sure you always call the Close or Abort methods of your proxies. I encountered a similar problem some days ago because I forgot to call these methods.
In case you are calling the Close() and Abort() methods accordingly and still receive this error consider the following scenario:
You run a Microsoft .NET Framework 3.0-based or .NET Framework 3.5-based Windows Communication Foundation (WCF) service.
The WCF service uses the Net.Tcp Port Sharing Service (Smsvchost.exe) and is hosted on a computer that is running Internet Information Services (IIS).
One of the following conditions is true:
The CPU usage is high on the computer that is running IIS.
A throttle occurs in a service model for the WCF service.
Multiple requests are sent to the WCF service at the same time.
In this scenario, the WCF service takes longer than one minute to process a request from a client application. Additionally, an error message that assembles the following event entry is logged in the event log:
Log Name: System
Source: SMSvcHost 3.0.0.0
Date:
Event ID: 8
Task Category: Sharing Service
Level: Error
Keywords: Classic
User: LOCAL SERVICE
Computer:
Description: An error occurred while dispatching a duplicated socket: this handle is now leaked in the process.
ID: 2620
Source: System.ServiceModel.Activation.TcpWorkerProcess
Exception:
System.TimeoutException: This request operation sent to did not receive a reply within the configured timeout (00:01:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.
Note: You must restart IIS to recover the WCF service from this issue.
Cause:
This issue occurs because of the Smsvchost.exe process times out after one minute when it tries to transfer an incoming connection request to the W3wp.exe worker process. Additionally, this time-out is not configurable.
When the CPU has a heavy workload, or when many concurrent connection requests are incoming, the Smsvchost.exe process cannot transfer the incoming connection to the W3wp.exe worker process within one minute. Therefore, the Smsvchost.exe process times out and eventually stops responding. When this issue occurs, the Smsvchost.exe process cannot route later requests to the W3wp.exe worker process until IIS is restarted.
Solution:
Microsoft suggests applying the hot fix 2504602 that is described in Microsoft Knowledge Base (KB) article. This hot fix is available for WCF in the .NET Framework 3.0 SP2, in the .NET Framework 3.5 SP1 and the .NET Framework 4.
In addition, Microsoft claims to have solved this issue in the .Net Framework 4.5, therefore, you should upgrade to the latest version.
In case you upgrade to the .Net Framework 4.5 and the problem persists the workaround is to modify the smsvchost.exe.config file to increase timeout and pending accepts and various other parameters.
In a Silverlight 4 app, I would like to increase the timeout for a specific RIA service load operation (not for all loads, just in a specific case). At Configuring the timeout for a WCF RIA Services call from a Silverlight 3 client I followed a link to instructions that purportedly would allow me to set the timeout.
It appeared to work fine (no compiler error, warning, exception, etc) except that the load operation still timed out early. It appears that with or without the code that modifies the endpoint the load operation is timing out after 2 minutes. There is an Opening event on the ChannelFactory which I subscribed to, and my handler was called at the start of the load operation, so that seems to confirm that the ChannelFactory is being used. Also I set all 4 timeout values (Receive, Open, Close, Send) to 10 minutes just to be sure that I wasn't setting the wrong one.
Why I am unable to actually change the timeout for the RIA load?
I discovered the problem was that multiple timeouts were in play. I was using an EntityFramework domain service for RIA, and I was getting an EntityFramework timeout. I was misinterpreting the source of the timeout as being from the RIA load until I noticed in the stack trace that the timeout was server side. I extended the allotted EntityFramework command timeout to fit my desired load behavior. I was able to confirm that after getting a reference to the channel factory for the domain context I could set the RIA client-side timeout.
NOTE TO SELF: a good way to troubleshoot a timeout is to start by setting it really short to confirm it is working as intended
I have two windows services. One ('server') acts as a WCF host to which the other ('client') connects. So I have configured a dependency from client to server. Both are also set up to start automatically.
When I start these services by hand, everything works fine. When I stop both services and tell client to start, then server will be started before client and all is fine.
However, when I reboot the machine only server is started.
When I add a diagnostic listener I see it got a TimeoutException with the helpful message:
The HTTP request to 'http://[server address]' has exceeded the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.
At some other SO question there was an answer that claims WCF is probably confused about what went wrong and therefore starts lying about the timeout.
Did I perhaps miss a dependency for either service? Does WCF require something that hasn't or is being started when client is trying to contact server?
I think you should check your client service. On startup windows services are starting while network devices are still being initialized. Services should be ready to start without network and without any network device. Usual approach is to keep periodic retries to establish connection. You can do little experiment on your machine by uninstalling all network adapters and trying to start up your services.
Additional quick workaround you can do is to setup recovery options on your service -- for example you can configure it to restart service on crash after some timeout -- you can do this through UI in services.msc or in command line using 'sc config' command.
Configuring the dependency between the two Windows Services is not necessarily sufficient to avoid there being a race condition: i.e. to avoid the client service calling the WCF service before the server's WCF channel stack is fully initialised.
The service dependency just ensures that the Windows Service Control Manager won't start the client service process before the server Windows Service has notified the SCM that it has started. Whether this is sufficient depends on how you write the server.
If the server service starts a new thread on which to initialize the WCF stack, your OnStart method is probably returning before the WCF stack is ready for clients. There is then a race condition as to whether the client's first call will succeed.
On the other hand, if the server service does not return from OnStart (and thus doesn't notify the SCM that it has started) until the channel stack is fully open, the dependency removes the race condition, but there is a different pitfall: you need to beware that the SCM's own timeout for starting the Windows service is not triggered while waiting for the WCF stack to initialise, as might well happen on a reboot if the WCF service depends on the network stack, for example. If the server's OnStart does not return within the SCM's timeout, the SCM will not try to start the dependent client service at all, because it does not receive the server's start notification. (There will be a message in the Windows event log from the SCM saying that the server service didn't start within the expected time.) You can extend the SCM timeout by calling ServiceBase.RequestAdditionalTime while the WCF service is being initialised.
Either way, the client service really ought to be written so that it doesn't fail completely if the first WCF call doesn't succeed.
You don't actually say what binding you are using. If client and server services are always running on the same machine, as you seem to indicate, then consider using the NetNamedPipeBinding: then your service won't be dependent on initialization of networking resources and startup should be quicker.
My application has 50 service endpoints (such as /mysite/myService.svc). It's hosted in IIS. Intermittently (once every two or three days) a service stops responding. It's never the same service that hangs. While a service is hung, some of the other services work fine and some other are also hung.
All clients (from different computers) get this error:
ServiceModel.CommunicationException
Message: An error occurred while receiving the HTTP response to
https://server/mysite/myservice1.svc.
This could be due to the service endpoint binding not using the HTTP
protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down).
See server logs for more details.
No exceptions are raised by the server when the client attempts to call the service that is hung. All I have is that error on the client side.
I have to manually recycle the application pool to fix the problem.
Do you know what could be the cause? How can I investigate this issue? I'm willing to take a memory dump of the worker process when a service is hung but I would not know what to search for in the dump.
Update (Aug 13 2009): I have almost ruled out the idea that the server runs out of connections (see comment in Shiraz Bhaiji's answer). I might have a new lead: I log all server-side exceptions in a log file. So in theory, when this occurs on the client, no exceptions are raised on the server; otherwise I'd have proof of that in my logs. But what if an error does occur on the server but is happening at a low level where exceptions are not routed to my exception handling code? I have posted this question about scenarios where low level exceptions cannot be handled. I'll keep you informed of the progress of my investigation.
Sounds like you are running out of connections.
By default WCF has a timeout and therefore holds a connection open for 10 mins.
When you recycle the app pool all connections are closed, and therefore things work again.
To fix it check your code to make sure that you close connections / dispose of proxies.
To resolve this, we set establishSecurityContext to False on the binding.
I have not come across this particular issue but would suggest to turn on tracing/message logging for the WCF service in the config for the service and/or the client app (if you have control over that). I've done this in the last few days for a service that I needed to troubleshoot.
The MSDN link here is a good starting point.
Also see the table in this post for the varying levels of trace detail you can configure. There are several levels which can go from exception only logging to full message details. It is quite quick to set this up in the app.config file.
To parse the log file output use the SvcTraceViewer.exe that comes with the Windows SDK, which if you have it installed should be located in this folder: C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin