I'm working on a self-hosted WCF application which runs just fine on my PC; however, when I try running it on a VM hosted locally using VMware Player, the service takes some two minutes to return data, whereas the original request took only a few seconds.
The VM is using 2Gb RAM and dual CPU running Windows Server 2008 R2 (on an 8Gb/quad core host running Windows 7).
Looking at the WCF service trace, I have the following log entries (time/description):
15:41:26.771 From: Processing message 1.
15:41:26.771 Activity boundary.
15:41:26.820 Received a message over a channel.
15:41:26.844 ServiceChannel information.
15:41:26.848 Incoming HTTP request to URI 'http://localhost:8000/Sql/Database' matched operation 'GetDatabase'
15:41:26.944 Message Log Trace
15:43:25.775 To: Execute 'MyProject.ISqlService.GetDatabase'
15:43:25.775 Activity boundary.
15:43:25.947 From: Execute 'MyProject.ISqlService.GetDatabase'
15:43:25.947 Activity boundary.
15:43:25.947 Message Log Trace
15:43:26.134 Throwing an exception.
15:43:26.134 RequestContext aborted
15:43:26.134 Activity boundary.
So the two minute delay occurs between receiving the incoming HTTP request and the dispatch to the service implementation. This delays occurs whether the request is the first (thus incurring the usual WCF warm-up penalty) or a subsequent request.
While I appreciate that I'm not going to get bare-metal performance from a VM, I'm still concerned about the dire performance, especially as the client tends to timeout before the end of the two minutes. Is there anything I can do to improve matters? It's making testing very difficult.
Maybe your proc does not support VT-x/AMD-V extension, so virtualization is not hardware-accelerated. Check your hardware using CPU-Z.
Related
I have a WCF SOAP service that receives too many synchronous requests from other systems
I am having problem when too many request comes at that time IIS Queue will not provide proper result and server memory and CPU usage is gone high and discard request or time out
I did a normal Load test (100 requests with 100 concurrent user) and the IIS started to discard the requests after the maximum queue length reach as well as all the request coming delays to provide the response and Other requests coming in are delayed until the first one either times out, or responds.
Below is server configuration
WCF application code is tested with Resharper tool and there is no object or memory dispose issue
Is there any settings for setup application pool or worker process to manage queue ?
Can i apply web garden in Application ?
Please help me to solve this issue
Thanks in Advance
We observed the following behavior on one of the servers hosting a WCF service on IIS 6.0:
The IIS log shows a high value for time-taken (> 100000)
The HTTP status code is 200
sc-win32-status code shows a value of 64
I found out that sc-win32-status code of 64 indicates "The specified network is no longer available"
Initially I suspected that it could be because of limits set on MinFileBytesPerSecond, which sets the minimum throughput rate that HTTP.sys enforces when sending data from the client to the server, and back from the server to the client.
But the value for sc-bytes and cs-bytes indicate that the amount of data is sent is within the range generally observed for the service.
Also note that the WCF service is hosted on four boxes and is load-balanced, but the problem occurs only one of the servers. (but not essentially on the same server). The problem is also intermittent.
Has anybody else encountered this error? Any clues about what could be wrong?
Update
Note: Observation on IIS 7.5 (IIS version does not really matter)
I was able to replicate the issue. The issue occurs if:
1. The WCF service takes a long time to respond
2. The client proxy times out before it receives a response from the server. In this case it leads to TimeoutException on the client.
3. The server keeps waiting for TCP ACK for the client, which it would never receive.
Hence a long timeout (TCP socket timeout (default value: 4 minutes) and sc-win32-status of 64
So essentially it appears that WCF code is taking a long time to respond and the client is timing out, what I observe in IIS log is just a symptom and not a problem.
The behavior you are describing will also occur if you exceed a WCF service's max sessions, calls or instances (depending on how you have your service instancecontext mode configured). If you observe the System.ServiceModel performance counters for %max concurrent sessions and/or %max concurrent calls (again depending on your service's instance context), you may see a correlation with the IIS log entries.
Note that these maxes can be configured in the service throttling behavior.
https://msdn.microsoft.com/en-us/library/vstudio/system.servicemodel.description.servicethrottlingbehavior(v=vs.100).aspx
I saw your question again and wanted to point out that I found a solution for this. It turned out to be this piece of code in the web.config:
<pages smartNavigation="true">
After turning this off I stopped receiving the same time-out errors. See also the answer here
IIS put the services into sleep to save recources.
Copied from here (WCF REST Service goes to sleep after inactivity)
The application pool hosting your service defines Idle Time-out property (advanced settings of app pool in IIS management console) which defaults to 20 minutes. If no request is received by the app pool within idle timeout the worker processes serving the pool is terminated. After receiving a new request the IIS must start the process again, the process must load application domain and all related assemblies, compile .svc file, run the service host and process the request.The solution can be increasing idle time-out but the meaning of this time-out is correct handling of server resources. If the process is not needed it should be stopped. Another ugly workaround is using some ping process (for example cron job or scheduled task on the server) which will regularly ping call some method on the service or page in the same application.
I have two windows services. One ('server') acts as a WCF host to which the other ('client') connects. So I have configured a dependency from client to server. Both are also set up to start automatically.
When I start these services by hand, everything works fine. When I stop both services and tell client to start, then server will be started before client and all is fine.
However, when I reboot the machine only server is started.
When I add a diagnostic listener I see it got a TimeoutException with the helpful message:
The HTTP request to 'http://[server address]' has exceeded the allotted timeout of 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.
At some other SO question there was an answer that claims WCF is probably confused about what went wrong and therefore starts lying about the timeout.
Did I perhaps miss a dependency for either service? Does WCF require something that hasn't or is being started when client is trying to contact server?
I think you should check your client service. On startup windows services are starting while network devices are still being initialized. Services should be ready to start without network and without any network device. Usual approach is to keep periodic retries to establish connection. You can do little experiment on your machine by uninstalling all network adapters and trying to start up your services.
Additional quick workaround you can do is to setup recovery options on your service -- for example you can configure it to restart service on crash after some timeout -- you can do this through UI in services.msc or in command line using 'sc config' command.
Configuring the dependency between the two Windows Services is not necessarily sufficient to avoid there being a race condition: i.e. to avoid the client service calling the WCF service before the server's WCF channel stack is fully initialised.
The service dependency just ensures that the Windows Service Control Manager won't start the client service process before the server Windows Service has notified the SCM that it has started. Whether this is sufficient depends on how you write the server.
If the server service starts a new thread on which to initialize the WCF stack, your OnStart method is probably returning before the WCF stack is ready for clients. There is then a race condition as to whether the client's first call will succeed.
On the other hand, if the server service does not return from OnStart (and thus doesn't notify the SCM that it has started) until the channel stack is fully open, the dependency removes the race condition, but there is a different pitfall: you need to beware that the SCM's own timeout for starting the Windows service is not triggered while waiting for the WCF stack to initialise, as might well happen on a reboot if the WCF service depends on the network stack, for example. If the server's OnStart does not return within the SCM's timeout, the SCM will not try to start the dependent client service at all, because it does not receive the server's start notification. (There will be a message in the Windows event log from the SCM saying that the server service didn't start within the expected time.) You can extend the SCM timeout by calling ServiceBase.RequestAdditionalTime while the WCF service is being initialised.
Either way, the client service really ought to be written so that it doesn't fail completely if the first WCF call doesn't succeed.
You don't actually say what binding you are using. If client and server services are always running on the same machine, as you seem to indicate, then consider using the NetNamedPipeBinding: then your service won't be dependent on initialization of networking resources and startup should be quicker.
We are running a glassfish server with around 20 jax-ws metro web services. The server specs are Core2Duo with 8GB RAM. We are using a single http listener for all the web services. Development is set to True. Request Thread Count is 2 and Acceptor Count is 1.
The Minimum and Maximum Heap Sizes are 1GB and the Perm Gen is set to 512MB.
The services access an Oracle database via a Hibernate layer and there are many interservice calls between the services.
The front end is ASP.Net. Our problem is that when 4-5 users try to access the application simultaneously for some time (1 hour) the glassfish server hangs with the CPU going to 100% but the memory utilization is around 10-11%.
We are not able to find any pointers as to how to debug this problem. On some instances the log file gives java.lang.OutofMemory Exception : PermGenSpace. But this is also not everytime, i.e. on many occassions the log file does not give any error on hanging. Also the glass fish server does not start if we try to increase the Perm Gen Space. We need some direction on how to diagnose and move towards the solution to this problem.
The Glass Fish Version we are using is v2.1.
We have the following observations:
1. Adding more http listeners (1 listener per 4-5 services) does prolong the failing time but not with much effect.
2. When calling some of the heavy services (one by one operation) with SOAP-UI we also get the hang problem when running many threads simultaneously. (e.g. 8-10 threads)
3. We have observed that when calling with SOAP-UI a service operation (which does not call any other services) rarely hangs while a service calling other services hangs much frequently.
My application has 50 service endpoints (such as /mysite/myService.svc). It's hosted in IIS. Intermittently (once every two or three days) a service stops responding. It's never the same service that hangs. While a service is hung, some of the other services work fine and some other are also hung.
All clients (from different computers) get this error:
ServiceModel.CommunicationException
Message: An error occurred while receiving the HTTP response to
https://server/mysite/myservice1.svc.
This could be due to the service endpoint binding not using the HTTP
protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down).
See server logs for more details.
No exceptions are raised by the server when the client attempts to call the service that is hung. All I have is that error on the client side.
I have to manually recycle the application pool to fix the problem.
Do you know what could be the cause? How can I investigate this issue? I'm willing to take a memory dump of the worker process when a service is hung but I would not know what to search for in the dump.
Update (Aug 13 2009): I have almost ruled out the idea that the server runs out of connections (see comment in Shiraz Bhaiji's answer). I might have a new lead: I log all server-side exceptions in a log file. So in theory, when this occurs on the client, no exceptions are raised on the server; otherwise I'd have proof of that in my logs. But what if an error does occur on the server but is happening at a low level where exceptions are not routed to my exception handling code? I have posted this question about scenarios where low level exceptions cannot be handled. I'll keep you informed of the progress of my investigation.
Sounds like you are running out of connections.
By default WCF has a timeout and therefore holds a connection open for 10 mins.
When you recycle the app pool all connections are closed, and therefore things work again.
To fix it check your code to make sure that you close connections / dispose of proxies.
To resolve this, we set establishSecurityContext to False on the binding.
I have not come across this particular issue but would suggest to turn on tracing/message logging for the WCF service in the config for the service and/or the client app (if you have control over that). I've done this in the last few days for a service that I needed to troubleshoot.
The MSDN link here is a good starting point.
Also see the table in this post for the varying levels of trace detail you can configure. There are several levels which can go from exception only logging to full message details. It is quite quick to set this up in the app.config file.
To parse the log file output use the SvcTraceViewer.exe that comes with the Windows SDK, which if you have it installed should be located in this folder: C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin