I am running a load test in order to see the performance of the WCF services during the peak time (heavy load). I am using the Step-Load where we push the virtual users Step-by-Step. When I start running the load test for the first few minutes the test runs smoothly and as the load increases by time, after some time all of a sudden the below error is triggering,
"Test method "XYZ" threw exception:
System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host".
I tried lot of solution that I found online but non of them worked for me. I tried changing the default time-outs, maxconnections, maxconcurrent connections etc., in config files. I would really appreciate any help on this.
I had similar problems while ago when moved to WCF. It have probably something to do with the way WCF handles connections.
Solution for me was to move each test into separate ApplicationDomain.
Related
I am frequently getting the below mentioned errors, the dll version used in the project is - 1.0.488.0
System.TimeoutException: Timeout performing GET
StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: GET
No connection is available to service this operation: EXISTS
Can anyone help me out in figuring what the issue can be?
Have also created an issue on StackExchange's Github repo for the same
Issue created on Github for the same
It looks like your connection broke. And when it did, any commands already sent to Redis would have timed out on the client application, even though they could have executed on the server. If you upgrade to a later version of the StackExchange.Redis client, you will get richer diagnostics information about what the state of the threadpool, CPU etc was on the client application side.
Azure .NET MVC application is encountering an exception i.e.
"The network path was not found"
Problem with the other queries that I searched before asking this question was that its not happening all the time, it's working as expected but this issue is occurring randomly like once every 15-25 days. It has been roughly 50 days since deployment of application on production and encountered this twice, while did not encounter this on Azure UAT environment(where low numbers of users were present).
Issue is temporarily resolved by IIS reset using the command:
iisreset
Source Error:
An unhandled exception was generated during the execution of the
current web request. Information regarding the origin and location of
the exception can be identified using the exception stack trace below.
Stack Trace:
[Win32Exception (0x80004005): The network path was not found]
[SqlException (0x80131904): A network-related or instance-specific
error occurred while establishing a connection to SQL Server. The
server was not found or was not accessible. Verify that the instance
name is correct and that SQL Server is configured to allow remote
connections. (provider: Named Pipes Provider, error: 40 - Could not
open a connection to SQL Server)]
Any idea?
Checklist:
Can you isolate the problem? If you have two web servers and one db, for example, when it happens does it happen on both web servers at the same time?
Set up monitoring of all critical network paths. Start monitoring possible causes (DHCP leases, even if you get a static IP from the server, would be one of the first).
Log as much information as possible. Can you determine if you are getting an ICMP error from a router or an issue from a host?
These things are, IME, usually intermittent network issues and often on peering connections, between data centers, and the like.
I have a WCF 4 service running on IIS 7 (Windows Server 2008), deployed to a development server. The service works when I run a small website that consumes the service (click a button, the page calls the service, data is returned. It's very basic).
I also created some unit tests that call the service and placed them into a load test. In a nutshell, the "unit tests" call the service and write the byte[] streamed back from the service to a file. The service performs as expected when running the load test.
The problem is the load test fails after a very small number of iterations. Sometimes foour calls are successful, other times after five calls. Again, individually, the tests run fine but when I put them together to simulate a live environment (multiple users, multiple test iterations) this problem appears.
The exception thrown is:
Test method TestProject1.UnitTest1.RunReport threw exception:
System.ServiceModel.CommunicationException:
An error occurred while receiving the HTTP response to http://xx.xx.xx.xx:yy/Service.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
Update #1:
I just set the test iteration to 10 (run the test 10x) and the constant user load to 1. The test was able to complete succesfully. This was using the most basic of the unit tests. The second, more complicated test, was omitted for this run.
Look in your application event log (eventvwr.exe). Assuming you're operating in a mostly-normal environment, there will be an entry for the server-side error.
My application has 50 service endpoints (such as /mysite/myService.svc). It's hosted in IIS. Intermittently (once every two or three days) a service stops responding. It's never the same service that hangs. While a service is hung, some of the other services work fine and some other are also hung.
All clients (from different computers) get this error:
ServiceModel.CommunicationException
Message: An error occurred while receiving the HTTP response to
https://server/mysite/myservice1.svc.
This could be due to the service endpoint binding not using the HTTP
protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down).
See server logs for more details.
No exceptions are raised by the server when the client attempts to call the service that is hung. All I have is that error on the client side.
I have to manually recycle the application pool to fix the problem.
Do you know what could be the cause? How can I investigate this issue? I'm willing to take a memory dump of the worker process when a service is hung but I would not know what to search for in the dump.
Update (Aug 13 2009): I have almost ruled out the idea that the server runs out of connections (see comment in Shiraz Bhaiji's answer). I might have a new lead: I log all server-side exceptions in a log file. So in theory, when this occurs on the client, no exceptions are raised on the server; otherwise I'd have proof of that in my logs. But what if an error does occur on the server but is happening at a low level where exceptions are not routed to my exception handling code? I have posted this question about scenarios where low level exceptions cannot be handled. I'll keep you informed of the progress of my investigation.
Sounds like you are running out of connections.
By default WCF has a timeout and therefore holds a connection open for 10 mins.
When you recycle the app pool all connections are closed, and therefore things work again.
To fix it check your code to make sure that you close connections / dispose of proxies.
To resolve this, we set establishSecurityContext to False on the binding.
I have not come across this particular issue but would suggest to turn on tracing/message logging for the WCF service in the config for the service and/or the client app (if you have control over that). I've done this in the last few days for a service that I needed to troubleshoot.
The MSDN link here is a good starting point.
Also see the table in this post for the varying levels of trace detail you can configure. There are several levels which can go from exception only logging to full message details. It is quite quick to set this up in the app.config file.
To parse the log file output use the SvcTraceViewer.exe that comes with the Windows SDK, which if you have it installed should be located in this folder: C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin
I'm having some problems with my WCF server and client connections.
If I use my service and client normally, meaning my client connects and disconnects thru normal operations (myclient.Close()) everything works fine. But in the instance that I have the server running and I use the visual studio "Stop" button to exit my client application, my server and client communications seems to mess up and the server doesn't clean up the connection correctly and any new connections from my client are very unstable and only certain function calls work until one function will throw an exception of:
System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://localhost:8080/Design_Time_Addresses/Service_Beta1/Service. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.
And many other exceptions after that, and I'll spare you from reading those.
If I shutdown the server and client and restart both, all my calls work fine until I do the drastic "Stop" in visual studio. How can I force the server to clean up improperly closed connections? I know in production the "Stop" button is gone and in theory there shouldn't be problems, but I don't want to have server connection problems from crashes of clients or bad disconnects. Because inevitably there will be those cases. It's best to fix this now before you have 20+ clients trying to connect and getting exceptions.
Thanks
Sorry for taking such a long time to post a reply. My problem is I was passing back a datatable to the client but wasn't giving the table a name on creation. See below.
Dim dt As New DataTable() 'Passing just a blank un-named table to client gave errors.
Dim dt As New DataTable("Table") 'Naming the table like so passes just fine.