when multiple users are connecting to WCF Service at the same time, we are getting socket Exception.
Error Message:
{"The socket transfer timed out after 00:00:09.9844000. You have exceeded the timeout set on your binding. The time allotted to this operation may have been a portion of a longer timeout."}
Error code:10060
Related
I have a .Net 6 webapplication, that is hosted in IIS on Windows server 2019.
Sometimes after deployment, it will start having issues connection to SQL server and WCF services.
The thing that strikes me is that both types of errors start appearing at the same time. The SQL server, net.tcp WCF services and HTTP WCF services all start throwing these errors at the same time.
This leads me to think it could be some kind of socket exhaustion or similar?
How would I diagnose this kind of broad connection issue?
Note: The application has been ported from .Net Framework to .Net 6, and the old version does not exhibit this behavior, so I suspect that the .Net 6 code has some sort of leak, but I need to find out what is causing it.
Examples of the exceptions:
SQL error after the default 15s timeout:
Begin failed with SQL exception Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Another sql error
could not execute batch command.[SQL: SQL not available] Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. The wait operation timed out.
WCF net.tcp service
The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 10675199.02:48:05.4775807. The time allotted to this operation may have been a portion of a longer timeout. The socket connection has been disposed.
Object name: 'System.ServiceModel.Channels.SocketConnection'.
HTTP HTTP service
The request channel timed out attempting to send after 00:01:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. The HTTP request to 'https://www.example.com/service' has exceeded the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout.
I have received a Transport level error using SQL Server. This exception seems to be a network blip and was fixed for the same query the next second.
Exception in SqlRetryStrategyExecutor System.Data.SqlClient.SqlException (0x80131904): A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The semaphore timeout period has expired.)
I have a retry executor to retry whitelisted transient exceptions. The exception raised in this instance was not whitelisted.
Here are the error details
Error Number:121,State:0,Class:20
From microsoft link - https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2008-r2/cc645611(v=sql.105). 121 is not a transient error but an insert violation.
How to identify this transient network blip to retry the operation when error code is 121 ? Should I rely on Class = 20 ?
When I run simple console app that calls UserPrincipal.GetGroups, it enumerates the users groups with no problems. However when I run the same code as the same user on the same server but from a windows service hosting WCF, I get the following chain of errors:
Message : The socket transfer timed out after 00:00:10. You have exceeded the timeout set on your binding. The time allotted to this operation may have been a portion of a longer timeout.
Inner Exception
---------------
Message : The read operation failed, see inner exception.
Inner Exception
---------------
Message : The socket transfer timed out after 00:00:10. You have exceeded the timeout set on your binding. The time allotted to this operation may have been a portion of a longer timeout.
Inner Exception
---------------
Message : A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Could this have something to do with the WCF thread impersonation? WindowsIdentity.GetCurrent().Name returns the same user, however Thread.CurrentPrincipal.Identity.Name is different - empty string in the case of the console app, but the impersonated WCF user in the case of the Windoes Service.
Sessionful client has Binding.CloseTimeout set to 20 seconds:
<netTcpBinding>
<binding name="NetTcpBinding_IService" closeTimeout="00:00:20" ...
a) If service isn't running at the time client calls proxy.Close, then client should wait for 20 seconds ( due to CloseTimeout being set to 20 seconds ) before throwing an exception, but instead exception is thrown almost immediately:
»The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:20'.«
Why is CloseTimeout value ignored by client?
thank you
Imagine the client and server are people, and they have a meeting at noon.
The client shows up promptly at noon. If the server isn't there then the client leaves immediately, saying "Screw it!" (throwing an exception).
If the server is there, they commence negotiations (setting up a connection). If the meeting lasts less than CloseTimeout it's because they've come to an agreement (the connection has been set up). If they run out of time for the meeting, they abandon their negotiations (the connection attempt times out).
Lest you think I'm making this up, it's confirmed by the answer to this question over on MSDN:
EncpointNotFoundException [sic] usually occurs if the service cannot be found on the specified address. This exception will be thrown immediately. But if the service is found, yet the sessionful channel cannot be established within 20 seconds, you'll encounter the timeout error.
I'm making a call to a WCF service but I get a CommunicationException on the client while receiving the response from the service.
System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://localhost:8080/Service. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
The client that makes the call is a WCF proxy client.
The service method executes without
any exceptions.
The WCF call works fine in those cases where it does not
take a long time for the serivce
method to finish.
The WCF call fails with the above exception message when the service method is taking long time to finish.
The sendTimeout property of the client's binding has been increased to 30 minutes to accommodate the time it can take for the service method to finish.
Try to set the receiveTimeout equal or greater than the time it takes for the service method to complete. The default value for the receiveTimeout property is 10 minutes. So if the service method takes longer time to complete the connection will be closed (if no other activity takes place before the receiveTimeoutoccurs). The receiveTimeout property is described here.
A very long operation like this should most likely be called asynchronously - in other words, the client asks the server to prepare the data, then gets on with something else while the server does the work. When the server's finished it calls the client back.
Asynchronous WCF operations are discussed here.