I have a server that is calling back to the client through a callback channel.
The callback contract operations are all marked as IsOneWay. The binding is netTcp.
I sometimes have the scenario where the server is generating more messages than the client can handle (I can simulate this by putting a sleep into the client method).
Eventually I get a "CommunicationException: The socket connection was aborted"
Unfortunately I have no idea what is going on under the hood.
Is the operation queued on the send
or receive side, or both?
Can I monitor these queues?
What causes the timeout?
Does WCF have threads that constantly write/read to the socket?
Does WCF on the receive side eventually stop reading from the socket hence the timeout?
To get more info on whats going on , try to turn on WCF tracing , and using the trace viewer to look at the output. here`s how to turn on tracing, and use the MS trace view utility SvcTraceViewer.exe
in a more direct answer to the question - WCF has a default incomming queue of 10 concurrent sessions, so i'm geussing that this is what you are experiencing when the server stresses the client. it`s possible to configure a larger value though, using the maxConcurrentSessions behaviur parameter.
Related
[ServiceContract]
public interface Service
{
[OperationContract(IsOneWay = true)]
void ServiceMethod();
}
I set server's code with IsOneWay = true, because the client does not care about the server's result and the server's method may need run a long time (e.g.30 mins) in some cases.
But I found the client still waits for the server's method to be finished. After the server finished in 30 mins, client requests again, report the CommunicationException:
"The socket connection was aborted. This could be caused by an error
processing your message or a receive timeout being exceeded by the
remote host, or an underlying network resource issue. Local socket
timeout was '00:01:00'".
I think the client still wait the result (default receiveTimeOut is 10 mins), then lead to timeout. I use WCF 3.0.
Can you help me? Thank you!
A one-way call in WCF is not the same thing as an asynchronous call.
Even though the client making a one-way call will not receive a response from the service, if the service does not have a thread available to dispatch or queue the incoming client request then the client will hang and eventually timeout if no dispatcher thread becomes available within the timeout period.
The number of available threads and the size of the request queue are managed by WCF and are determined by the service concurrency mode, session mode, and whether the service was configured with reliable messaging, amongst other factors.
MSDN ServiceBehviorAttribute.concurrencyMode states:
Setting ConcurrencyMode to Single instructs the system to restrict
instances of the service to one thread of execution at a time, which
frees you from dealing with threading issues.
That means that server side all calls on the service will come in on a unique thread. Which is great as you don't have to worry about multithreading but also not so great in that if you block that one thread with a long operation then other calls from your client that happen while its processing will not get through. Hence the exception.
ConcurrancyMode = Single is the default. You could try setting the concurrancy mode to Multiple - which will mean that calls will now come in on random threadpool threads and if one of those is busy processing a request another one is available for another request. But because the enviroment is now multithreading you will have to protect server objects from access by multiple thread with locks or other syncronisation mechanisms.
Have you tried re-generating the service client? It may be that the client still has a reference to a synchronous operation, whereas the server has been re-defined as one way / async.
I have already asked a similar question here: WCF Service calling an external web service results in timeouts in heavy load environment but I've got a better idea now as to what's happening so posting a new question.
This is what is happening:
.NET client sends multiple requests at the same time to a WCF service (if it helps - I'm replicating this scneario by using Visual Studio Load Tests)
The client has got a "sendTimeout" set to 5 seconds
The WCF service receives it and start processing it. The processing involves sending a request to an external service which could take about 1 second to come back with a response
This is where I think the problem is: the client has sent many requests to the service and since the service is still busy processing the concurrent requests, some of the reqeusts from the client are timing out after 5 seconds
I have tried the following:
Changed the InstanceContextMode to PerCall
Increased the values of maxConcurrentCalls & maxConcurrentInstances
Increased the value of connectionManagement.maxconnection in machine.config
But none of that seems to be making any difference. Does anyone has any idea how can I ensure that I don't run into this timeout issue?
OK, you say WCF and that is not enough. What binding are you using and where are you hosting it? If you are using IIS, the could be different underlying problem than self-hosting.
The likely reason is the small number of ThreadPool size. You can use ThreadPool.SetMaxThreads() to change this but beware this is a sensitive value. Have a look here.
Check out the following link:
http://weblogs.asp.net/paolopia/archive/2008/03/23/wcf-configuration-default-limits-concurrency-and-scalability.aspx
I'm not sure what you're trying to achieve. Since the WCF service is doing a time consuming operation, you can't overload it and expect it to function. You can do the following (check the link about to set the following):
Increase the receiving capacity of the wcf service
Increase the send timeout of the service
Increase the send timeout of the client
Increase the receive timeout of the client
Limit the outgoing connections to the wcf service
The best and most robust option would be to configure and use MSMQ with the WCF service.
Imagine the following setup: a Silverlight client tunnels a serialized command over the network using a WCF service which in turn deserializes the command and sends it using NServiceBus to a generic host which is responsible for processing the command. The WCF service has - upon sending the command - registered a callback to be invoked. The generic host validates the command and 'returns' an error code (either 0 == success or >0 == failure).
Note: The WCF service is modelled after the built-in WCF service. The difference is that this WCF service receives a 'universal command' (not an IMessage), deserializes it into a real command (which does implement IMessage), and consequently sends the deserialized command off to the bus.
When unexpected exceptions occur, the command gets (after a certain amount of retries) queued in an error queue. At this point, the initiating WCF service sits there idle, unaware of what just happened. At some later point, the Silverlight client will time out according to the WCF client proxy configuration.
Things which are fuzzy in my head:
Does NServiceBus handle this scenario in any way? When does the timeout exception get thrown (if at all)? Or is this something exclusive to sagas?
Presuming I use [OperationContract(AsyncPattern=true)], are there any WCF related timeout settings that will kill the service operation? Or will the EndXXX method be somehow called? Or will it sit there forever, leaking, waiting for something that will never come?
Ways to proceed:
reuse existing timeout mechanisms, provided things don't leak.
build my own timeout mechanism between the wcf service and nservicebus.
notify the wcf service somehow when the command lands in the error queue.
build my own async notifcation mechanism using full blown callback message handler in the WCF service layer.
Things I've done:
run the example provided with NServiceBus.
spiked the happy case.
Any guidance on how to proceed is welcome, be it blog post, mailing list entries, ...
Some motivations for picking my current approach
I'm trying to leverage some of the scalability advantages (using distributor in a later phase) of NServiceBus.
I don't want to host a gazillion WCF services (one for each command), that's why I cooked up a bus-like WCF service.
Even though this is somewhat request/response style, I'm mostly concerned with gracefully handling a command reply not coming through.
You can develop any sort of message type you desire, IMessage is simply a marker interface. If you inspect the WSDL file that the service mex endpoint provides, there is no reference to IMessage, therefore you can define any command you like in you service. That being the case you should be able to use the provided WCF host.
I was able to reproduce the issue you describe using the built-in WCF hosting option. When an exception is thrown, the entire transaction is rolled back and this includes the Bus.Return, and therefore the service never gets a response.
I found a hack around this that I could provide, but I recommend reconsidering how you are using the service. If you are truly looking to do some expensive operations in a separate process then I would recommend in your WCF endpoint that you do a Bus.Send to a different process altogether. This would ensure to your client that the command was successfully received and that work is in progress. From there it would be up to the server to complete the command(some up front validation would help ensure its success). If the command was not completed successfully this should be made known on another channel(some background polling from the client would do).
I have a lot of client programs and one service.
This Client programs communicate with the server with http channel with WCF.
The clients have dynamic IP.
They are online 24h/day.
I need the following:
The server should notify all the clients in 3 min interval. If the client is new (started in the moment), is should notify it immediately.
But because the clients have dynamic IP and they are working 24h/day and sometimes the connection is unstable, is it good idea to use wcf duplex?
What happens when the connection goes down? Will it automatically recover?
Is is good idea to use remote MSMQ for this type of notification ?
Regards,
WCF duplex is very resource hungry and per rule of thumb you should not use more than 10. There is a lot of overhead involved with duplex channels. Also there is not auto-recover.
If you know the interval of 3 minutes and you want the client to get information when it starts why not let the client poll the information from the server?
When the connection goes down the callback will throw an exception and the channel will close.
I am not sure MSMQ will work for you unless each client will create an MSMQ queue for you and you push messages to each one of them. Again with an unreliable connection it will not help. I don't think you can "push" the data if you loose the connection to a client, client goes off-line or changes an IP without notifying your system.
I like using WCF callbacks when I can because to me it is better than the client having to poll the server and its more real time than polling. The question I have is when I subscribe to a WCF service event is there any kind of heart beat that keeps the connection alive between the client and the server. I starting to think that there is not because when the server goes away the subscription is lost and the client does not throw an exception (could be the exception is be swallowed by the WCF runtime). Same is true for the server, when the client goes away and the server attempts to invoke the callback and exception is throw. Any thoughts?
Thanks
There is a good short description of the Duplex contract (WCF callbacks) in this link. The duplex contract is basically two one-way channels and there is no implied message correlation. You are right, there are no "heartbeat" messages are involved, only the normal wsHTTP handshaking traffic occurs when making a duplex call.
I fired up the HTTP traffic sniffer called Fiddler2 (an unsupported Microsoft tool) to verify the session traffic. Didn't see any under-the-hood HTTP "heartbeat" communication occurring during and after the service calls. I left the client running for a good while. Good question, it got me digging a bit.
I went ahead and created a recurring heartbeat to the subscribed clients (basically a call to a function they're hosting).
I've run this for hours and it works, this helps ensure the connection.