wcf deadlock on client when bursting requests - wcf

Question

I've been banging my head against this issue for at least a week now (learned something new too - WCF is a major PITA).
Here's my problem: I have a scenario in my app that at some point freezes the whole client for, like, forever (because I disabled the timeouts, as both the client and server are in a controlled environment). The deadlock happens exactly on the same call, I presume due to the burst of requests preceding it.
Inspecting the deadlock stack trace on the client gives me this:
[In a sleep, wait, or join]
WindowsBase.dll!System.Windows.Threading.DispatcherSynchronizationContext.Wait(System.IntPtr[] waitHandles, bool waitAll, int millisecondsTimeout) + 0x26 bytes
mscorlib.dll!System.Threading.SynchronizationContext.InvokeWaitMethodHelper(System.Threading.SynchronizationContext syncContext, System.IntPtr[] waitHandles, bool waitAll, int millisecondsTimeout) + 0x1c bytes
[Native to Managed Transition]
[Managed to Native Transition]
mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext) + 0x2b bytes
mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext) + 0x2d bytes
mscorlib.dll!System.Threading.WaitHandle.WaitOne() + 0x10 bytes
System.Runtime.DurableInstancing.dll!System.Runtime.TimeoutHelper.WaitOne(System.Threading.WaitHandle waitHandle, System.TimeSpan timeout) + 0x7c bytes
System.ServiceModel.dll!System.ServiceModel.Channels.OverlappedContext.WaitForSyncOperation(System.TimeSpan timeout, ref object holder) + 0x40 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.PipeConnection.WaitForSyncRead(System.TimeSpan timeout, bool traceExceptionsAsErrors) + 0x38 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.PipeConnection.Read(byte[] buffer, int offset, int size, System.TimeSpan timeout) + 0xef bytes
System.ServiceModel.dll!System.ServiceModel.Channels.DelegatingConnection.Read(byte[] buffer, int offset, int size, System.TimeSpan timeout) + 0x21 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ConnectionUpgradeHelper.InitiateUpgrade(System.ServiceModel.Channels.StreamUpgradeInitiator upgradeInitiator, ref System.ServiceModel.Channels.IConnection connection, System.ServiceModel.Channels.ClientFramingDecoder decoder, System.ServiceModel.IDefaultCommunicationTimeouts defaultTimeouts, ref System.Runtime.TimeoutHelper timeoutHelper) + 0xb3 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(System.ServiceModel.Channels.IConnection connection, System.ArraySegment<byte> preamble, ref System.Runtime.TimeoutHelper timeoutHelper) + 0x155 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(System.ServiceModel.Channels.IConnection connection, ref System.Runtime.TimeoutHelper timeoutHelper) + 0x25 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(System.TimeSpan timeout) + 0xe2 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(System.TimeSpan timeout) + 0x37 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.CommunicationObject.Open(System.TimeSpan timeout) + 0x13f bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.OnOpen(System.TimeSpan timeout) + 0x52 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.CommunicationObject.Open(System.TimeSpan timeout) + 0x13f bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(System.ServiceModel.Channels.ServiceChannel channel, System.TimeSpan timeout) + 0x12 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(System.TimeSpan timeout, System.ServiceModel.Channels.ServiceChannel.CallOnceManager cascade) + 0x10c bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.Call(string action, bool oneway, System.ServiceModel.Dispatcher.ProxyOperationRuntime operation, object[] ins, object[] outs, System.TimeSpan timeout) + 0x18b bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(System.Runtime.Remoting.Messaging.IMethodCallMessage methodCall, System.ServiceModel.Dispatcher.ProxyOperationRuntime operation) + 0x59 bytes
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.Runtime.Remoting.Messaging.IMessage message) + 0x65 bytes
mscorlib.dll!System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(ref System.Runtime.Remoting.Proxies.MessageData msgData, int type) + 0xee bytes
MyService.dll!MyService.Controller.CallMethod() + 0x9 bytes
The reason I suspect the bursting calls sequence is that if I insert a sleep of 60s before the call is made, the deadlock doesn't occur.
Does anybody have any suggestion on how to avoid this issue?
P.S. I'm using named pipes.
EDIT:
The call to the WCF service on the client side happens on the GUI thread. Am I right to assume (from the callstack) that it tries to access the GUI thread, which is causing the deadlock?
EDIT:
Client side channel factory initialization:
var binding = new NetNamedPipeBinding
{
OpenTimeout = TimeSpan.MaxValue,
CloseTimeout = TimeSpan.MaxValue,
SendTimeout = TimeSpan.MaxValue,
ReceiveTimeout = TimeSpan.MaxValue,
ReaderQuotas = { MaxStringContentLength = Int32.MaxValue, MaxArrayLength = Int32.MaxValue },
MaxBufferPoolSize = Int32.MaxValue,
MaxBufferSize = Int32.MaxValue,
MaxReceivedMessageSize = Int32.MaxValue
};
CustomBinding pipeBinding = new CustomBinding(binding);
pipeBinding.Elements.Find<NamedPipeTransportBindingElement>().ConnectionPoolSettings.IdleTimeout = TimeSpan.FromDays(24);
channelFactory = new ChannelFactory<ITestsModule>(pipeBinding,
new EndpointAddress(string.Format("net.pipe://localhost/app_{0}/TestsModule", ProcessId)));
Server side host initialization:
var host = new ServiceHost(m_testModule, new Uri[] { new Uri(string.Format("net.pipe://localhost/app_{0}", Process.GetCurrentProcess().Id)) });
ServiceThrottlingBehavior throttle = host.Description.Behaviors.Find<ServiceThrottlingBehavior>();
if (throttle == null)
{
throttle = new ServiceThrottlingBehavior();
throttle.MaxConcurrentCalls = 500;
throttle.MaxConcurrentSessions = 200;
throttle.MaxConcurrentInstances = 100;
host.Description.Behaviors.Add(throttle);
}
ThreadPool.SetMinThreads(1000, 1000);
var binding = new NetNamedPipeBinding
{
OpenTimeout = TimeSpan.MaxValue,
CloseTimeout = TimeSpan.MaxValue,
SendTimeout = TimeSpan.MaxValue,
ReceiveTimeout = TimeSpan.MaxValue,
ReaderQuotas = { MaxStringContentLength = Int32.MaxValue, MaxArrayLength = Int32.MaxValue },
MaxBufferPoolSize = Int32.MaxValue,
MaxBufferSize = Int32.MaxValue,
MaxReceivedMessageSize = Int32.MaxValue
};
CustomBinding pipeBinding = new CustomBinding(binding);
pipeBinding.Elements.Find<NamedPipeTransportBindingElement>().ConnectionPoolSettings.IdleTimeout = TimeSpan.FromDays(24);
host.AddServiceEndpoint(typeof(ITestsModule), pipeBinding, "TestsModule");
Service class behavior:
[ServiceBehavior(
InstanceContextMode = InstanceContextMode.Single,
ConcurrencyMode = ConcurrencyMode.Multiple,
UseSynchronizationContext = false,
IncludeExceptionDetailInFaults = true
)]

Answer 1

First, you know what you're locking on server side? Is lock contention coming only from WCF interface? Or your server also locking from other components/classes elsewhere? That's the most important question and it has nothing to do with WCF.
Now, this said, try this to help narrow down the issue:
OPTION 1:
Timeout's on client side - don't set to Int32.MaxValue, set to say ten seconds, and implement client side re-try on timeout's.
OPTION 2:
ServiceThrottlingBehavior ThrottleBehavior = new ServiceThrottlingBehavior();
ThrottleBehavior.MaxConcurrentSessions = 4;
ThrottleBehavior.MaxConcurrentCalls = 4;
ThrottleBehavior.MaxConcurrentInstances = 4;
ServiceHost Host = ...
Host.Description.Behaviors.Add(ThrottleBehavior);
If OPTION 2 helps, stress test it (well should do same with OPTION 1)- also watch out for build up of thread counts if MaxConcurrentXXX set to a big number.
Hope this helps