Asynchronous WCF fails with MessageSecurityException when multiple worker processes (Web Garden) - wcf

Question

I have a simple WCF service using ws2007HttpBinding which puts in a configurable delay, then returns. The service is configured to be PerCall. The service is hosted on IIS 7.5 on Windoes 2008 Server R2 SP1
I am call this asynchronously via console app using an imported service reference.
I can easily inject 10000 requests and they are all successful. Each request uses a new WCF proxy. The CPU was running about 70%
I wanted to see if adding another worker process would help throughput and utilise the full 100% cpu. I configured it to have 2 worker processes.
When I ran this it began to fail virtually immediately with a MessageSecurityException. Even if I set the number of requsts to 10, it still fails. I noticed that it fails more if the timer in the WCF service is short ( between 10 and 50 ms). A higher delay causes less failures.
It either fails on the proxy.Open() or proxy.Close()
I investigated this and thought it might be related to secure conversation. So I disabled this by
setting
establishSecurityContext="false"
(Is this correct?) I am confused why this would be the case if I am creating a new proxy each time
This dis not seem to make any difference.
When I return the AppPool to have only 1 worker process it works again
Has anybody had the same issue, and managed to get a solution. If so can you post you solution in explicit detail.
The errors received are as follows
In the WCF trace it reported
The incoming message is not part of an existing security session
The exception shown in trace
System.ServiceModel.EndpointNotFoundException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
There was no channel that could accept the message with action 'http://tempuri.org/ILoadedService/RunLoad'.
at System.ServiceModel.Dispatcher.ErrorBehavior.ThrowAndCatch(Exception e, Message message)
at System.ServiceModel.Channels.DatagramChannelDemuxer2.ProcessItem(TInnerItem item)
at System.ServiceModel.Channels.DatagramChannelDemuxer2.HandleReceiveResult(IAsyncResult result)
at System.ServiceModel.Channels.DatagramChannelDemuxer2.OnReceiveComplete(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.ServiceModel.Channels.ReplyChannel.HelpReceiveRequestAsyncResult.OnReceiveRequest(IAsyncResult result)
at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
at System.Runtime.InputQueue1.AsyncQueueReader.Set(Item item)
at System.Runtime.InputQueue1.EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)
at System.Runtime.InputQueue1.EnqueueAndDispatch(T item, Action dequeuedCallback, Boolean canDispatchOnThisThread)
at System.ServiceModel.Channels.SingletonChannelAcceptor`3.Enqueue(QueueItemType item, Action dequeuedCallback, Boolean canDispatchOnThisThread)
at System.ServiceModel.Channels.HttpChannelListener.HttpContextReceived(HttpRequestContext context, Action callback)
at System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(HostedHttpRequestAsyncResult result)
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.HandleRequest()
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest()
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(Object state)
at System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
System.ServiceModel.EndpointNotFoundException: There was no channel that could accept the message with action 'http://tempuri.org/ILoadedService/RunLoad'.
Exception caught on the client
Exception thrown of type [System.ServiceModel.Security.MessageSecurityException] Source [mscorlib]
Exception when receiving response [9] - Message [An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail.] StackTrace [
Server stack trace:
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at SimpleIISTestClient.SlowClient.ILoadedService.EndRunLoad(IAsyncResult result)
at SimpleIISTestClient.Program.PerfAsynchCallback(IAsyncResult result) in C:\CRF Prototypes\Load\SimpleIISTestClient\Program.cs:line 116]
Inner Exception [9] - Message [The security context token is expired or is not valid. The message was not processed.] StackTrace []