I have a load tester that calls my WCF service and I've built it with options to run the calls in parallel or not. Only when running in parallel, I get the following error for all threads: "The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error."
This is more or less my code:
if (runMultiThreaded)
{
ParallelOptions options = new ParallelOptions();
options.MaxDegreeOfParallelism = System.Environment.ProcessorCount;
ParallelLoopResult loopResult = Parallel.For(0, numberOfTimesToTest, options,
(i, loopState) =>
{
myService.MyOperation();
if (loopState.ShouldExitCurrentIteration) return;
});
}
else
{
for (int i = 0; i < test1NumberOfRuns; i++)
{
myService.MyOperation();
}
}
Any ideas? Let me know if you need more details.
UPDATE: myService is an instance of my service's operation contract interface that was created with a ChannelFactory using the CreateChannel method.
Thanks!
I'm assuming your myService is a ClientBase<T> subclass or a channel created explicitly via ChannelFactory<T>::CreateChannel? If so those instances are not guaranteed to be thread safe and you so you need an instance per worker thread.
Related
I hate the questions that have "Not Enough Info". So I will try to give detailed information. And in this case it is code.
Server:
64 bit of https://github.com/MSOpenTech/redis/tree/2.6/bin/release
There are three classes:
DbOperationContext.cs: https://gist.github.com/glikoz/7119628
PerRequestLifeTimeManager.cs: https://gist.github.com/glikoz/7119699
RedisRepository.cs https://gist.github.com/glikoz/7119769
We are using Redis with Unity ..
In this case we are getting this strange message:
"Redis Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use.";
We checked these:
Is the problem configuration issue
Are we using wrong RedisServer.exe
Is there any architectural problem
Any idea? Any similar story?
Thanks.
Extra Info 1
There is no rejected connection issue on server stats (I've checked it via redis-cli.exe info command)
I have continued to debug this problem, and have fixed numerous things on my platform to avoid this exception. Here is what I have done to solve the issue:
Executive summary:
People encountering this exception should check:
That the PooledRedisClientsManager (IRedisClientsManager) is registed in a singleton scope
That the RedisMqServer (IMessageService) is registered in a singleton scope
That any utilized RedisClient returned from either of the above is properly disposed of, to ensure that the pooled clients are not left stale.
The solution to my problem:
First of all, this exception is thrown by the PooledRedisClient because it has no more pooled connections available.
I'm registering all the required Redis stuff in the StructureMap IoC container (not unity as in the author's case). Thanks to this post I was reminded that the PooledRedisClientManager should be a singleton - I also decided to register the RedisMqServer as a singleton:
ObjectFactory.Configure(x =>
{
// register the message queue stuff as Singletons in this AppDomain
x.For<IRedisClientsManager>()
.Singleton()
.Use(BuildRedisClientsManager);
x.For<IMessageService>()
.Singleton()
.Use<RedisMqServer>()
.Ctor<IRedisClientsManager>().Is(i => i.GetInstance<IRedisClientsManager>())
.Ctor<int>("retryCount").Is(2)
.Ctor<TimeSpan?>().Is(TimeSpan.FromSeconds(5));
// Retrieve a new message factory from the singleton IMessageService
x.For<IMessageFactory>()
.Use(i => i.GetInstance<IMessageService>().MessageFactory);
});
My "BuildRedisClientManager" function looks like this:
private static IRedisClientsManager BuildRedisClientsManager()
{
var appSettings = new AppSettings();
var redisClients = appSettings.Get("redis-servers", "redis.local:6379").Split(',');
var redisFactory = new PooledRedisClientManager(redisClients);
redisFactory.ConnectTimeout = 5;
redisFactory.IdleTimeOutSecs = 30;
redisFactory.PoolTimeout = 3;
return redisFactory;
}
Then, when it comes to producing messages it's very important that the utilized RedisClient is properly disposed of, otherwise we run into the dreaded "Timeout Expired" (thanks to this post). I have the following helper code to send a message to the queue:
public static void PublishMessage<T>(T msg)
{
try
{
using (var producer = GetMessageProducer())
{
producer.Publish<T>(msg);
}
}
catch (Exception ex)
{
// TODO: Log or whatever... I'm not throwing to avoid showing users that we have a broken MQ
}
}
private static IMessageQueueClient GetMessageProducer()
{
var producer = ObjectFactory.GetInstance<IMessageService>() as RedisMqServer;
var client = producer.CreateMessageQueueClient();
return client;
}
I hope this helps solve your issue too.
I've got a service and have verified using "netstat -anb" that when the service is running, it's listening on the correct port (8040). The service contract contains the following contract:
[OperationContract]
bool RegisterPlayer();
The service class itself implements the contract explicitly:
bool IMechService.RegisterPlayer()
{
if (P1 != null)
{
P1 = OperationContext.Current.GetCallbackChannel<IMechServiceCallback>();
return true;
}
else if (P2 != null)
{
P2 = OperationContext.Current.GetCallbackChannel<IMechServiceCallback>();
return true;
}
return false;
}
And the svcutil generated proxy creates the following method:
public bool RegisterPlayer()
{
return base.Channel.RegisterPlayer();
}
This code attempts to generate a proxy and call the method. I've tried both using DuplexChannelFactory and the svcutil generated proxy class, and both give the same results:
client = new MechServiceClient(new InstanceContext(this));
//client = DuplexChannelFactory<IMechService>.CreateChannel(this, new NetTcpBinding(), new EndpointAddress("net.tcp://localhost:8040/MechService"));
client.RegisterPlayer();
Code execution reaches the RegisterPlayer in the proxy class, but proceeds to time out, never running RegisterPlayer on the service. Unfortunately, as it's just timing out, I'm not getting any exceptions or errors to help indicate where to look for issues. So far, I've verified the service is running and appears to be listening on port 8040 using "netstat -anb", and I've established that the mex endpoint is working as intended and publishing metadata. I turned off Windows Firewall. I've also created a separate test project with much simpler implementations to verify I was doing the steps correctly, and the simpler test project works fine. I'm out of ideas for what's causing this to fail, and any advice would be appreciated.
Have you tried setting the ConcurrencyMode to ConcurrencyMode.Multiple?
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)]
class MechServiceImpl : IMechService
{
// ..
}
The default concurrency mode for a service is ConcurrencyMode.Single, which can cause complications with callbacks.
Andrew's suggestion to logging helped, basically what fixed it was declaring my OperationContracts to isoneway=true.
My client is using one WCF service which is throwing an exception
(EXCEPTION: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state).
All subsequent calls throwing an same exception.
I read on internet that client need to close()/Abort() channel, this will solve the problem. is it completely right?
Also I am using customer serviceChannel factory provided by service developers. When I create channel it does not show the close and abort methods. So how do I get these close and abort methods when I create custom service channel instance on client side?
Assuming that you have a proxy instance that implements the IClientChannel interface, here is a way (hopefully the right way) to use it.
IClientChannel clientChannel = (IClientChannel)proxy;
bool success = false;
try
{
// do something with the proxy
clientChannel.Close();
success = true;
}
finally
{
if (!success)
{
clientChannel.Abort();
}
}
You may also want to check this. You can wrap your operations using a shared class or function.
I have a WCF client that has thrown this common error, just to be resolved with retrying the HTTP call to the server. For what it's worth this exception was not generated within 1 minute. It was generated in 3 seconds.
The request operation sent to xxxxxx
did not receive a reply within the
configured timeout (00:01:00). The
time allotted to this operation may
have been a portion of a longer
timeout. This may be because the
service is still processing the
operation or because the service was
unable to send a reply message. Please
consider increasing the operation
timeout (by casting the channel/proxy
to IContextChannel and setting the
OperationTimeout property) and ensure
that the service is able to connect to
the client
How are professionals handling these common WCF errors? What other bogus errors should I handle.
For example, I'm considering timing the WCF call and if that above (bogus) error is thrown in under 55 seconds, I retry the entire operation (using a while() loop). I believe I have to reset the entire channel, but I'm hoping you guys will tell me what's right to do.
What other
I make all of my WCF calls from a custom "using" statement which handles exceptions and potential retires. My code optionally allows me to pass a policy object to the statement so I can easily change the behavior, like if I don't want to retry on error.
The gist of the code is as follows:
[MethodImpl(MethodImplOptions.NoInlining)]
public static void ProxyUsing<T>(ClientBase<T> proxy, Action action)
where T : class
{
try
{
proxy.Open();
using(OperationContextScope context = new OperationContextScope(proxy.InnerChannel))
{
//Add some headers here, or whatever you want
action();
}
}
catch(FaultException fe)
{
//Handle stuff here
}
finally
{
try
{
if(proxy != null
&& proxy.State != CommunicationState.Faulted)
{
proxy.Close();
}
else
{
proxy.Abort();
}
}
catch
{
if(proxy != null)
{
proxy.Abort();
}
}
}
}
You can then use the call like follows:
ProxyUsing<IMyService>(myService = GetServiceInstance(), () =>
{
myService.SomeMethod(...);
});
The NoInlining call probably isn't important for you. I need it because I have some custom logging code that logs the call stack after an exception, so it's important to preserve that method hierarchy in that case.
The backgound: I am trying to forward the server-side ApplyChangeFailed event that is fired by a Sync Services for ADO 1.0 DBServerSyncProvider to the client. All the code examples for Sync Services conflict resolution do not use WCF, and when the client connects to the server database directly, this problem does not exist. My DBServerSyncProvider is wrapped by a head-less WCF service, however, and I cannot show the user a dialog with the offending data for review.
So, the obvious solution seemed to be to convert the HTTP WCF service that Sync Services generated to TCP, make it a duplex connection, and define a callback handler on the client that receives the SyncConflict object and sets the Action property of the event.
When I did that, I got a runtime error (before the callback was attempted):
System.InvalidOperationException: This operation would deadlock because the
reply cannot be received until the current Message completes processing. If
you want to allow out-of-order message processing, specify ConcurrencyMode of
Reentrant or Multiple on CallbackBehaviorAttribute.
So I did what the message suggested and decorated both the service and the callback behavior with the Multiple attribute. Then the runtime error went away, but the call results in a "deadlock" and never returns. What do I do to get around this? Is it not possible to have a WCF service that calls back the client before the original service call returns?
Edit: I think this could be the explanation of the issue, but I am still not sure what the correct solution should be.
After updating the ConcurrencyMode have you tried firing the callback in a seperate thread?
This answer to another question has some example code that starts another thread and passes through the callback, you might be able to modify that design for your purpose?
By starting the sync agent in a separate thread on the client, the callback works just fine:
private int kickOffSyncInSeparateThread()
{
SyncRunner syncRunner = new SyncRunner();
Thread syncThread = new Thread(
new ThreadStart(syncRunner.RunSyncInThread));
try
{
syncThread.Start();
}
catch (ThreadStateException ex)
{
Console.WriteLine(ex);
return 1;
}
catch (ThreadInterruptedException ex)
{
Console.WriteLine(ex);
return 2;
}
return 0;
}
And this is my SyncRunner:
class SyncRunner
{
public void RunSyncInThread()
{
MysyncAgent = new MySyncAgent();
syncAgent.addUserIdParameter("56623239-d855-de11-8e97-0016cfe25fa3");
Microsoft.Synchronization.Data.SyncStatistics syncStats =
syncAgent.Synchronize();
}
}