Recover connection when a new master is elected in the cluster - redis

Question

I've a Redis cluster with 3 nodes; 1 is the master and the other 2 are slaves, holding the replica of the master. When I kill the master instance, Redis Sentinel promotes another node to be the master, which starts to accept writes.
During my tests I noticed that once the new master is promoted, the first operation in Redis with SE.Redis fails with:
StackExchange.Redis.RedisConnectionException: SocketFailure on GET
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote
host. ---> System.Net.Sockets.SocketException: An existing connection
was forcibly closed by the remote host
To avoid it, I've implemented a retry logic as below. Is there any better alternative?
private RedisValue RedisGet(string key)
{
return RedisOperation(() =>
{
RedisKey redisKey = key;
RedisValue redisValue = connection.StringGet(redisKey);
return (redisValue);
});
}
private T RedisOperation<T>(Func<T> act)
{
int timeToSleepBeforeRetryInMiliseconds = 20;
DateTime startTime = DateTime.Now;
while (true)
{
try
{
return act();
}
catch (Exception e)
{
Debug.WriteLine("Failed to perform REDIS OP");
TimeSpan passedTime = DateTime.Now - startTime;
if (this.retryTimeout < passedTime)
{
Debug.WriteLine("ABORTING re-try to REDIS OP");
throw;
}
else
{
int remainingTimeout = (int)(this.retryTimeout.TotalMilliseconds - passedTime.TotalMilliseconds);
// if remaining time is less than 1 sec than wait only for that much time and than give a last try
if (remainingTimeout < timeToSleepBeforeRetryInMiliseconds)
{
timeToSleepBeforeRetryInMiliseconds = remainingTimeout;
}
}
Debug.WriteLine("Sleeping " + timeToSleepBeforeRetryInMiliseconds + " before next try");
System.Threading.Thread.Sleep(timeToSleepBeforeRetryInMiliseconds);
}
}
}

Answer 1

TLDR: don't use Sentinel with Stackexchange.Redis as Sentinel support is still not implemented in this client library.
See https://github.com/StackExchange/StackExchange.Redis/labels/sentinel for all the open issues, there is also a pretty good PR open for ~1 year now.
That being said, I also had relatively good experience with retries, but I would never use that approach in production as it is not reliable at all.