Stack Exchange Redis client timeout - redis

Question

I'm struggling to work out what this error message from the Stack Exchange Redis client is telling me:
Unhandled Exception: System.AggregateException: One or more errors
occurred. (Timeout awaiting response (outbound=0KiB, inbound=0KiB,
32516ms elapsed, timeout is 30000ms), inst: 0, qu: 0, qs: 1, in: 0,
serverEndpoint: Unspecified/XXX:6379, mgr: 9 of
10 available, clientName: YYY, IOCP:
(Busy=0,Free=1000,Min=2,Max=1000), WORKER:
(Busy=1025,Free=31742,Min=1024,Max=32767), v: 2.0.571.20511 (Please
take a look at this article for some common client-side issues that
can cause timeouts:
https://stackexchange.github.io/StackExchange.Redis/Timeouts)) --->
StackExchange.Redis.RedisTimeoutException: Timeout awaiting response
(outbound=0KiB, inbound=0KiB, 32516ms elapsed, timeout is 30000ms),
inst: 0, qu: 0, qs: 1, in: 0, serverEndpoint:
Unspecified/XXX:6379, mgr: 9 of 10 available,
clientName: YYY, IOCP: (Busy=0,Free=1000,Min=2,Max=1000),
WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767), v: 2.0.571.20511
(Please take a look at this article for some common client-side issues
that can cause timeouts:
https://stackexchange.github.io/StackExchange.Redis/Timeouts
I'm using this code to select the least loaded connection multiplexer and back off, but I'm still getting timeouts when firing a large number of string set commands.
public class RedisConnectionManager : IRedisConnectionManager
{
private const int MaxQueueLength = 10;
private readonly List<Lazy<ConnectionMultiplexer>> _connectionMultiplexers;
public RedisConnectionManager(List<Lazy<ConnectionMultiplexer>> connectionMultiplexers)
{
this._connectionMultiplexers = connectionMultiplexers;
}
public async Task<ConnectionMultiplexer> GetLeastBusyConnectionAsync()
{
var leastBusyConnection = this._connectionMultiplexers.OrderBy(connection => connection.Value.GetCounters().Interactive.TotalOutstanding).First();
await WaitUntilConnectionAvailableAsync(leastBusyConnection);
return this._connectionMultiplexers.OrderBy(connection => connection.Value.GetCounters().Interactive.TotalOutstanding).First().Value;
}
private static async Task WaitUntilConnectionAvailableAsync(Lazy<ConnectionMultiplexer> leastBusyConnection)
{
while (leastBusyConnection.Value.GetCounters().Interactive.TotalOutstanding > MaxQueueLength)
{
await Task.Delay(100);
}
}
}

Answer 1

There are several answers on SO over regarding this.
In your particular case, it looks like the issue with ThreadPool throtlling, where busy threads are 1025 which are more than Min configured 1024 threads.
WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767)
Have a look here, however beware of the other server performance consequences (depending on your use case) as mentioned in the article.
https://gist.github.com/JonCole/e65411214030f0d823cb#file-threadpool-md