Azure Redis Timeout Under load - redis

I've implemented this second level cache for EF Core exactly as described in the readme. Tonight I ran a load test on the system with about 500 virtual users and I am getting lots of timeouts like the following below:
Azure Redis Premium 6 GB
Message: Timeout performing GET (5000ms), next: GET 8664A5E9, inst: 51, qu: 0, qs: 101, aw: False, rs: ReadAsync, ws: Idle, in: 65536, serverEndpoint: xxxxx.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: RD2818785C3C7F, IOCP: (Busy=6,Free=994,Min=4,Max=1000), WORKER: (Busy=63,Free=8128,Min=4,Max=8191), v: 2.1.58.34321 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
Relevant Startup Code
const string providerName1 = "Redis1";
services.AddEFSecondLevelCache(options =>
options.UseEasyCachingCoreProvider(providerName1, CacheExpirationMode.Absolute, TimeSpan.FromMinutes(5)).DisableLogging(true)
);
ConnectionMultiplexer.SetFeatureFlag("preventthreadtheft", true);
services.AddEasyCaching(option => {
option.UseRedis(config => {
config.DBConfig.Configuration = AppConfig.RedisConnectionString;
config.DBConfig.AllowAdmin = true;
}, providerName1);
});
private DbContextOptionsBuilder<DBContext[![enter image description here][2]][2]> GetDBConfig(IContext context) {
var optionsBuilder = new DbContextOptionsBuilder<DBContext>();
optionsBuilder.UseSqlServer(AppConfig.DatabaseConnection)
.AddInterceptors(context.GetInstance<SecondLevelCacheInterceptor>())
//.UseLoggerFactory()
.EnableSensitiveDataLogging(true);
return optionsBuilder;
}

Related

Mapbox - Failed to resolve tileset descriptors: Loading request canceled

I have the following code for download map regions. This function is called on multiple regions asynchronously. In debug/development mode I rarely get this error. But in production release app I get this issue sometimes and the whole offline user experience is ruined. Notice that for users and fields same function is being used to download regions but it fails mostly for fields and not for users.
const downloadPack = async (packName, bounds) => {
console.log("bounds", packName, bounds);
// Delete old pack
await MapboxGL.offlineManager.deletePack(packName);
const progressListener = (offlineRegion, status) => {
if (status.state === "complete") {
console.log(
`Pack: ${packName}`,
formatBytes(status.completedResourceSize),
status.percentage,
status.state,
);
}
};
const errorListener = (offlineRegion, err) => {
console.log("pack error", offlineRegion, err);
};
const packConfig = {
name: packName,
styleURL: styleURL,
minZoom: MAP_CONFIG.MIN_ZOOM_ALLOWED,
maxZoom: MAP_CONFIG.MAX_ZOOM_ALLOWED,
bounds,
};
await MapboxGL.offlineManager.createPack(
packConfig,
progressListener,
errorListener,
);
// console.log("packConfig", packConfig);
};
I am getting the following errors for following regions:
Regions:
LOG bounds field-region-74 [[67.05473691050432, 24.859774187835313], [67.05879400202377, 24.864996599470516]]
LOG bounds field-region-73 [[67.05603555964123, 24.863974062407152], [67.06011966003771, 24.86902696726922]]
LOG bounds field-region-72 [[67.05860259878298, 24.868365914318616], [67.06177770104551, 24.871585472853567]]
LOG bounds field-region-71 [[67.0527601437771, 24.865385527088804], [67.06248793623786, 24.878936670953124]]
LOG bounds field-region-70 [[67.0575708775574, 24.859917106865936], [67.06827409941002, 24.868984873055723]]
Errors:
LOG pack error {"pack":{"metadata":"{\n \"name\" : \"field-region-74\"\n}","bounds":"{\"type\":\"FeatureCollection\",\"features\":[{\"type\":\"Feature\",\"properties\":{},\"geometry\":{\"type\":\"Point\",\"coordinates\":[67.05473691050432,24.859774187835313]}},{\"type\":\"Feature\",\"properties\":{},\"geometry\":{\"type\":\"Point\",\"coordinates\":[67.05879400202377,24.864996599470516]}}]}"},"_metadata":null} {"message":"Failed to resolve tileset descriptors: Loading request canceled","name":"field-region-74"}
LOG pack error {"pack":{"metadata":"{\n \"name\" : \"field-region-73\"\n}","bounds":"{\"type\":\"FeatureCollection\",\"features\":[{\"type\":\"Feature\",\"properties\":{},\"geometry\":{\"type\":\"Point\",\"coordinates\":[67.05603555964123,24.863974062407152]}},{\"type\":\"Feature\",\"properties\":{},\"geometry\":{\"type\":\"Point\",\"coordinates\":[67.06011966003771,24.86902696726922]}}]}"},"_metadata":null} {"name":"field-region-73","message":"Failed to resolve tileset descriptors: Loading request canceled"}
LOG pack error {"pack":{"metadata":"{\n \"name\" : \"field-region-72\"\n}","bounds":"{\"type\":\"FeatureCollection\",\"features\":[{\"type\":\"Feature\",\"properties\":{},\"geometry\":{\"type\":\"Point\",\"coordinates\":[67.05860259878298,24.868365914318616]}},{\"type\":\"Feature\",\"properties\":{},\"geometry\":{\"type\":\"Point\",\"coordinates\":[67.06177770104551,24.871585472853567]}}]}"},"_metadata":null} {"name":"field-region-72","message":"Failed to resolve tileset descriptors: Loading request canceled"}
LOG pack error {"pack":{"metadata":"{\n \"name\" : \"field-region-71\"\n}","bounds":"{\"type\":\"FeatureCollection\",\"features\":[{\"type\":\"Feature\",\"properties\":{},\"geometry\":{\"type\":\"Point\",\"coordinates\":[67.0527601437771,24.865385527088804]}},{\"type\":\"Feature\",\"properties\":{},\"geometry\":{\"type\":\"Point\",\"coordinates\":[67.06248793623786,24.878936670953124]}}]}"},"_metadata":null} {"name":"field-region-71","message":"Failed to resolve tileset descriptors: Loading request canceled"}
LOG Pack: field-region-70 2.18 MB 100 complete
Notice that last pack is downloaded successfully. I even tried to await the function to make them synchronous but still I get the error.

StackExchange.Redis.RedisTimeoutException Timeout performing PSETEX

.NET 7.0 and StackExchange.Redis version=2.6.70.
I am facing this error when the cache is on the server but not when running locally with the cache in container on my machine.
StackExchange.Redis.RedisTimeoutException: Timeout performing PSETEX (5000ms),
inst: 0,
qu: 1,
qs: 0,
aw: False,
bw: SpinningDown,
rs: NotStarted,
ws: Idle,
in: 0,
serverEndpoint: 0.0.24.235:6379,
mc: 1/1/0,
mgr: 10 of 10 available,
clientName: 6fe450d7aad5(SE.Redis-v2.6.70.49541),
IOCP: (Busy=0,Free=1000,Min=1,Max=1000),
WORKER: (Busy=1,Free=32766,Min=32,Max=32767),
POOL: (Threads=5,QueuedItems=0,CompletedItems=279236),
v: 2.6.70.49541 (Please take a look at this article for some common client-side issues that can
cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)\n at
StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1
processor, ServerEndPoint server, T defaultValue) in
//src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1887\n at
StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor,
ServerEndPoint server, T defaultValue) in //src/StackExchange.Redis/RedisBase.cs:line 62\n at
StackExchange.Redis.RedisDatabase.StringSet(RedisKey key, RedisValue value, Nullable`1 expiry,
Boolean keepTtl, When when, CommandFlags flags) in
/_/src/StackExchange.Redis/RedisDatabase.cs:line 3128\n at
ConfigurationManagementMicroService.Cache.CacheService.SetData[T](String key, T value,
DateTimeOffset expirationTime) in
/src/ConfigurationManagementMicroService/Cache/CacheService.cs:line 36\n
This is how I am connecting:
ConnectionHelper.lazyConnection = new Lazy<ConnectionMultiplexer>(() =>
{
var config = new ConfigurationOptions()
{
EndPoints = { { ip, Convert.ToInt32(port) } },
AbortOnConnectFail = false,
};
return ConnectionMultiplexer.Connect(config);
}
Sync handshake could be an issue or big sized payload? Give analysis on this.

The timeout was reached before the message could be written to the output buffer

I'm trying to connect to the AWS Redis instance via the below c# code.
var option = new Microsoft.Extensions.Caching.StackExchangeRedis.RedisCacheOptions
{
ConfigurationOptions = ConfigurationOptions.Parse("master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379,ssl=true,sslProtocols=Tls,abortConnect=false"),
InstanceName = "testapi",
};
IDistributedCache distributedCache = new RedisCache(option);
distributedCache.SetAsync("testKey", System.Text.Encoding.UTF8.GetBytes("testData"));
When I'm trying to set the value in the cache, I'm getting the below error. Though there is enough Threads, i'm getting this error.
{
"Redis-Message": "HMGET testapi1fb31d10-ebee-4f39-9a73-9bc8901461e5",
"Redis-Timeout": "5000",
"Redis-OpsSinceLastHeartbeat": "0",
"Redis-Queue-Awaiting-Write": "0",
"Redis-Queue-Awaiting-Response": "0",
"Redis-Active-Writer": "False",
"Redis-Backlog-Writer": "CheckingForTimeout",
"Redis-Server-Endpoint": "master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379",
"Redis-Multiplexer-Connects": "1/1/0",
"Redis-Manager": "10 of 10 available",
"Redis-Client-Name": "testapi-app-deployment-7674cf88nnl5(SE.Redis-v2.6.70.49541)",
"Redis-ThreadPool-IO-Completion": "(Busy=0,Free=1000,Min=16,Max=1000)",
"Redis-ThreadPool-Workers": "(Busy=0,Free=32767,Min=16,Max=32767)",
"Redis-ThreadPool-Items": "(Threads=16,QueuedItems=0,CompletedItems=495)",
"Redis-Busy-Workers": "0",
"Redis-Version": "2.6.70.49541",
"redis-command": "HMGET testapi1fb31d10-ebee-4f39-9a73-9bc8901461e5",
"redis-server": "master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379",
"ex_msg": "The timeout was reached before the message could be written to the output buffer, and it was not sent, command=HMGET, timeout: 5000, inst: 0, qu: 0, qs: 0, aw: False, bw: CheckingForTimeout, serverEndpoint: master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: testapi-app-deployment-7674cf88nnl5(SE.Redis-v2.6.70.49541), IOCP: (Busy=0,Free=1000,Min=16,Max=1000), WORKER: (Busy=0,Free=32767,Min=16,Max=32767), POOL: (Threads=16,QueuedItems=0,CompletedItems=495), v: 2.6.70.49541 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)"
}
Not able to figure out why.
Since there is no connection error, it seems the code was able to connect to Redis. Also in error payload size is not there so again huge payload size possibility is ruled out.
Please note:
There are many similar questions in stack overflow and tried all of them. But, no luck.
We recently updated our API from .net 3.1 to 6. After this, we started to face this issue, and currently, API is not working at all. There is a sufficient number of threads and works I can see. Also we tried by increasing the timeout but again no luck.

Asp.net Core SignalR loadbalancer (close connection error)

We have a signalR structure that clients only listen to, which is triggered from the Admin Panel. Client's connection can stay connected as long as loadbalancer's (Google Cloud Platform) request timeout. After this time, the browser throws a 1006 close connection error and drops off the connection. How can we increase this time(is it good idea to increase to thousands of seconds?) What's the error we're skipping here?
startup.cs
services.AddSignalR(HubOptions =>
{
HubOptions.EnableDetailedErrors = true;
HubOptions.KeepAliveInterval = TimeSpan.FromSeconds(15);
HubOptions.ClientTimeoutInterval = TimeSpan.FromMinutes(45);
}).AddJsonProtocol().AddStackExchangeRedis(Configuration.GetSection("RedisConfiguration:Host").Value, options =>
{
options.Configuration.ConfigurationChannel = "dataCache";
options.Configuration.ChannelPrefix = "BoardApp";
options.Configuration.AbortOnConnectFail = false;
});
js.Code
var connection = new signalR.HubConnectionBuilder()
.withUrl("/boardscreenhub", { transport: signalR.HttpTransportType.WebSockets })
.withAutomaticReconnect([1500, 1500, 1500, 1500, 1500, 1500, 1500, 1500, 1500, 1500, null])
.configureLogging(signalR.LogLevel.Information).build();
connection.serverTimeoutInMilliseconds = 1800000;
connection.start();
connection.on("receiveMessage", function (message) {
_this4.setBoardScreenData(message);
});
Do you have session affinity enabled for the backend of your loadbalancer? We have had the same issue, and I believe that if you do not, then the connections will not be stable and so keep-alives will get lost and give you this behavior.

Stack Exchange Redis client timeout

I'm struggling to work out what this error message from the Stack Exchange Redis client is telling me:
Unhandled Exception: System.AggregateException: One or more errors
occurred. (Timeout awaiting response (outbound=0KiB, inbound=0KiB,
32516ms elapsed, timeout is 30000ms), inst: 0, qu: 0, qs: 1, in: 0,
serverEndpoint: Unspecified/XXX:6379, mgr: 9 of
10 available, clientName: YYY, IOCP:
(Busy=0,Free=1000,Min=2,Max=1000), WORKER:
(Busy=1025,Free=31742,Min=1024,Max=32767), v: 2.0.571.20511 (Please
take a look at this article for some common client-side issues that
can cause timeouts:
https://stackexchange.github.io/StackExchange.Redis/Timeouts)) --->
StackExchange.Redis.RedisTimeoutException: Timeout awaiting response
(outbound=0KiB, inbound=0KiB, 32516ms elapsed, timeout is 30000ms),
inst: 0, qu: 0, qs: 1, in: 0, serverEndpoint:
Unspecified/XXX:6379, mgr: 9 of 10 available,
clientName: YYY, IOCP: (Busy=0,Free=1000,Min=2,Max=1000),
WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767), v: 2.0.571.20511
(Please take a look at this article for some common client-side issues
that can cause timeouts:
https://stackexchange.github.io/StackExchange.Redis/Timeouts
I'm using this code to select the least loaded connection multiplexer and back off, but I'm still getting timeouts when firing a large number of string set commands.
public class RedisConnectionManager : IRedisConnectionManager
{
private const int MaxQueueLength = 10;
private readonly List<Lazy<ConnectionMultiplexer>> _connectionMultiplexers;
public RedisConnectionManager(List<Lazy<ConnectionMultiplexer>> connectionMultiplexers)
{
this._connectionMultiplexers = connectionMultiplexers;
}
public async Task<ConnectionMultiplexer> GetLeastBusyConnectionAsync()
{
var leastBusyConnection = this._connectionMultiplexers.OrderBy(connection => connection.Value.GetCounters().Interactive.TotalOutstanding).First();
await WaitUntilConnectionAvailableAsync(leastBusyConnection);
return this._connectionMultiplexers.OrderBy(connection => connection.Value.GetCounters().Interactive.TotalOutstanding).First().Value;
}
private static async Task WaitUntilConnectionAvailableAsync(Lazy<ConnectionMultiplexer> leastBusyConnection)
{
while (leastBusyConnection.Value.GetCounters().Interactive.TotalOutstanding > MaxQueueLength)
{
await Task.Delay(100);
}
}
}
There are several answers on SO over regarding this.
In your particular case, it looks like the issue with ThreadPool throtlling, where busy threads are 1025 which are more than Min configured 1024 threads.
WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767)
Have a look here, however beware of the other server performance consequences (depending on your use case) as mentioned in the article.
https://gist.github.com/JonCole/e65411214030f0d823cb#file-threadpool-md