.NET 7.0 and StackExchange.Redis version=2.6.70.
I am facing this error when the cache is on the server but not when running locally with the cache in container on my machine.
StackExchange.Redis.RedisTimeoutException: Timeout performing PSETEX (5000ms),
inst: 0,
qu: 1,
qs: 0,
aw: False,
bw: SpinningDown,
rs: NotStarted,
ws: Idle,
in: 0,
serverEndpoint: 0.0.24.235:6379,
mc: 1/1/0,
mgr: 10 of 10 available,
clientName: 6fe450d7aad5(SE.Redis-v2.6.70.49541),
IOCP: (Busy=0,Free=1000,Min=1,Max=1000),
WORKER: (Busy=1,Free=32766,Min=32,Max=32767),
POOL: (Threads=5,QueuedItems=0,CompletedItems=279236),
v: 2.6.70.49541 (Please take a look at this article for some common client-side issues that can
cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)\n at
StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1
processor, ServerEndPoint server, T defaultValue) in
//src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1887\n at
StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor,
ServerEndPoint server, T defaultValue) in //src/StackExchange.Redis/RedisBase.cs:line 62\n at
StackExchange.Redis.RedisDatabase.StringSet(RedisKey key, RedisValue value, Nullable`1 expiry,
Boolean keepTtl, When when, CommandFlags flags) in
/_/src/StackExchange.Redis/RedisDatabase.cs:line 3128\n at
ConfigurationManagementMicroService.Cache.CacheService.SetData[T](String key, T value,
DateTimeOffset expirationTime) in
/src/ConfigurationManagementMicroService/Cache/CacheService.cs:line 36\n
This is how I am connecting:
ConnectionHelper.lazyConnection = new Lazy<ConnectionMultiplexer>(() =>
{
var config = new ConfigurationOptions()
{
EndPoints = { { ip, Convert.ToInt32(port) } },
AbortOnConnectFail = false,
};
return ConnectionMultiplexer.Connect(config);
}
Sync handshake could be an issue or big sized payload? Give analysis on this.
Related
I'm trying to connect to the AWS Redis instance via the below c# code.
var option = new Microsoft.Extensions.Caching.StackExchangeRedis.RedisCacheOptions
{
ConfigurationOptions = ConfigurationOptions.Parse("master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379,ssl=true,sslProtocols=Tls,abortConnect=false"),
InstanceName = "testapi",
};
IDistributedCache distributedCache = new RedisCache(option);
distributedCache.SetAsync("testKey", System.Text.Encoding.UTF8.GetBytes("testData"));
When I'm trying to set the value in the cache, I'm getting the below error. Though there is enough Threads, i'm getting this error.
{
"Redis-Message": "HMGET testapi1fb31d10-ebee-4f39-9a73-9bc8901461e5",
"Redis-Timeout": "5000",
"Redis-OpsSinceLastHeartbeat": "0",
"Redis-Queue-Awaiting-Write": "0",
"Redis-Queue-Awaiting-Response": "0",
"Redis-Active-Writer": "False",
"Redis-Backlog-Writer": "CheckingForTimeout",
"Redis-Server-Endpoint": "master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379",
"Redis-Multiplexer-Connects": "1/1/0",
"Redis-Manager": "10 of 10 available",
"Redis-Client-Name": "testapi-app-deployment-7674cf88nnl5(SE.Redis-v2.6.70.49541)",
"Redis-ThreadPool-IO-Completion": "(Busy=0,Free=1000,Min=16,Max=1000)",
"Redis-ThreadPool-Workers": "(Busy=0,Free=32767,Min=16,Max=32767)",
"Redis-ThreadPool-Items": "(Threads=16,QueuedItems=0,CompletedItems=495)",
"Redis-Busy-Workers": "0",
"Redis-Version": "2.6.70.49541",
"redis-command": "HMGET testapi1fb31d10-ebee-4f39-9a73-9bc8901461e5",
"redis-server": "master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379",
"ex_msg": "The timeout was reached before the message could be written to the output buffer, and it was not sent, command=HMGET, timeout: 5000, inst: 0, qu: 0, qs: 0, aw: False, bw: CheckingForTimeout, serverEndpoint: master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: testapi-app-deployment-7674cf88nnl5(SE.Redis-v2.6.70.49541), IOCP: (Busy=0,Free=1000,Min=16,Max=1000), WORKER: (Busy=0,Free=32767,Min=16,Max=32767), POOL: (Threads=16,QueuedItems=0,CompletedItems=495), v: 2.6.70.49541 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)"
}
Not able to figure out why.
Since there is no connection error, it seems the code was able to connect to Redis. Also in error payload size is not there so again huge payload size possibility is ruled out.
Please note:
There are many similar questions in stack overflow and tried all of them. But, no luck.
We recently updated our API from .net 3.1 to 6. After this, we started to face this issue, and currently, API is not working at all. There is a sufficient number of threads and works I can see. Also we tried by increasing the timeout but again no luck.
We have a signalR structure that clients only listen to, which is triggered from the Admin Panel. Client's connection can stay connected as long as loadbalancer's (Google Cloud Platform) request timeout. After this time, the browser throws a 1006 close connection error and drops off the connection. How can we increase this time(is it good idea to increase to thousands of seconds?) What's the error we're skipping here?
startup.cs
services.AddSignalR(HubOptions =>
{
HubOptions.EnableDetailedErrors = true;
HubOptions.KeepAliveInterval = TimeSpan.FromSeconds(15);
HubOptions.ClientTimeoutInterval = TimeSpan.FromMinutes(45);
}).AddJsonProtocol().AddStackExchangeRedis(Configuration.GetSection("RedisConfiguration:Host").Value, options =>
{
options.Configuration.ConfigurationChannel = "dataCache";
options.Configuration.ChannelPrefix = "BoardApp";
options.Configuration.AbortOnConnectFail = false;
});
js.Code
var connection = new signalR.HubConnectionBuilder()
.withUrl("/boardscreenhub", { transport: signalR.HttpTransportType.WebSockets })
.withAutomaticReconnect([1500, 1500, 1500, 1500, 1500, 1500, 1500, 1500, 1500, 1500, null])
.configureLogging(signalR.LogLevel.Information).build();
connection.serverTimeoutInMilliseconds = 1800000;
connection.start();
connection.on("receiveMessage", function (message) {
_this4.setBoardScreenData(message);
});
Do you have session affinity enabled for the backend of your loadbalancer? We have had the same issue, and I believe that if you do not, then the connections will not be stable and so keep-alives will get lost and give you this behavior.
I've implemented this second level cache for EF Core exactly as described in the readme. Tonight I ran a load test on the system with about 500 virtual users and I am getting lots of timeouts like the following below:
Azure Redis Premium 6 GB
Message: Timeout performing GET (5000ms), next: GET 8664A5E9, inst: 51, qu: 0, qs: 101, aw: False, rs: ReadAsync, ws: Idle, in: 65536, serverEndpoint: xxxxx.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: RD2818785C3C7F, IOCP: (Busy=6,Free=994,Min=4,Max=1000), WORKER: (Busy=63,Free=8128,Min=4,Max=8191), v: 2.1.58.34321 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
Relevant Startup Code
const string providerName1 = "Redis1";
services.AddEFSecondLevelCache(options =>
options.UseEasyCachingCoreProvider(providerName1, CacheExpirationMode.Absolute, TimeSpan.FromMinutes(5)).DisableLogging(true)
);
ConnectionMultiplexer.SetFeatureFlag("preventthreadtheft", true);
services.AddEasyCaching(option => {
option.UseRedis(config => {
config.DBConfig.Configuration = AppConfig.RedisConnectionString;
config.DBConfig.AllowAdmin = true;
}, providerName1);
});
private DbContextOptionsBuilder<DBContext[![enter image description here][2]][2]> GetDBConfig(IContext context) {
var optionsBuilder = new DbContextOptionsBuilder<DBContext>();
optionsBuilder.UseSqlServer(AppConfig.DatabaseConnection)
.AddInterceptors(context.GetInstance<SecondLevelCacheInterceptor>())
//.UseLoggerFactory()
.EnableSensitiveDataLogging(true);
return optionsBuilder;
}
I'm struggling to work out what this error message from the Stack Exchange Redis client is telling me:
Unhandled Exception: System.AggregateException: One or more errors
occurred. (Timeout awaiting response (outbound=0KiB, inbound=0KiB,
32516ms elapsed, timeout is 30000ms), inst: 0, qu: 0, qs: 1, in: 0,
serverEndpoint: Unspecified/XXX:6379, mgr: 9 of
10 available, clientName: YYY, IOCP:
(Busy=0,Free=1000,Min=2,Max=1000), WORKER:
(Busy=1025,Free=31742,Min=1024,Max=32767), v: 2.0.571.20511 (Please
take a look at this article for some common client-side issues that
can cause timeouts:
https://stackexchange.github.io/StackExchange.Redis/Timeouts)) --->
StackExchange.Redis.RedisTimeoutException: Timeout awaiting response
(outbound=0KiB, inbound=0KiB, 32516ms elapsed, timeout is 30000ms),
inst: 0, qu: 0, qs: 1, in: 0, serverEndpoint:
Unspecified/XXX:6379, mgr: 9 of 10 available,
clientName: YYY, IOCP: (Busy=0,Free=1000,Min=2,Max=1000),
WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767), v: 2.0.571.20511
(Please take a look at this article for some common client-side issues
that can cause timeouts:
https://stackexchange.github.io/StackExchange.Redis/Timeouts
I'm using this code to select the least loaded connection multiplexer and back off, but I'm still getting timeouts when firing a large number of string set commands.
public class RedisConnectionManager : IRedisConnectionManager
{
private const int MaxQueueLength = 10;
private readonly List<Lazy<ConnectionMultiplexer>> _connectionMultiplexers;
public RedisConnectionManager(List<Lazy<ConnectionMultiplexer>> connectionMultiplexers)
{
this._connectionMultiplexers = connectionMultiplexers;
}
public async Task<ConnectionMultiplexer> GetLeastBusyConnectionAsync()
{
var leastBusyConnection = this._connectionMultiplexers.OrderBy(connection => connection.Value.GetCounters().Interactive.TotalOutstanding).First();
await WaitUntilConnectionAvailableAsync(leastBusyConnection);
return this._connectionMultiplexers.OrderBy(connection => connection.Value.GetCounters().Interactive.TotalOutstanding).First().Value;
}
private static async Task WaitUntilConnectionAvailableAsync(Lazy<ConnectionMultiplexer> leastBusyConnection)
{
while (leastBusyConnection.Value.GetCounters().Interactive.TotalOutstanding > MaxQueueLength)
{
await Task.Delay(100);
}
}
}
There are several answers on SO over regarding this.
In your particular case, it looks like the issue with ThreadPool throtlling, where busy threads are 1025 which are more than Min configured 1024 threads.
WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767)
Have a look here, however beware of the other server performance consequences (depending on your use case) as mentioned in the article.
https://gist.github.com/JonCole/e65411214030f0d823cb#file-threadpool-md
I'm using kurento media server 6.6.1 on a Ubuntu 10.4 box in AWS. It runs fine normally, and I'm attempting to use wss, so I uncomment the security lines in /etc/kurento/kurento.conf.json so the file looks like:
{
"mediaServer" : {
"resources": {
// //Resources usage limit for raising an exception when an object creation is attempted
// "exceptionLimit": "0.8",
// // Resources usage limit for restarting the server when no objects are alive
// "killLimit": "0.7",
// Garbage collector period in seconds
"garbageCollectorPeriod": 240
},
"net" : {
"websocket": {
"port": 8888,
"secure": {
"port": 8433,
"certificate": "/etc/letsencrypt/archive/<MYDOMAIN>/fullchain1.pem",
"password": ""
},
//"registrar": {
// "address": "ws://localhost:9090",
// "localAddress": "localhost"
//},
"path": "kurento",
"threads": 10
}
}
}
}
I've also verified that my certs are in the correct location and are valid.
When I restart the service, I get the following output in my /var/log/kurento-media-server/media-server_error.log file, and the service doesn't start (or give any other logs).
kurento-media-server: /usr/include/boost/smart_ptr/shared_ptr.hpp:648: typename boost::detail::sp_member_access<T>::type boost::shared_ptr<T>::operator->() const [with T = boost::log::v2_mt_posix::core; typename boost::detail::sp_member_access<T>::type = boost::log::v2_mt_posix::core*]: Assertion `px != 0' failed.
Aborted (thread 140077491206272, pid 7622)
Stack trace:
[gsignal]
/lib/x86_64-linux-gnu/libc.so.6:0x35428
[abort]
/lib/x86_64-linux-gnu/libc.so.6:0x3702A
[uselocale]
/lib/x86_64-linux-gnu/libc.so.6:0x2DBD7
[__assert_fail]
/lib/x86_64-linux-gnu/libc.so.6:0x2DC82
0x4CC67A at /usr/bin/kurento-media-server
[kurento::kms_init_logging(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)]
/usr/bin/kurento-media-server:0x4EF234
[gst_debug_log_valist]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x62EE6
[gst_debug_log]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.5.so.0:0x6303B
[kurento::MediaSet::~MediaSet()]
/usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6:0xA809B
[std::_Sp_counted_ptr<kurento::MediaSet*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()]
/usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6:0xB06D2
[std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()]
/usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6:0xA3916
[__cxa_finalize]
/lib/x86_64-linux-gnu/libc.so.6:0x3A36A
0xA2B53 at /usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6