I'm trying to connect to the AWS Redis instance via the below c# code.
var option = new Microsoft.Extensions.Caching.StackExchangeRedis.RedisCacheOptions
{
ConfigurationOptions = ConfigurationOptions.Parse("master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379,ssl=true,sslProtocols=Tls,abortConnect=false"),
InstanceName = "testapi",
};
IDistributedCache distributedCache = new RedisCache(option);
distributedCache.SetAsync("testKey", System.Text.Encoding.UTF8.GetBytes("testData"));
When I'm trying to set the value in the cache, I'm getting the below error. Though there is enough Threads, i'm getting this error.
{
"Redis-Message": "HMGET testapi1fb31d10-ebee-4f39-9a73-9bc8901461e5",
"Redis-Timeout": "5000",
"Redis-OpsSinceLastHeartbeat": "0",
"Redis-Queue-Awaiting-Write": "0",
"Redis-Queue-Awaiting-Response": "0",
"Redis-Active-Writer": "False",
"Redis-Backlog-Writer": "CheckingForTimeout",
"Redis-Server-Endpoint": "master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379",
"Redis-Multiplexer-Connects": "1/1/0",
"Redis-Manager": "10 of 10 available",
"Redis-Client-Name": "testapi-app-deployment-7674cf88nnl5(SE.Redis-v2.6.70.49541)",
"Redis-ThreadPool-IO-Completion": "(Busy=0,Free=1000,Min=16,Max=1000)",
"Redis-ThreadPool-Workers": "(Busy=0,Free=32767,Min=16,Max=32767)",
"Redis-ThreadPool-Items": "(Threads=16,QueuedItems=0,CompletedItems=495)",
"Redis-Busy-Workers": "0",
"Redis-Version": "2.6.70.49541",
"redis-command": "HMGET testapi1fb31d10-ebee-4f39-9a73-9bc8901461e5",
"redis-server": "master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379",
"ex_msg": "The timeout was reached before the message could be written to the output buffer, and it was not sent, command=HMGET, timeout: 5000, inst: 0, qu: 0, qs: 0, aw: False, bw: CheckingForTimeout, serverEndpoint: master.test-redis-cluster.89run.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: testapi-app-deployment-7674cf88nnl5(SE.Redis-v2.6.70.49541), IOCP: (Busy=0,Free=1000,Min=16,Max=1000), WORKER: (Busy=0,Free=32767,Min=16,Max=32767), POOL: (Threads=16,QueuedItems=0,CompletedItems=495), v: 2.6.70.49541 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)"
}
Not able to figure out why.
Since there is no connection error, it seems the code was able to connect to Redis. Also in error payload size is not there so again huge payload size possibility is ruled out.
Please note:
There are many similar questions in stack overflow and tried all of them. But, no luck.
We recently updated our API from .net 3.1 to 6. After this, we started to face this issue, and currently, API is not working at all. There is a sufficient number of threads and works I can see. Also we tried by increasing the timeout but again no luck.
Related
.NET 7.0 and StackExchange.Redis version=2.6.70.
I am facing this error when the cache is on the server but not when running locally with the cache in container on my machine.
StackExchange.Redis.RedisTimeoutException: Timeout performing PSETEX (5000ms),
inst: 0,
qu: 1,
qs: 0,
aw: False,
bw: SpinningDown,
rs: NotStarted,
ws: Idle,
in: 0,
serverEndpoint: 0.0.24.235:6379,
mc: 1/1/0,
mgr: 10 of 10 available,
clientName: 6fe450d7aad5(SE.Redis-v2.6.70.49541),
IOCP: (Busy=0,Free=1000,Min=1,Max=1000),
WORKER: (Busy=1,Free=32766,Min=32,Max=32767),
POOL: (Threads=5,QueuedItems=0,CompletedItems=279236),
v: 2.6.70.49541 (Please take a look at this article for some common client-side issues that can
cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)\n at
StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1
processor, ServerEndPoint server, T defaultValue) in
//src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1887\n at
StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor,
ServerEndPoint server, T defaultValue) in //src/StackExchange.Redis/RedisBase.cs:line 62\n at
StackExchange.Redis.RedisDatabase.StringSet(RedisKey key, RedisValue value, Nullable`1 expiry,
Boolean keepTtl, When when, CommandFlags flags) in
/_/src/StackExchange.Redis/RedisDatabase.cs:line 3128\n at
ConfigurationManagementMicroService.Cache.CacheService.SetData[T](String key, T value,
DateTimeOffset expirationTime) in
/src/ConfigurationManagementMicroService/Cache/CacheService.cs:line 36\n
This is how I am connecting:
ConnectionHelper.lazyConnection = new Lazy<ConnectionMultiplexer>(() =>
{
var config = new ConfigurationOptions()
{
EndPoints = { { ip, Convert.ToInt32(port) } },
AbortOnConnectFail = false,
};
return ConnectionMultiplexer.Connect(config);
}
Sync handshake could be an issue or big sized payload? Give analysis on this.
I've implemented this second level cache for EF Core exactly as described in the readme. Tonight I ran a load test on the system with about 500 virtual users and I am getting lots of timeouts like the following below:
Azure Redis Premium 6 GB
Message: Timeout performing GET (5000ms), next: GET 8664A5E9, inst: 51, qu: 0, qs: 101, aw: False, rs: ReadAsync, ws: Idle, in: 65536, serverEndpoint: xxxxx.redis.cache.windows.net:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: RD2818785C3C7F, IOCP: (Busy=6,Free=994,Min=4,Max=1000), WORKER: (Busy=63,Free=8128,Min=4,Max=8191), v: 2.1.58.34321 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
Relevant Startup Code
const string providerName1 = "Redis1";
services.AddEFSecondLevelCache(options =>
options.UseEasyCachingCoreProvider(providerName1, CacheExpirationMode.Absolute, TimeSpan.FromMinutes(5)).DisableLogging(true)
);
ConnectionMultiplexer.SetFeatureFlag("preventthreadtheft", true);
services.AddEasyCaching(option => {
option.UseRedis(config => {
config.DBConfig.Configuration = AppConfig.RedisConnectionString;
config.DBConfig.AllowAdmin = true;
}, providerName1);
});
private DbContextOptionsBuilder<DBContext[![enter image description here][2]][2]> GetDBConfig(IContext context) {
var optionsBuilder = new DbContextOptionsBuilder<DBContext>();
optionsBuilder.UseSqlServer(AppConfig.DatabaseConnection)
.AddInterceptors(context.GetInstance<SecondLevelCacheInterceptor>())
//.UseLoggerFactory()
.EnableSensitiveDataLogging(true);
return optionsBuilder;
}
I'm struggling to work out what this error message from the Stack Exchange Redis client is telling me:
Unhandled Exception: System.AggregateException: One or more errors
occurred. (Timeout awaiting response (outbound=0KiB, inbound=0KiB,
32516ms elapsed, timeout is 30000ms), inst: 0, qu: 0, qs: 1, in: 0,
serverEndpoint: Unspecified/XXX:6379, mgr: 9 of
10 available, clientName: YYY, IOCP:
(Busy=0,Free=1000,Min=2,Max=1000), WORKER:
(Busy=1025,Free=31742,Min=1024,Max=32767), v: 2.0.571.20511 (Please
take a look at this article for some common client-side issues that
can cause timeouts:
https://stackexchange.github.io/StackExchange.Redis/Timeouts)) --->
StackExchange.Redis.RedisTimeoutException: Timeout awaiting response
(outbound=0KiB, inbound=0KiB, 32516ms elapsed, timeout is 30000ms),
inst: 0, qu: 0, qs: 1, in: 0, serverEndpoint:
Unspecified/XXX:6379, mgr: 9 of 10 available,
clientName: YYY, IOCP: (Busy=0,Free=1000,Min=2,Max=1000),
WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767), v: 2.0.571.20511
(Please take a look at this article for some common client-side issues
that can cause timeouts:
https://stackexchange.github.io/StackExchange.Redis/Timeouts
I'm using this code to select the least loaded connection multiplexer and back off, but I'm still getting timeouts when firing a large number of string set commands.
public class RedisConnectionManager : IRedisConnectionManager
{
private const int MaxQueueLength = 10;
private readonly List<Lazy<ConnectionMultiplexer>> _connectionMultiplexers;
public RedisConnectionManager(List<Lazy<ConnectionMultiplexer>> connectionMultiplexers)
{
this._connectionMultiplexers = connectionMultiplexers;
}
public async Task<ConnectionMultiplexer> GetLeastBusyConnectionAsync()
{
var leastBusyConnection = this._connectionMultiplexers.OrderBy(connection => connection.Value.GetCounters().Interactive.TotalOutstanding).First();
await WaitUntilConnectionAvailableAsync(leastBusyConnection);
return this._connectionMultiplexers.OrderBy(connection => connection.Value.GetCounters().Interactive.TotalOutstanding).First().Value;
}
private static async Task WaitUntilConnectionAvailableAsync(Lazy<ConnectionMultiplexer> leastBusyConnection)
{
while (leastBusyConnection.Value.GetCounters().Interactive.TotalOutstanding > MaxQueueLength)
{
await Task.Delay(100);
}
}
}
There are several answers on SO over regarding this.
In your particular case, it looks like the issue with ThreadPool throtlling, where busy threads are 1025 which are more than Min configured 1024 threads.
WORKER: (Busy=1025,Free=31742,Min=1024,Max=32767)
Have a look here, however beware of the other server performance consequences (depending on your use case) as mentioned in the article.
https://gist.github.com/JonCole/e65411214030f0d823cb#file-threadpool-md
There are publisher and consumer custom modules that are running on an Edge IoT device. The publisher module keeps producing messages at constant rate no matter consumer module processes it or not. The consumer module POSTs the message to external service and given that there is no Internet connection, the consumer module would like to requeue the messsage so that is not lost and tried again.
I do not prefer to write an infinite loop to keep retrying; also if the module is restarted the message would be lost. So i prefer to requeue the message to edgeHub/RocksDB.
Where do I find documentation on available responses that can be provided for IoTHubMessageDispositionResult? what is the response to be sent if message needs to be requeued?
if message.processed():
return IoTHubMessageDispositionResult.ACCEPTED
else:
return IoTHubMessageDispositionResult.??
You don't have to implement your own requeuing of messages. IotEdge provides offline functionality as described in this blog post and on this documentation page.
The edgeHub will locally store messages on the edgeDevice if there is no connection to the IotHub. It will automatically start sending those messages (in the correct order) once the connection is established again.
You can configure how long edgeHub will buffer messages like this:
"$edgeHub": {
"properties.desired": {
"schemaVersion": "1.0",
"routes": {},
"storeAndForwardConfiguration": {
"timeToLiveSecs": 7200
}
}
}
The 7200 seconds (2 hours) is also the default if you don't configure anything.
By default, the messages will be written to a folder within the edgeHub docker container. If you want to store them somewhere else you can do so with this configuration:
"edgeHub": {
"type": "docker",
"settings": {
"image": "mcr.microsoft.com/azureiotedge-hub:1.0",
"createOptions": {
"HostConfig": {
"Binds": ["<HostStoragePath>:<ModuleStoragePath>"],
"PortBindings": {
"8883/tcp": [{"HostPort":"8883"}],
"443/tcp": [{"HostPort":"443"}],
"5671/tcp": [{"HostPort":"5671"}]
}
}
}
},
"env": {
"storageFolder": {
"value": "<ModuleStoragePath>"
}
},
"status": "running",
"restartPolicy": "always"
}
Replace HostStoragePath and ModuleStoragePath with the wanted values. Example:
"createOptions": {
"HostConfig": {
"Binds": [
"/etc/iotedge/storage/:/iotedge/storage/"
],
...
}
}
},
"env": {
"storageFolder": {
"value": "/iotedge/storage/"
},
...
Please note that you probably have to manually give the iotEdge user (or all users) access to that folder (using chmod).
Update:
If you are just looking for the available values of IoTHubMessageDispositionResult you will find the answer here:
class IoTHubMessageDispositionResult(Enum):
ACCEPTED = 0
REJECTED = 1
ABANDONED = 2
Update 2:
Messages that have been ACCEPTED are removed from the message queue because they have been successfully delivered.
Messages that are ABANDONED are added to the message queue again and the module will try to send it again as defined in the retryPolicy. For more insight on the retryPolicy you can read this thread.
Messages that are REJECTED are not added to the message queue again.
According to
https://developer.yodlee.com/Aggregation_API/Aggregation_Services_Guide/API_Flow/Refresh_Account
once I've received response with isMessageAvailable = true and errorCode is empty I should call getMFAResponse (actually I'm using getMFAResponseForSite)
this response should contain not null mfaRefreshInfo.fieldInfo. Unfortunately all cases I was able to test fieldInfo were null.
Could you please help me to understand what I'm doing wrong. SiteAccountId is 10038630.
If you are using getMFAResponseForSite then use this refresh flow
Now response of getMFAResponseForSite depends upon how soon yodlee encounters the MFA after login attempt.
There are 3 types of responses you can get -
{ "isMessageAvailable": false, "timeOutTime": 0, "itemId": 0,
"memSiteAccId": 10000502, "retry": true }
On your 1st call to the MFA API you may get this if there is a delay in bringing the MFA data , hence you need to go back and call this API again (as suggested in the flow to loop).In this case if with messageAvailable if false and timeoutTime > 0 then it means that Yodlee has timed out the request and you can abort the refresh and try re-initiating it.
{
"isMessageAvailable": true,
"fieldInfo": {
"responseFieldType": "text",
"minimumLength": -1,
"maximumLength": 6,
"displayString": "Security Key"
},
"timeOutTime": 97640,
"itemId": 0,
"memSiteAccId": 10000052,
"retry": false
}
This is the case where you get the MFA response. Here you can see that isMessageAvailable is true and you have field info populated. This response can come at 1st call of getMFAResponseForSite or after 1 or 2 loops.
{
"isMessageAvailable": true,
"fieldInfo": {
"questionAndAnswerValues": [],
"numOfMandatoryQuestions": -1
},
"timeOutTime": 98480,
"itemId": 0,
"errorCode": 0,
"memSiteAccId": 10039332,
"retry": false
}
In this case you can see that there is an error code present, if it's 0(zero) that means login was successful without MFA and Yodlee is able to navigate to account summary page. In case error code is non zero then please refer error code page
Now in your case the siteAccountID is failing with error code 429 and hence you should be getting that information in getMFAResponseForSite API.
can you make sure you are following the right refresh flow and making the right API calls as ideally the behavior will not change from what is represented in the flow diagram.