Why does DistributedCache SessionHandler throw connection issues? - asp.net-core

I have a .net core app running in VM's in azure where I use Redis as an implementation for DistributedCache. This way we have user sessions stored in Redis and can be shared in the web farm. We only use Redis for storing sessions. We are using Azure Cache for Redis with a normal instance. Both the VM and Redis are in the same region.
Add in startup:
services.AddStackExchangeRedisCache(options => {
options.Configuration = configuration["RedisCache:ConnectionString"];
});
In the web app we are having intermittent problems with redis closing connections. All calls to Redis are managed by calling session Async-methods like below.
public static async Task<T> Get<T>(this ISession session, string key) {
if (!session.IsAvailable)
await session.LoadAsync();
var value = session.GetString(key);
return value == null ? default(T) : JsonConvert.DeserializeObject<T>(value);
}
The errors we are seeing are:
StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: EVAL; An existing connection was forcibly closed by the remote host.; IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=3,Free=32764,Min=512,Max=32767), Local-CPU: n/a
---> StackExchange.Redis.RedisConnectionException: SocketFailure on myredis.redis.cache.windows.net:6380/Interactive, Idle/Faulted, last: EVAL, origin: ReadFromPipe, outstanding: 1, last-read: 34s ago, last-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.0.593.37019
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
And
StackExchange.Redis.RedisConnectionException: SocketFailure on myredis.redis.cache.windows.net:6380/Interactive, Idle/Faulted, last: EXPIRE, origin: ReadFromPipe, outstanding: 1, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.0.593.37019
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
We are not experiencing traffic spikes during the timeouts and the Redis instance is not under any heavy load.
I have no idea how to troubleshoot this further. Any idea?

The connection might be closed by Redis server because of idling for too long.
In your Azure Cache control you can find the config for Redis server, see if you can find timeout setting.
If you can issue commands through command line, you can also issue this command
CONFIG get timeout
If it's zero, it means no timeout.
Then the issue is with your redis client. I'm not familiar with .Net, whatever client you're using to connect to Redis server, check the timeout option or Google search (Name of the client)+ timeout and see if you can find any useful information.

Related

Can not run Redis commands using StackExchangeRedis

Hello i am trying to connect to a Redis database from a ASP NET Core 3.1 application and i keep getting this error when i issue a command.
> 'No connection is active/available to service this operation: SET a; A
> blocking operation was interrupted by a call to
> WSACancelBlockingCall., mc: 1/1/0, mgr: 10 of 10 available,
> clientName: [ClientName], IOCP: (Busy=2,Free=998,Min=8,Max=1000),
> WORKER:
I think it has something to do with the library StackExchangeRedis since until now it worked, up until it stopped working randomly.I have updated to the last version, restarted pc, whatever and nothing.
I can connect to my local redis and issue commands with both the Redis-Cli and using telnet 127.0.0.1 6379 , so that is why i think the culprit is the library.
ConnectionString
localhost:6379,ssl=True,allowAdmin=True,abortConnect=False,defaultDatabase=0
How i use it:
var con=ConnectionMultiplexer.Connect(connectionString); //passes
con.GetDatabase().StringSet("a","a"); //throws
If just using it for localhost development purposes you can try disabling ssl : localhost:6379,**ssl=false**,allowAdmin=True,abortConnect=False,defaultDatabase=0

Unable to connect to redis from Console Application

I have redis server in a separate environment which can be connected via SSH tunnel. I could able to access the Redis server by using the Redis Desktop manager by giving the appropriate server address and password but I am not able to connect it from my Console Application. Below is the code I have used
ConnectionMultiplexer _connection =
ConnectionMultiplexer.Connect("some_address,password=redis,abortConnect=false");
IDatabase _cache = _connection.GetDatabase();
_cache.StringSet("TestKey", "TestValue");
I am getting the below error while trying to insert a string value in the Redis database.
StackExchange.Redis.RedisConnectionException
HResult=0x80131500
Message=No connection is available to service this operation: SET CHeckingTest; UnableToConnect on some_address:6379/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConnectAsync, outstanding: 0, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, global: 0s ago, v: 2.0.601.3402; IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=0,Free=2047,Min=4,Max=2047), Local-CPU: n/a
Source=StackExchange.Redis
StackTrace:
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\ConnectionMultiplexer.cs:line 2237
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisBase.cs:line 54
at StackExchange.Redis.RedisDatabase.StringSet(RedisKey key, RedisValue value, Nullable`1 expiry, When when, CommandFlags flags) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisDatabase.cs:line 2407
at Redis_sample.Program.Main(String[] args) in C:\Users\Anish George\Desktop\Redis\CacheComponent\Redis_sample\Program.cs:line 66
Inner Exception 1:
RedisConnectionException: UnableToConnect on some_address:6379/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConnectAsync, outstanding: 0, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, global: 0s ago, v: 2.0.601.3402
Which password I should be giving? I have one password for SSH Tunnel and another password for Redis server also the port number is different. How can I connect ot Redis via the console application?
You're using port 6379 which is usually used for unsecured traffic, it might be blocked on your server. Try using port 6380 instead.

RabbitMQ Ack Timeout

I'm using RPC Pattern for processing my objects with RabbitMQ.
You suspect,I have an object, and I want to have that process finishes and After that send ack to RPC Client.
Ack as default has a timeout about 3 Minutes.
My process Take long time.
How can I change this timeout for ack of each objects or what I must be do for handling these like processess?
Modern versions of RabbitMQ have a delivery acknowledgement timeout:
In modern RabbitMQ versions, a timeout is enforced on consumer delivery acknowledgement. This helps detect buggy (stuck) consumers that never acknowledge deliveries. Such consumers can affect node's on disk data compaction and potentially drive nodes out of disk space.
If a consumer does not ack its delivery for more than the timeout value (30 minutes by default), its channel will be closed with a PRECONDITION_FAILED channel exception. The error will be logged by the node that the consumer was connected to.
Error message will be:
Channel error on connection <####> :
operation none caused a channel exception precondition_failed: consumer ack timed out on channel 1
Timeout by default is 30 minutes (1,800,000ms)note 1 and is configured by the consumer_timeout parameter in rabbitmq.conf.
note 1: Timeout was 15 minutes (900,000ms) before RabbitMQ 3.8.17.
if you run rabbitmq in docker, you can describe volume with file rabbitmq.conf, then create this file inside volume and set consumer_timeout
for example:
docker compose
version: "2.4"
services:
rabbitmq:
image: rabbitmq:3.9.13-management-alpine
network_mode: host
container_name: 'you name'
ports:
- 5672:5672
- 15672:15672 ----- if you use gui for rabbit
volumes:
- /etc/rabbitmq/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
And you need create file
rabbitmq.conf
on you server by this way
/etc/rabbitmq/
documentation with params: https://github.com/rabbitmq/rabbitmq-server/blob/v3.8.x/deps/rabbit/docs/rabbitmq.conf.example

HubConnection::Start().Wait() times out if behind web proxy

Without a web proxy, Start().Wait() works fine. Connection trace:
11:31:15.0221694 - null - ChangeState(Disconnected, Connecting)
11:31:17.1749694 - 054a636a-10dc-4d39-a77b-709639ea4e5f - SSE: GET http://<removed>/signalr/connect?transport=serverSentEvents&connectionToken=TFuti92AamDL%2FsFNOE8LF1N6T10bDcosIqdkmHbLxYpPwNtW9szZNHHDkrLPR1mFa0Pu%2FUgqmU6fkA%2Fh6iuOY9tTMfjfwqwa%2F5vpZk%2B9iuESgPD5OFYZelTG%2FZn16USK&connectionData=[{"Name":"myHub"}]
11:31:17.9549694 - 054a636a-10dc-4d39-a77b-709639ea4e5f - ChangeState(Connecting, Connected)
But behind a web proxy, it times out or returns after a long time (4-5 minutes) if TransportConnectTimeout is increased. Connection trace:
05:04:05.4397657 - null - ChangeState(Disconnected, Connecting)
05:04:06.1727657 - 7d8ed176-4ca7-461b-97bb-d32b2e71d950 - SSE: GET http://<removed>/signalr/connect?transport=serverSentEvents&connectionToken=Q0FllYmOPNl0%2BQqI643N%2Bzed2zuNAEvLywMLnqkPV4H6%2BPMaiwlrEYGJsNBvrG8QMWdnEJh%2B0qf5UBDj1rpp9JNktaISXa4vhwpK6KnUo32R6d4vBEgunh9Ju%2FRZTm%2Bu&connectionData=[{"Name":"myHub"}]
05:04:11.1737657 - 7d8ed176-4ca7-461b-97bb-d32b2e71d950 - Auto: Failed to connect to using transport serverSentEvents. System.TimeoutException: Transport timed out trying to connect
05:04:11.1837657 - 7d8ed176-4ca7-461b-97bb-d32b2e71d950 - LP Connect: http://<removed>/signalr/connect
05:04:11.8147657 - 7d8ed176-4ca7-461b-97bb-d32b2e71d950 - ChangeState(Connecting, Connected)
05:04:11.8217657 - 7d8ed176-4ca7-461b-97bb-d32b2e71d950 - LP Poll: http://<removed>/signalr/poll
So if behind the web proxy, SignalR fails to connect with SSE protocol and falls back to long polling and connects in about 5 seconds, but still Start().Wait() does not return.
So, how to get it working behind a web proxy? I am using SignalR version 2.0.1.
Here's a workaround: Instead of waiting on Start(), handle the HubConnection.StateChanged event. This event is fired on time.

RavenDB Replication Issue - database cannot be found

Have followed the documentation, but am unable to make replication work for RavenDB over the WAN.
Scenario:
Using Raven build #2261
Master DB: has a local name of "it23"
Slave DB: has a remote name of "http://184.169.xxx.xxx" (xxx's are for
privacy)
On both servers I have created a database called "TonyTest".
On the Master db, I have set up replication using the following document:
{
"Destinations": [
{
"Url": "http://184.169.xxx.xxx:8080",
"Username": null,
"Password": null,
"Domain": null,
"ApiKey": null,
"Database": "TonyTest",
"TransitiveReplicationBehavior": "None",
"IgnoredClient": false,
"Disabled": false,
"ClientVisibleUrl": null
}
]
}
When browsing to the remote server using the same URL of: http://184.169.xxx.xxx:8080, the RavenDB studio launches correctly, and I can see the TestTony database. This seems to confirm that the URL is formatted correctly.
However, the master database immediately generates a document showing failures:
{
"Destination": "http://184.169.xxx.xxx:8080/databases/TonyTest",
"FailureCount": 142
}
When we look at the logs for the REMOTE db, we see that there IS communication with the master, but the replication doesn't complete.
Debug 3/9/2013 12:19:44 AM Document with key 'Raven/Replication/Sources/http://it23:8080/databases/TonyTest' was not found Raven.Storage.Esent.StorageActions.DocumentStorageActions
It looks like the remote server is saying that the db "TonyTest' can't be found, but it IS created.
Can anyone spot my mistake?
Per Ayende's request, here are some log samples from LOCAL server after attempting to setup replication (again I replaced IPs with xxx for privacy). We do not see any errors in the LOCAL db's log. And we do see errors popup in the REMOTE db log. This seems to imply that the LOCAL db is connecting to the REMOTE db, but the replication does not happen. Here are the LOCAL logs:
Debug 3/11/2013 3:17:00 PM No work was found, workerWorkCounter: 17626, for: ReducingExecuter, will wait for additional work Raven.Database.Indexing.WorkContext
Debug 3/11/2013 3:17:00 PM Going to index 1 documents in IndexName: Raven/DocumentsByEntityName, LastIndexedEtag: 00000001-0000-0100-0000-000000002265: (Raven/Replication/Destinations/184.169.xxx.xxx8080databasesTonyTest) Raven.Database.Indexing.AbstractIndexingExecuter
Debug 3/11/2013 3:17:00 PM Document with key 'Raven/Studio/PriorityColumns' was not found Raven.Storage.Esent.StorageActions.DocumentStorageActions
Debug 3/11/2013 3:16:56 PM Going to index 1 documents in IndexName: Raven/DocumentsByEntityName, LastIndexedEtag: 00000001-0000-0100-0000-000000002256: (Raven/Replication/Destinations/184.169.xxx.xxx8080databasesTonyTest) Raven.Database.Indexing.AbstractIndexingExecuter
Update 3/11 8:24p Pacific time
I am now seeing the following errors in the MASTER/Local raven logs:
Failed to close response
System.AggregateException: One or more errors occurred. ---> System.Net.HttpListenerException: An operation was attempted on a nonexistent network connection
at System.Net.HttpResponseStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at Raven.Database.Util.Streams.BufferPoolStream.Dispose(Boolean disposing) in c:\Builds\RavenDB-Stable\Raven.Database\Util\Streams\BufferPoolStream.cs:line 144
at System.IO.Stream.Close()
at Raven.Database.Impl.ExceptionAggregator.Execute(Action action) in c:\Builds\RavenDB-Stable\Raven.Database\Impl\ExceptionAggregator.cs:line 23
--- End of inner exception stack trace ---
at Raven.Database.Impl.ExceptionAggregator.ThrowIfNeeded() in c:\Builds\RavenDB-Stable\Raven.Database\Impl\ExceptionAggregator.cs:line 38
at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.Close() in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerResponseAdapter.cs:line 94
at Raven.Database.Server.Abstractions.HttpListenerContextAdpater.FinalizeResponse() in c:\Builds\RavenDB-Stable\Raven.Database\Server\Abstractions\HttpListenerContextAdpater.cs:line 92
---> (Inner Exception #0) System.Net.HttpListenerException (0x80004005): An operation was attempted on a nonexistent network connection
at System.Net.HttpResponseStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at Raven.Database.Util.Streams.BufferPoolStream.Dispose(Boolean disposing) in c:\Builds\RavenDB-Stable\Raven.Database\Util\Streams\BufferPoolStream.cs:line 144
at System.IO.Stream.Close()
at Raven.Database.Impl.ExceptionAggregator.Execute(Action action) in c:\Builds\RavenDB-Stable\Raven.Database\Impl\ExceptionAggregator.cs:line 23<---
Solved this. Although I don't understand the full reason why.
On the SLAVE server, you must set the raven.server.exe config file to have the following key:
<add key="Raven/AnonymousAccess" value="All"/>
The default was
<add key="Raven/AnonymousAccess" value="Get"/>.
The default worked fine when the master and slave were on the same machine. But when the master and slave were on separate machines (either on the LAN, or across the WAN) replication failed.
I could never find a log entry on the master that pointed toward his problem. The only log entry I could see was on the slave which said that Raven/Replication/Sources/ was not found. I realized that the master was connecting to the slave, but the slave was unable to create the "Raven/Replication/Sources/" document remotely.