I use Google Cloud Run with Google Memorystore Redis.
My application is written on Node.js (Express) and connected to the Redis Instance in this way:
const asyncRedis = require("async-redis");
const redisClient = asyncRedis.createClient(process.env.REDISPORT, String(process.env.REDISHOST));
redisClient.on('error', (err) => console.error('ERR:REDIS:', err));
.....
const value = await redisClient.get("Code");
Every half an hour the application loses its connection to the Redis and I receive the error:
AbortError: Redis connection lost and command aborted. It might have been processed.
at RedisClient.flush_and_error (/usr/src/app/node_modules/redis/index.js:362)
at RedisClient.connection_gone (/usr/src/app/node_modules/redis/index.js:664)
at RedisClient.on_error (/usr/src/app/node_modules/redis/index.js:410)
at Socket.<anonymous> (/usr/src/app/node_modules/redis/index.js:279)
at Socket.emit (events.js:315)
at emitErrorNT (internal/streams/destroy.js:92)
at emitErrorAndCloseNT (internal/streams/destroy.js:60)
at processTicksAndRejections (internal/process/task_queues.js:84)
In two or three minutes the connection returns and the application works correctly during half an hour until next disconnect.
Any idea?
According to the official GCP documentation there can be several scenarios that could cause the connectivity issues with your Redis instance. However, as the connectivity issues you are experiencing are intermittent and normally you connect to the instance successfully, none of the mentioned scenarios seems to be applicable to your situation.
There is a similar issue opened on GitHub, where some users where experiencing it when reaching the maximum number of clients for their Redis instance, however, I would recommend you to contact the GCP technical support, so that they could review your particular configuration and inspect your instance with the internal tools.
Related
I am trying to achieve the multicloud architecture. My network has 2 peers, 1 orderer and a webclient. This network is in Azure. I am trying to add a peer from Google Cloud Platform to the channel of Azure. For this, I created a crypto-config for 3rd peer from Azure webclient. But in the crypto-config, I made the changes like peers in Azure have their own certificates while for the 3rd peer, I placed the newly created certificates. Now I can install, instantiate, invoke and do queries in the peers(1 and 2). And I can install the chaincodes in 3rd peer. But I am unable to instantiate the chaincodes.
Getting the following error: Error: could not assemble transaction, err proposal response was not successful, error code 500, msg error starting container: error starting container: Post http://unix.sock/containers/create?name=dev-(CORE_PEER_ID)-documentCC-1: dial unix /var/run/docker.sock: connect: permission denied
Can anyone guide me on this.
Note: All the peers, orderer, webclient are running in different vm(s)
#soundarya
It doesn’t matter how many places your solution is deployed
The problem is you are running docker by using sudo command try to add docker to sudo group
Below block will help you out
https://www.digitalocean.com/community/questions/how-to-fix-docker-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socket
To learn more concept about docker.sock
You can refer to my answer in another Can anyone explain docker.sock
I am currently running some Google Cloud functions (in typescript) that require a connection to a Redis instance in order to LPUSH into the queue (on other instances, I am using Redis as a queue worker).
Everything is fine, except I am getting a huge number of ECONNECTRESET and ECONNECTIMEOUT related errors despite everything working properly.
The following code can execute successfully on the cloud function but still, I am seeing constant errors thrown related to the connection to the Redis.
I think it is somehow related to how I am importing my client- ioredis. I have utils/index.ts, utils/redis.js and inside the redis.js I have:
const Redis = require('ioredis');
module.exports = new Redis(6380, 'MYCACHE.redis.cache.windows.net', { tls: true, password: 'PASS' });
Then I am importing this in my utils/index.ts like so: code missing
And exporting some aysnc function like: code missing
When executing in the GCF environment, I get the # of expected results in results.length and I see (by monitoring the Redis internally) this list was pushed as expected to the queue.
Nevertheless, these errors continue to appear incessantly.
ioredis] Unhandled error event: Error: read ECONNRESET at _errnoException (util.js:1022:11) at TLSWrap.onread (net.js:628:25)
I periodically I get these exceptions:
RedisResponseException
Unexpected reply: +OK, sPort: 60957, LastCommand:
It seems to happen when lots of activity occurs simultaneously. Using even the latest Amazon ElastiCache server, as well as local Mac & Ubuntu flavors.
Other errors occur to but this is the most common. Is there some gotcha with Redis in terms of config settings etc?
Are you using PooledRedisClientManager or BasicRedisClientManager?
I got a lot of Unexpected reply whith BasicRedisClientManager
I had the same error (Unexpected reply) on a multi-threaded Azure Worker Role.
Solution:
The ServiceStack.Redis.RedisClient is not Thread safe.
On multithreaded applications you need to create a new insance of the RedisClient for each thread!
var myRedisClient = new RedisClient(accessKey);
...or use the client manager as fvoncina said. But I didn´t use it so far.
I'm using the ServiceStack Redis Client and I was hoping that I could get a clarification on what might cause the following error ... "Unable to Connect: sPort: 50071"? I'm using the "PooledRedisClientManager" object for connections. Thanks for any assistance.
IF YOU ARE USING A SELF HOSTED REDIS SERVER AND USING THE Service Stack Redis Client THEN BUYER BEWARE
As of 9/23/2015
Service Stack does license validation in the client code (rather than the server). If you are ripping through a lot of messages 6000+ an hour you will get. The resulting error is
Unable to Connect: sPort:
However, it is not handling their custom LicenseExpection and exposing the error correctly. The error would be something like this:
The free-quota limit on '6000 Redis requests per hour' has been reached. Please see https://servicestack.net to upgrade to a commercial license or visit https://github.com/ServiceStackV3/ServiceStackV3 to revert back to the free ServiceStack v3.
I doubt you have imposed such a limit on your server :-)
This could be a time out issue, try increasing it:
pooledRedisClientManager.ConnectTimeout = 1000
You need to check that you are not creating a new PooledRedisClientManager for each request / usage. You will quickly run out of ports. Use a singleton approach in a web environment.
I'm using a large app instance to run a basic java web application (GWT + Spring). There's an expensive operation within my application (report) which takes a long time to execute.
I've tried running it with the cloudbees SDK on my local machine with similar settings as it would be on the cloud and it seems to function just fine. It runs in about 3-4 minutes.
On the cloud, it seems to be taking longer. The problem isn't the fact that it takes long. What happens in that cloudbees terminates the session after 5 minutes and gives me an error in my browser saying 'Unable to connect to server. Please contact your administrator'. A report which doesn't take as long runs just fine. My application has a session timeout of 30 minutes, so that isn't a problem either.
What could possibly be going wrong? Is it something to do with cloudbees?
This may be due to proxy buffering of your request through the routing layer (revproxy) - so it most likely isn't a session timeout - but the http connection getting cut.
You can either set proxyBuffering=false via the bees CLI command (eg when you deploy the app) - this will ensure longer running connections can work.
Ideally, however, you could change the app slightly to return to the browser with some token which you can poll with to get completion status, as even with a connection that lasts that long, over the internet it may provide a bad experience vs locally.