I'm using ioredis to set values to a redis enterprise instance.
How do I close the connection after a few seconds to reduce the number of clients created ?
Do I need to close the connection at all ?
Does every invocation spawn a redis client ?
Do I need a VPC connector ?
How do I ensure the fucntion runs on the same ip:port to prevent creation of other instances
import * as functions from "firebase-functions";
import Redis = require("ioredis");
const redis = new Redis("redis://default:jMSNpTm3JXs1#redis-15360.c228.us-central1-1.gce.cloud.redislabs.com:15360");
export const SaveRecord = functions.https.onCall(async (data,context)=>{
const {record,recordID} = data;
try{
var result = await redis.multi().set(recordID, record).expire(recordID,3600).exec();
return result;
}catch(e){
console.log(e);
return "error";
}
})
I never deployed Redis Cloud Enterprise on my project but I can already provide these answers
1,2,3. You don't need to close the connection at every connexion. In your code, you start (line 4) a connexion to your Redis and you store it in global variable (in the global scope, not in function body).
That means, when to Cloud Functions instance start, it call all the global code (your first 4 lines, and then call your functions. the redis variable is kept between Cloud Functions invocation.
However, you need to check the redis Cloud default connexion timeout and be able to perform a reconnect in case of closed connexion (exception handling in your code).
About connexion to close, it also depends on the Redis Cloud configuration: is the idle connexion are automatically closed? If not, it will be difficult. Try to close the idle connexion by Redis server configuration after 1 hour of idle activity or less. (Cloud Functions instances are usually offloaded after 30 minutes of inactivity, without any Google Cloud commitment).
Yes; If you check the IP registered on your URL (for example nelookup redis-15360.c228.us-central1-1.gce.cloud.redislabs.com), you will find a URL in 172.26.x.x. It's a RFC 1918 private IP. The DNS resolution is external (ask redislab URL the IP of the server), but the targeted server is internal to your project. You can route only internal traffic.
The Cloud Functions service create and destroy instances according to the HTTP traffics. You can't prevent the creation or the destruction of instances, it's serverless. Similarly, you doesn't have an IP/PORT associated to your Cloud Functions instance, it's serverless. If you want to have only 1 Cloud Functions instance, you can set the max instance to 1. But, Be careful, Cloud functions can handle only 1 request at a time on a same instance. With a max instance to 1, is you have 2 concurrent requests, 1 will be processed, the other will wait, and after 10 seconds a 417 HTTP error will be returned.
Related
Currently I am using Aerospike in my application.
I faced lots of timeout issues as shown below when I was creating new java client for each transaction and I was not closing it so number of connection ramp up dramatically.
Aerospike Error: (9) Client timeout: timeout=1000 iterations=1 failedNodes=0 failedConns=0
so to resolve this timeout issue,I didn't made any changes to client, read and write policy, I just created only one client, stored it's instance in some variable and used this same client for all transaction (get or put requests).
now I want to understand how moving from multiple client to one client resolved my timeout issue.
how these connection were not closing automatically.
The AerospikeClient constructor requests peers, partition maps and racks for all nodes in the cluster and initializes connection pools and async eventloops. This is an expensive process that is only meant to be performed once per cluster at application startup. AerospikeClient is thread-safe, so instances can be shared between threads.
If AerospikeClient close() is not called, connections residing in the pools (at least one connection pool per node) will not be closed. There are no finalize() methods in AerospikeClient.
The first transaction(s) usually need to create new connections. This adds to the latency and can cause timeouts.
The client does more than just the application's transactions. It also monitors the cluster for changes so that it can maintain one hop per transaction. Also, I believe when we initialize the client, we create an initial pool of sockets.
It is expected that most apps would only need one global client.
I'm using "redis-rs" for rust and I tested sending a few thousand requests locally
and it works really well at the few first, except that at some point it stops accepting requests
and starts showing me this error when I send a command to redis:
"Only one usage of each socket address (protocol/network address/port) is normally permitted"
I am opening a client and a connection on every request to the http server that takes them in,
that's probably a bad idea in the first place, but shouldn't the connections stop existing and close after the function that opened it ends?
Is there any better solution, like some kind of global connection?
thanks
Well if it is an http server, the crate you are using likely is doing multithreading to handle requests. It is possible that one thread got caught in the process of closing the connection just as another thread began processing the next request.
Or in your case, maybe the remote database has not finished closing the previous request by the time the next connection is created. Either way, it's easier to think of it as a race condition between threads.
Since you don't know what thread will request a connection next, it may be better to store the connection as a global resource. Since I assume a mutex lock is faster than opening and closing a socket, I used lazy_static to create a single thread safe connection.
use lazy_static::lazy_static;
use parking_lot::Mutex;
use std::sync::Arc;
lazy_static! {
pub static ref LOCAL_DB: Arc<Mutex<Connection>> = {
let connection = Connection::open("local.sqlite").expect("Unable to open local DB");
connection.execute_batch(CREATE_TABLE).unwrap();
Arc::new(Mutex::new(connection))
};
}
// I can then just use it anywhere in functions without any complications.
let lock = LOCAL_DB.lock();
lock.execute_batch("begin").unwrap();
// etc.
Each morning at 4am a Scheduled Task creates around 500 messages on a message queue. Each message is an id of an email to send. Each message is picked up and a url created and sent via await HttpClient.GetAsync(url) The url target then creates and sends the email. This has worked well for months.
I've just upgraded to SQL Azure v12 and all is now not well.
The very first messages to be processed (after 2 minutes running time) threw a
"System.Threading.Tasks.TaskCanceledException"
I'm also seeing
"System.Data.Entity.Core.EntityException: The underlying provider
failed on Open. ---> System.InvalidOperationException: Timeout
expired. The timeout period elapsed prior to obtaining a connection
from the pool. This may have occurred because all pooled connections
were in use and max pool size was reached."
and a couple of
"The timeout period elapsed prior to completion of the operation or
the server is not responding. This failure occurred while attempting
to connect to the routing destination. The duration spent while
attempting to connect to the original server was - [Pre-Login]
initialization=6; handshake=426; [Login] initialization=0;
authentication=0;"
The webjob that sends the request to the api is awaiting a response. I'm wondering if because it's async, while awaiting the response the thread is freed to go off and processes another queue item - and therefore creates another api request, essentially this hits the api again and again until there are so many requests being processed by the api all the theads are in use - and that I might be better off NOT making the webjob async because then it (the 'trapped' thread) would send a request only after the first request completes? Is that right? edit: actually the IIS logs suggest that the api requests are not all happening at once. So my question is "what should I look at next? Are these common SQL v12 errors or is the recent upgrade a red-herring?"
just to clarify, the webjob that fires in response to the queue message simply does:
using (HttpClient client = new HttpClient())
{
response = await client.GetAsync(url);
}
and hits the web api of an Always On standard tier azure website. Database DTU % is about 25% while this happens.
edit:
"Gateway no longer provides retry logic in V12 Before version V12,
Azure SQL Database had a gateway that acted as a proxy to buffer all
interactions between the database and your client program. The gateway
provided automated retry logic for some transient errors.
V12 eliminated the gateway. Now your program must more fully handle
transient errors."
adding DbConfiguration class for SqlAzureExecutionStrategy. Will so how it runs tonight.
Adding the EF retry config class fixed the transient errors. The cancelled tasks are a different issue (new question)
//https://msdn.microsoft.com/en-us/data/jj680699
public class SqlAzureConfiguration : DbConfiguration
{
public SqlAzureConfiguration()
{
this.SetExecutionStrategy("System.Data.SqlClient", () => new SqlAzureExecutionStrategy());
}
}
and in web.config (because I have multiple contexts)
<entityFramework codeConfigurationType="Abc.DataService.SqlAzureConfiguration, Abc.DataService">
We are implementing a pattern where our client checks to see if a document exists in Redis, and if it does not, we then fetch the data from the database.
We are trying to handle a case where the Redis server is down or unreachable so we can then immediately fetch from the database.
However, when we test our code by intentionally taking down the Redis server, the call to Redis via the ServiceStack client does not timeout for approximately 20 seconds.
We tried using the RedisClient .SendTimeout property to various values (1000, 100, 1), but the timeout always happens after approx 20 seconds. We also tried using the .Ping() method but have the same problem.
Question: how can we handle the scenario where the Redis server is down and we want to switch to a DB fetch more quickly?
I had a similar problem sending e-mail: sometimes there's no answer and the build-in timeout (of SmtpClient) does nothing. Eventually I'd get a timeout which I believe comes from the underlying TCP/IP layer. I'd set the timeout in the client a little shorter than the "brutal timeout" on Task.Wait.
My solution was to wrap the call in a Task, and use a timeout on that:
// this special construct is to set a timeout (the SmtpClient timeout does not seem to work)
var task = Task.Factory.StartNew(() => SendEmail(request));
if (!task.Wait(6000))
Log.Error("Could not send mail to {0}. Timeout (probably on TCP layer).".Fmt(request.To));
Maybe something similar would work for you, just replace the SendEmail with a method that does the Redis thing.
You should not rely on the redis server to tell you how long the request should wait before flipping to plan B. Put this logic in the code actioning the request so that it is independent of how the redis server is set up
I have a requirement where I need to send continuous updates to my clients. Client is browser in this case. We have some data which updates every sec, so once client connects to our server, we maintain a persistent connection and keep pushing data to the client.
I am looking for suggestions of this implementation at the server end. Basically what I need is this:
1. client connects to server. I maintain the socket and metadata about the socket. metadata contains what updates need to be send to this client
2. server process now waits for new client connections
3. One other process will have the list of all the sockets opened and will go through each of them and send the updates if required.
Can we do something like this in Apache module:
1. Apache process gets the new connection. It maintains the state for the connection. It keeps the state in some global memory and returns back to root process to signify that it is done so that it can accept the new connection
2. the Apache process though has returned the status to root process but it is also executing in parallel where it going through its global store and sending updates to the client, if any.
So can a Apache process do these things:
1. Have more than one connection associated with it
2. Asynchronously waiting for new connection and at the same time processing the previous connections?
This is a complicated and ineffecient model of updating. Your server will try to update clients that have closed down. And the server has to maintain all that client data and meta data (last update time, etc).
Usually, for continuous updates ajax is used in a polling model. The client has a javascript timer that when it fires, hits a service that provides updated data. The client continues to get updates at regular intervals without having to write an apache module.
Would this model work for your scenario?
More reasons to opt for poll instead of push
Periodic_Refresh
With a little patch to resume a SUSPENDED mpm_event connection, I've got an asynchronous Apache module working. With this you can do the improved polling:
javascript connects to Apache and asks for an update;
if there's no updates, then instead of answering immediately the module uses SUSPENDED;
some time later, after an update or a timeout happens, callback fires somewhere;
callback gives an update (or a "no updates" message) to the client and resumes the connection;
client goes to step 1, repeating the poll which with Keep-Alive will use the same connection.
That way the number of roundtrips between the client and the server can be decreased and the client receives the update immediately. (This is known as Comet's Reverse Ajax, AFAIK).