Does REDIS has built-in mechanism that will use slave when master is down?
Can I use virtual IP to direct to master and when Master is down is it possible to direct to slave?
As per the documentaion:
elect the slave to master using the SLAVEOF NO ONE command, and shut down your master.
But how the application will know about the changed IP?
mysql has a third party utility that is called MMM (master master replication with monitor). Is there such an utility for REDIS?
You can use a virtual IP in a load balancer, though this is not built in to Redis. Any quality hardware or software load balancer should be able to do it. For example you can use "balance" or HAProxy to front the VIP and use a script or rules that checks the status of Redis instances to see which is master and sets that as the destination in the load balancer (LB).
Going this route would require one or more additional servers (or VMs depending on your setup) but it would provide you with a configuration that has clients talking to a single IP and being clueless about which server they need to talk to on the back end. How you update the LB with which server to talk to is dependent on what LB you use. Fortunately, none of them would need to know or handle the Redis protocol; they are just balancing a port.
When I go this route I go with a Slave-VIP and a Master-VIP. The slave-VIP load balances across all Redis instances, whereas the Master-VIP only has the current master enabled. If your write load is very heavy you can leave the current master out of the Slave-VIP pool. Otherwise I would leave it in; that eliminates the need for failover updating of the Slave-VIP pool.
Related
I am trying to implement a 3 master 3 slave architecture with redis cluster. I want to redirect my client to slave if master is blocked (like undergoing a MULTI EXEC query) or redirect to master if slave is synchronising the MULTI EXEC query. Is there any way I can achieve it through redis configuration, or do I need to manually implement this logic with the client library (redis-rb) I am using?
Thanks in advance.
As I know, there isn't any proxy or balancing in redis cluster that you can control. In Redis Cluster nodes don't proxy commands to the right node in charge for a given key, but instead, they redirect clients to the right nodes serving a given portion of the keyspace. So you can't somehow control this from config.
Maybe your case with MULTI EXEC will be handled by the client library because it knows all about redis master nodes config
I have a Redis Cluster that clients are connecting to via HAPRoxy with a Virtual IP. The Redis cluster has three nodes (with each node sharing the same server with a running sentinel instance).
My question is, when i clients gets a "MOVED" error/message from a cluster node upon sending a request, does it bypass the HAProxy the second time when it connects since it has been provided with an IP:port when the MOVEd message was issued? If not, how does the HAProxy know the second time to send it to the correct node?
I just need to understand how this works under the hood.
If you want to use HAProxy in front of Redis Cluster nodes, you will need to either:
Set up an HAProxy for each master/slave pair, and wire up something to update HAProxy when a failure happens, as well as probably intercept the topology related commands to insert the virtual IPs rather than the IPs the nodes themselves have and report via the topology commands/responses.
Customize HAProxy to teach it how to be the cluster-aware Redis client so the actual client doesn't know about cluster at all. This means teaching it the Redis protocol, storing the cluster's topology information, and selecting the node to query based on the key(s) being accessed by the consumer code.
With Redis Cluster the client must be able to access every node in the cluster. Of the two options above Option 2 is the "easier" one, but at this point I wouldn't recommend either.
Conceivably you could use the VIP as a "first place to get the topology info" IP but I suspect you'd have serious issues develop as that original IP would not be one of the ones properly being reported as a nod handling data. For that you could simply use round-robin DNS and avoid that problem, or use the built-in "here is a list of cluster IPs (or names?)" to the initial connection configuration.
Your simplest, and least likely to be problematic, route is to go "full native" and simply give full and direct access to every node in the cluster to your clients and not use HAProxy at all.
I'm testing out ElastiCache backed by Redis with the following specs:
Using Redis 2.8, with Multi-AZ
Single replication group
1 master node in us-east-1b, 1 slave node in us-east-1c, 1 slave node in us-east-1d
The part of the application writing is directly using the endpoint for the master node (primary-node.use1.cache.amazonaws.com)
The part of the application doing only reads is pointing to a custom endpoint (readonly.redis.mydomain.com) configured in HAProxy, which then points to the two other read slave end points. (readslave1.use1.cache.amazonaws.com and readslave2.use1.cache.amazonaws.com)
Now lets say the primary node (master) fails in us-east-1b.
From what I understand, if the master instance fails, I won't have to change the url for the end point for writing to Redis (primary-node.use1.cache.amazonaws.com), although from there, I still have the following questions:
Do I have to change the endpoint names for the read only slaves?
How long until the missing slave is added into the pool?
If there's anything else I'm missing, I'd appreciate the advice/information.
Thanks!
If you are using ElastiCache, you should make use the "Primary EndpointThe" provided by AWS.
That endpoint actually is backed by Route53, if the primary (master) redis is down, since you enable MutliA-Z, it will auto fail over to one of the read replica (slave).
In that case, you don't need to modify the endpoint of your redis.
I don't know why you have such design, seems you only want write to master, but always read from slave.
For HA Proxy part, you should include TCP check for ALL 3 redis nodes, using their "Read Endpoint"
In haproxy, you can check if the endpoint is SLAVE, if yes, your haproxy should redirect the traffic to that.
Notice that in the application layer, if your redis driver don't support auto reconnect, your script will fail to connect to the new master nodes.
In addition to "auto reconnect", since AWS is using Route53 DNS to do fail over, some lib will NOT do NS lookup again, which means the DNS is still pointing to the OLD ip which is the old master.
Using HAproxy can solve this problem.
I have installed Redis for my nodeJS application and configured it to be a slave for another instance of Redis DB running on a different server. Can I have the same instance (different DB) of Redis (running as slave) act as Master for locally installed application?
Thanks in advance
Yes, you can, but with a big caveat.
Any slave instance can be master of one or several other instances. So you can imagine daisy chaining slaves and build a hierarchical replication system.
Now, my understanding is you don't need your slave to feed another Redis instance, but just allow an application to perform read/write operations in another database of the slave instance.
To allow it, you need to set the value of the slave-read-only parameter to "no" in the slave configuration:
# You can configure a slave instance to accept writes or not. Writing against
# a slave instance may be useful to store some ephemeral data (because data
# written on a slave will be easily deleted after resync with the master) but
# may also cause problems if clients are writing to it because of a
# misconfiguration.
#
# Since Redis 2.6 by default slaves are read-only.
#
# Note: read only slaves are not designed to be exposed to untrusted clients
# on the internet. It's just a protection layer against misuse of the instance.
# Still a read only slave exports by default all the administrative commands
# such as CONFIG, DEBUG, and so forth. To a limited extend you can improve
# security of read only slaves using 'rename-command' to shadow all the
# administrative / dangerous commands.
slave-read-only no
Now, all the write operations you will run on this instance will be ephemeral. If the link is lost between the master and the slave, the slave will completely synchronize again to the master. All the data you have set on the slave will be lost. If you stop and restart the slave, you will also loose all the ephemeral data.
It may or may not suit your needs.
There is no way to parameter the synchronization (or the persistence options) at the database level. You cannot tell Redis to synchronize a given database, and not another one. Configuration always applies at the instance level.
I have a Glassfish v2u2 cluster with two instances and I want to to fail-over between them. Every document that I read on this subject says that I should use a load balancer in front of Glassfish, like Apache httpd. In this scenario failover works, but I again have a single point of failure.
Is Glassfish able to do that fail-over without a load balancer in front?
The we solved this is that we have two IP addresses which both respond to the URL. The DNS provider (DNS Made Easy) will round robin between the two. Setting the timeout low will ensure that if one server fails the other will answer. When one server stops responding, DNS Made Easy will only send the other host as the server to respond to this URL. You will have to trust the DNS provider, but you can buy service with extremely high availability of the DNS lookup
As for high availability, you can have cluster setup which allows for session replication so that the user won't loose more than potentially one request which fails.
Hmm.. JBoss can do failover without a load balancer according to the docs (http://docs.jboss.org/jbossas/jboss4guide/r4/html/cluster.chapt.html) Chapter 16.1.2.1. Client-side interceptor.
As far as I know glassfish the cluster provides in-memory session replication between nodes. If I use Suns Glassfish Enterprise Application Server I can use HADB which promisses 99.999% of availability.
No, you can't do it at the application level.
Your options are:
Round-robin DNS - expose both your servers to the internet and let the client do the load-balancing - this is quite attractive as it will definitely enable fail-over.
Use a different layer 3 load balancing system - such as "Windows network load balancing" , "Linux Network Load balancing" or the one I wrote called "Fluffy Linux cluster"
Use a separate load-balancer that has a failover hot spare
In any of these cases you still need to ensure that your database and session data etc, are available and in sync between the members of your cluster, which in practice is much harder.