I have a Redis Cluster that clients are connecting to via HAPRoxy with a Virtual IP. The Redis cluster has three nodes (with each node sharing the same server with a running sentinel instance).
My question is, when i clients gets a "MOVED" error/message from a cluster node upon sending a request, does it bypass the HAProxy the second time when it connects since it has been provided with an IP:port when the MOVEd message was issued? If not, how does the HAProxy know the second time to send it to the correct node?
I just need to understand how this works under the hood.
If you want to use HAProxy in front of Redis Cluster nodes, you will need to either:
Set up an HAProxy for each master/slave pair, and wire up something to update HAProxy when a failure happens, as well as probably intercept the topology related commands to insert the virtual IPs rather than the IPs the nodes themselves have and report via the topology commands/responses.
Customize HAProxy to teach it how to be the cluster-aware Redis client so the actual client doesn't know about cluster at all. This means teaching it the Redis protocol, storing the cluster's topology information, and selecting the node to query based on the key(s) being accessed by the consumer code.
With Redis Cluster the client must be able to access every node in the cluster. Of the two options above Option 2 is the "easier" one, but at this point I wouldn't recommend either.
Conceivably you could use the VIP as a "first place to get the topology info" IP but I suspect you'd have serious issues develop as that original IP would not be one of the ones properly being reported as a nod handling data. For that you could simply use round-robin DNS and avoid that problem, or use the built-in "here is a list of cluster IPs (or names?)" to the initial connection configuration.
Your simplest, and least likely to be problematic, route is to go "full native" and simply give full and direct access to every node in the cluster to your clients and not use HAProxy at all.
Related
In this post, if I read it correctly, it was suggested that replication with ActiveMQ Artemis could be achieved with only two nodes as follows:
However, it's still possible for the virtual IP address to direct traffic to one of the two VMs based on the broker's availability since the backup broker will not be able to receive connections until the primary broker fails at which point the backup broker will become active and begin accepting connections.
I'm assuming from the answer that this be accomplished simply using the classic HA policy to configure one as the master, the other as the slave and configuring them to be a part of the same cluster, as per the documentation.
Is this a correct assumption?
The point from what you quoted from my answer was really just about the use of a virtual IP address in front of a primary/backup pair of brokers. I didn't mean to imply anything about the configuration of the primary/backup pair itself.
In short, even with a virtual IP address in front of the pair of brokers you still need a way to mitigate split brain and the minimum viable way to do that is with a ZooKeeper node. If you use the "classic" configuration approach then there will be no mitigation for split brain.
Hoping someone might be able to give me some guidance on this. I have setup a RabbitMQ cluster on 3 nodes.
- Node1 (Master), Node2 and Node3
The part I can't figure out is I cannot find any information about what happens if Master goes down, what is the client pointing to?
If Node1 goes down, then surely the clients which are configured to point to Node1 will not be able to automatically fail-over to whatever becomes the new master?
My idea was to setup a load balancer which detects which node is master and only ever redirects requests to that node?
Any thoughts or ideas?
Thanks :)
Spring amqp helps in solving this easily. It provides the failover mechanism out of the box. You can just configure the address of the nodes with comma separated and pass it to addresses parameter while creating the connection factory.
For eg:
<rabbit:connection-factory id="rabbitConnectionFactory" addresses="192.168.1.2:5672,192.168.1.3:5672" username="${rabbitmq.username}"
connection-factory="clientConnectionFactory"
cache-mode="${rabbit.cacheMode}"
channel-cache-size="${rabbit.channelCacheSize}"
password="${rabbitmq.password}" virtual-host="${rabbitmq.vhost}"/>
This way, when the first node goes down, the connection will be made to the second node.
If clients are only capable of connecting to one node, and that node goes down, clients will lose connectivity. You need (as you surmise) some kind of setup whereby either:
you place a load-balancer in front of the nodes, and the clients connect only to the load-balancer; OR
the clients can connect to any of the available nodes, trying each in turn.
Whichever solution, there is no need to direct all client traffic to the master node. Clients nodes can connect to either a master or slave broker.
At CloudAMQP we have one single URL to the cluster that clients connect to and we recommend to always configure clients to automatically reconnect on connection loss.
We have fail over and load balancing between nodes, and it's done over DNS, with a low TTL (30s).
https://www.cloudamqp.com/blog/2015-12-29-cloudamqp-plan-setup-pause-minority-mirrored-nodes-and-the-cap-theorem.html
I am setting up a cluster of servers using vagrant and playing with Redis sentinel and HAProxy for Postgresql db connection (with pgpool). I was curious if it make sense to put haproxy and redis sentinel on each of my web server nodes and have them connect directly to those. The thought is that it can create a distributed connection to the DB and redis and reduce the single point of failure to having a single haproxy that they connect to and then split to different db nodes. I can also keep the database connect (via haproxy) and redis (via sentinel) encapsulated to the localhost. Does this make sense?
It only makes sense if you're trying to save up on resources/costs.
Please note that redis sentinel must have a finite list of sentinel instances, which doesn't fit the scenario of placing one per machine, as your maching count would probably scale/change.
Otherwise , it's always makes the most sense to put different infrastructure components ( especially those with clustering/HA nature, such as redis ) on different machines.
By mixing them all together, you usually end up with applications getting in the way of each other and stealing CPU from each-other once the load increases. You also risk designing your applications/scripts/flows to be location aware (i.e assume external resources are always local ) which is also not a really good practice.
I'm testing out ElastiCache backed by Redis with the following specs:
Using Redis 2.8, with Multi-AZ
Single replication group
1 master node in us-east-1b, 1 slave node in us-east-1c, 1 slave node in us-east-1d
The part of the application writing is directly using the endpoint for the master node (primary-node.use1.cache.amazonaws.com)
The part of the application doing only reads is pointing to a custom endpoint (readonly.redis.mydomain.com) configured in HAProxy, which then points to the two other read slave end points. (readslave1.use1.cache.amazonaws.com and readslave2.use1.cache.amazonaws.com)
Now lets say the primary node (master) fails in us-east-1b.
From what I understand, if the master instance fails, I won't have to change the url for the end point for writing to Redis (primary-node.use1.cache.amazonaws.com), although from there, I still have the following questions:
Do I have to change the endpoint names for the read only slaves?
How long until the missing slave is added into the pool?
If there's anything else I'm missing, I'd appreciate the advice/information.
Thanks!
If you are using ElastiCache, you should make use the "Primary EndpointThe" provided by AWS.
That endpoint actually is backed by Route53, if the primary (master) redis is down, since you enable MutliA-Z, it will auto fail over to one of the read replica (slave).
In that case, you don't need to modify the endpoint of your redis.
I don't know why you have such design, seems you only want write to master, but always read from slave.
For HA Proxy part, you should include TCP check for ALL 3 redis nodes, using their "Read Endpoint"
In haproxy, you can check if the endpoint is SLAVE, if yes, your haproxy should redirect the traffic to that.
Notice that in the application layer, if your redis driver don't support auto reconnect, your script will fail to connect to the new master nodes.
In addition to "auto reconnect", since AWS is using Route53 DNS to do fail over, some lib will NOT do NS lookup again, which means the DNS is still pointing to the OLD ip which is the old master.
Using HAproxy can solve this problem.
Does REDIS has built-in mechanism that will use slave when master is down?
Can I use virtual IP to direct to master and when Master is down is it possible to direct to slave?
As per the documentaion:
elect the slave to master using the SLAVEOF NO ONE command, and shut down your master.
But how the application will know about the changed IP?
mysql has a third party utility that is called MMM (master master replication with monitor). Is there such an utility for REDIS?
You can use a virtual IP in a load balancer, though this is not built in to Redis. Any quality hardware or software load balancer should be able to do it. For example you can use "balance" or HAProxy to front the VIP and use a script or rules that checks the status of Redis instances to see which is master and sets that as the destination in the load balancer (LB).
Going this route would require one or more additional servers (or VMs depending on your setup) but it would provide you with a configuration that has clients talking to a single IP and being clueless about which server they need to talk to on the back end. How you update the LB with which server to talk to is dependent on what LB you use. Fortunately, none of them would need to know or handle the Redis protocol; they are just balancing a port.
When I go this route I go with a Slave-VIP and a Master-VIP. The slave-VIP load balances across all Redis instances, whereas the Master-VIP only has the current master enabled. If your write load is very heavy you can leave the current master out of the Slave-VIP pool. Otherwise I would leave it in; that eliminates the need for failover updating of the Slave-VIP pool.