If I have a setup of 3 sentinel nodes with quorum of 2 and one of the Sentinel node fails, during failover process the Quorum can still be reached with 2 sentinels that are up, but what happens if leader election fails?
From what I have read, leader election happens via a voting mechanism, so there is a chance that both the Sentinels will vote for any 1 sentinel and leader election is successful and failover can begin.
But what if each sentinel voted for itself and leader election could not happen? Will the leader election process be triggered again until a leader is selected or the failover process will abort?
Related
I am doing POC on RabbitMQ's Quorum Queues, especially focusing on fail-over mechanism. In my case I have two nodes (for example NodeA and NodeB) and one Quorum queue which resides on NodeA. Now whenever I am publishing a test message to Quorum queue of NodeA, I can see the same message on NodeB.
Now when testing the failover mechanism and stopping NodeA, I am unable to publish any message, also I can not see any messages in quorum queue, I think the NodeB is not promoted to be a new leader. I am supposing the leader would be promoted automatically, do I need to do anything to make the other Node leader ?
Kind Regards
quorum queues do not support two nodes clusters and two-node clusters are highly recommended against for any clusters.
From Quorum Queues documentation guide:
A quorum queue requires a quorum of the declared nodes to be available to function.
When a RabbitMQ node hosting a quorum queue's leader fails or is stopped another node
hosting one of that quorum queue's follower will be elected leader and resume operations.
I have a Redis HA deployment with 3 nodes (1 Master, 2 Slaves) and a sentinel running on every node. On the client side, I use a WAIT 2 0 to block indefinitely until my write reached the 2 slaves (and I am OK with that).
What would be the behavior of the WAIT command upon:
1) a network partition isolates the master and the client from the 2 slaves so my client is currently blocked by the WAIT
2) the majority of sentinels elects one of the slave as the new master (since there is still a quorum)
3) the network partition heals and the old master become slave of the new one
Would the WAIT still be blocking? Or would it release the client returning "0" slaves reached?
Many thanks
I have 3 nodes this disc mode and "ha-mode is all". rabbitmq version 3.6.4
when I try to stop all nodes, first I stop two slave nodes,end stop master nodes. Assume that master node is broken and can't be started. I use rabbitmqctl force_boot setup one slave node, I found queue state is down.
I don't think this is right. I think the slave node setup become master, and queue is available. Do not consider whether the message is lost.
But, first stop master node, then stop new master node, end last node. I can
rabbitmqctl force_boot setup any node. any node is available.
Sounds like you're ending up with unsynchronized slaves and by default RabbitMQ will refuse to fail over to an unsynchronised slave on controlled master shutdown.
Stopping master nodes with only unsynchronised slaves
It's possible that when you shut down a master node that all available slaves are unsynchronised. A common situation in which this can occur is rolling cluster upgrades. By default, RabbitMQ will refuse to fail over to an unsynchronised slave on controlled master shutdown (i.e. explicit stop of the RabbitMQ service or shutdown of the OS) in order to avoid message loss; instead the entire queue will shut down as if the unsynchronised slaves were not there. An uncontrolled master shutdown (i.e. server or node crash, or network outage) will still trigger a failover even to an unsynchronised slave.
If you would prefer to have master nodes fail over to unsynchronised slaves in all circumstances (i.e. you would choose availability of the queue over avoiding message loss) then you can set the ha-promote-on-shutdown policy key to always rather than its default value of when-synced.
https://www.rabbitmq.com/ha.html
Hello stack community,
I have a question about Redis sentinel for a specific problem case. I use AWS with Multi AZ to create a sensu cluster.
On eu-central-1a I have a sensu+redis(M), a RBMQ+Sentinel and 2 others Sentinels. Same on eu-central-1b but the redis is my slave on this AZ.
What happen if there is a problem and eu-central-1a can not communicate with eu-central-1b ? What I think is that Sentinel on eu-central-1b should promote my redis slave to master, because he can not contact my redis master. So I should have 2 redis masters running together on 2 different AZ.
But when the link is retrieved between AZ, I will still have 2 masters, with 2 different datas. What will happen in this case ? One master will become a slave and data will be replicated without loss ? Do we need to restart a master and he will be a slave ?
Sentinel detects changes to the master for example
If the master goes down and is unreachable a new slave is elected. This is based on the quorum where multiple sentinels agree that the master has gone down. The failover then occurs.
Once the sentinel detects the master come back online it is then a slave I believe thus the new master continues I believe. You will loose data in the switchover from master to new master that in inevitable.
If you loose connection then yes sentinel wont work correctly as it relies on multiple sentinels to agree the master redis is down. You shouldn't use sentinel in a 2 sentinel system.
Basic solution would be for you to put a extra sentinel on another server maybe the client/application server that isn't running redis/sentinel this way you can make use of the quorum and sentinels agreeing the master is down.
I have 3 sentinels monitoring a master slave setup with a quorum of 2, I would like to increase this to 5 sentinels and a quorum of 3. However when I run SENTINEL SET master quorum 3 the change is not propagated to the other 2 sentinels. Is this correct, if a fail over does happen is the value of the last change taken?
This is intended behavior. All master level commands must be sent to each sentinel individually, it does not propagate commands to other sentinels. Send the command to each of your five sentinels and you will get the effect you are after.