I need help from some rabbit experts around the ha-mode being set to exactly
At the moment we have a cluster of 3 nodes as shown below:
We have a policy setup as shown below:
I can see that a queue is mirrored to nodes 1 and 2 as shown below:
Question
Is there the case that node 3 could become the primary? and then not have the mirrored queue data in it? I'm worried that we should be mirroring to all 3 nodes not just the primary and one of the secondary nodes.
Is there the case that node 3 could become the primary?
Yes. You should read these resources:
https://www.rabbitmq.com/ha.html#behaviour
https://jack-vanlightly.com/blog/2018/9/10/how-to-lose-messages-on-a-rabbitmq-cluster
But! This does NOT mean you should always mirror to all nodes.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
In a RabbitMQ Quorum Queue (using raft) cluster of say 4 nodes (N1-N4),
Can I have a consumer that can read only from N1/N2? In this case, will a message produced in N3, be delivered to a consume via N1/N2?
As per the documentation from the below post:
https://www.cloudamqp.com/blog/2019-04-03-quorum-queues-internals-a-deep-dive.html
With Raft, all reads and writes go through a leader whose job it is to
replicate the writes to its followers. When a client attempts to
read/write to a follower, it is told who the leader is and told to
send all writes to that node. The leader will only confirm the write
to the client once a quorum of nodes has confirmed they have written
the data to disk. A quorum is simply a majority of node
If this is the case, How can scaling be achieved if it's just the leader node that's gonna do all the work?
First of all, RabbitMQ clusters should have an odd number of nodes, so that a majority can always be established in the event of a network partition.
Consumers can always read from any node in a RabbitMQ cluster. If a queue master/mirror is not running on the node to which the consumer is connected, the communication will be forwarded to another node.
How can scaling be achieved if it's just the leader node that's gonna
do all the work?
"scaling" is so non-specific a word that I hesitate to answer this. But I assume you're asking what happens with multiple quorum queues. The answer is that each queue has its own leader, and these leaders are distributed around the cluster.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
In this example I have a setup of 2 consumers and 2 publishers in my network. The centre is a RabbitMQ broker as shown in the screenshot below. Due to fail-safe reasons, I am wondering if RabbitMQ supports load-balancing or mirroring of the server (broker) in any way. I just would like to get rid of the star topology for two reasons:
1) If one broker fails, another publisher can take over immediately
2) If one brokers network throughput is not good enough the other takes over
Solving one or the other (or even both) would be great.
My current infrastructure
Preferred infrastructure
RabbitMQ clustering (docs) can meet your first requirement. Use three nodes and be sure your applications are coded and tested to take failure scenarios into account.
I don't know of anything out-of-the-box that can meet your second requirement. You will have to implement something that uses cluster statistics or application statistics to determine when to switch to another cluster due to lower throughput.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Is it possible to use RabbitMQ HA using multiple(2) RabbitMQ clusters?
Here is my requirement:
We have 2 RabbitMQ clusters (each with 4 nodes). All the nodes in both the clusters will be using same Erlang cookie. So that, even though these 2 clusters are physically in separate locations, but will act as a single cluster with 8 nodes.
We are planning to use HAProxy to load balance both the clusters (8 nodes). Both publisher and consumer will be using this proxy to connect to the broker.
We would like to use mirrored queues for HA with ha-mode:exactly, ha-params:4, ha-sync-mode:automatic along with auto-heal for cluster_partition_handling.
Question:
In case of HA, is there a way we can specify to use 2 nodes from the first cluster and 2 nodes from the second cluster. As I understand, this can be done via policy ha-mode:nodes and use node names but that way it will always use the same node, can this setup be dynamic?
As both the clusters are very reliable, will it be the right approach to use auto-heal for cluster_partition_handling (in case of split brain)?
As per this "By default, queues within a RabbitMQ cluster are located on a single node (the node on which they were first declared). This is in contrast to exchanges and bindings, which can always be considered to be on all nodes.". Does this mean exchanges are mirrored by default? So when a message arrives at an exchange and that node goes down, will the message be available on the other exchange on the other node?
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
So that, even though these 2 clusters are physically in separate locations, but will act as a single cluster with 8 nodes.
Please do not do this. RabbitMQ clusters require reliable network connections with low latency. If your cluster crosses a WAN or availability zone your chance of having network partitions greatly increases. See this section of the docs for more information. You should use either the shovel or federation feature.
Does this mean exchanges are mirrored by default? So when a message arrives at an exchange and that node goes down, will the message be available on the other exchange on the other node?
Yes and yes.
we are exploring the queue-master-locator 'min-masters' policy, and it looks like it works OK when we create new Queues, but when for some reason we need to stop one node of the cluster, every queues existing on that node get promoted (master) to the same node, for example:
Node A has 30 queues
Node B has 0 queues
Node C has 2 queues
Node D has 3 queues
When we stop Node A, all 30 queues get promoted to Node B, it is that the expected result?. We was hoping that the 30 queues would be distribuited across Node B, C, and D...
Please we are starting to get crazy arround this. Has everyone experimented this scenario? and It is posible to achieve what we are expecting to happen when shutdown Node A in some automatic way?
Our policy is defined as follow:
Listing policies ...
prod ha queues ^ {"ha-mode":"exactly","ha-params":3,"ha-sync-mode":"automatic","queue-master-locator":"min-masters"}
We have 4 nodes in the cluster, 2 RAM nodes and 2 Disk Nodes.
The policy works just fine when we create new queues but it does nothing when we stop one node.
Thanks
According to the official documentation:
If the master fails […] the longest running mirror is promoted to master
And the reason just follows:
the assumption being that it is most likely to be fully synchronised with the master.
https://www.rabbitmq.com/ha.html
The RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions here.
I just tried out your specific scenario using the latest pre-release code for 3.6.13 and I can reproduce it. The issue is that RabbitMQ does not move existing queue masters to rebalance queues - it will only move queues whose master existed on the node that went down. I have opened this issue to address what you report here.
You can use this unsupported script to rebalance queue masters in your cluster. If you have issues running the script, please post to rabbitmq-users and I will see it.
I have to say the official website provides very little information to understand RabbitMQ clearly.
The official website suggests using three nodes to build a cluster. What is the reason for that? I suppose it's like ZooKeeper, which needs an odd number of nodes to do a quorum and elect the master.
Also, what is the advantage of using a non-HA cluster? Improve the performance or what? If the node which a queue resides is down, then the queue is not working. So for all situation, is it necessary to set the cluster to be mirror queue and auto-sync?
Three nodes is the minimum to have a reasonable HA.
Suppose you have a queue mirrored in two nodes, if one gets down, another one will be promoted as the new slave or master.
Please read here section Automatically handling partitions and the section More about pause-minority mode
is therefore not a good idea to enable pause-minority mode on a
cluster of two nodes since in the event of any network partition or
node failure, both nodes will pause
RabbitMQ can handle the cluster in different ways, depending on where you deploy it - LAN or WAN or unstable LAN etc. And you can also use federation, shovel
what is the advantage of using a non-HA cluster? Improve the performance or what?
I'd say yes, or simply you have an environment where you don't need to have HA queues since you can have only temporary queues.
is it necessary to set the cluster to be mirror queue and auto-sync?
You can also decide for manual-sync, since when you sync the queue is blocked, and if you have lots of messages to sync, it can be a problem. For example, you can decide to sync the queues when you don't have traffic.
Here (section Unsynchronised Slaves) it is explained clearly.
Your question is a bit general, and it depends on what are you looking for.