Hoping someone might be able to give me some guidance on this. I have setup a RabbitMQ cluster on 3 nodes.
- Node1 (Master), Node2 and Node3
The part I can't figure out is I cannot find any information about what happens if Master goes down, what is the client pointing to?
If Node1 goes down, then surely the clients which are configured to point to Node1 will not be able to automatically fail-over to whatever becomes the new master?
My idea was to setup a load balancer which detects which node is master and only ever redirects requests to that node?
Any thoughts or ideas?
Thanks :)
Spring amqp helps in solving this easily. It provides the failover mechanism out of the box. You can just configure the address of the nodes with comma separated and pass it to addresses parameter while creating the connection factory.
For eg:
<rabbit:connection-factory id="rabbitConnectionFactory" addresses="192.168.1.2:5672,192.168.1.3:5672" username="${rabbitmq.username}"
connection-factory="clientConnectionFactory"
cache-mode="${rabbit.cacheMode}"
channel-cache-size="${rabbit.channelCacheSize}"
password="${rabbitmq.password}" virtual-host="${rabbitmq.vhost}"/>
This way, when the first node goes down, the connection will be made to the second node.
If clients are only capable of connecting to one node, and that node goes down, clients will lose connectivity. You need (as you surmise) some kind of setup whereby either:
you place a load-balancer in front of the nodes, and the clients connect only to the load-balancer; OR
the clients can connect to any of the available nodes, trying each in turn.
Whichever solution, there is no need to direct all client traffic to the master node. Clients nodes can connect to either a master or slave broker.
At CloudAMQP we have one single URL to the cluster that clients connect to and we recommend to always configure clients to automatically reconnect on connection loss.
We have fail over and load balancing between nodes, and it's done over DNS, with a low TTL (30s).
https://www.cloudamqp.com/blog/2015-12-29-cloudamqp-plan-setup-pause-minority-mirrored-nodes-and-the-cap-theorem.html
Related
According to https://groups.google.com/forum/#!topic/rabbitmq-users/vvWAymjDww4, if a queue is mirrored across multiple servers (nodes), it doesn't matter which queue a publisher writes to - RabbitMQ will always forward the message to the master node (the node where the queue was first created).
If this is the case, what's the point of putting a load balancer in front of the nodes if every message will end up being routed to the same node anyway? It seems the master node will always take the entire load.
I'll put an example scenario to make it simple to understand:
You have a 2 node cluster, and you have a load balancer on top
The client connects to the LB, and gets connected to node1, which is where queue1 resides (master)
The queues have HA enabled, therefore queue1 is also replicated on node2
node1 fails for some reason, so the new master for queue1 is now node2
Client connection drops, and the LB connects him now to node2, so everything is still OK for the client
Basically, it's all about service availability. When you connect to a node, and the queue you're consuming from is not the master, internally RabbitMQ redirects you to the master node, which can end up creating a lot of inter-connections in the cluster.
But if the master node fails, RabbitMQ elects a new node, so having an LB is a great addition there, where the client will reconnect to the LB and it'll get redirected to a working instance. The process for the client is transparent.
Finally, depending on your use-case, you can do clever things to make sure the client always connects to the instance where the master queue resides, but that really depends on your specific requirements.
So the documentation to the "Replicated LevelDB Store" says:
The elected master broker node starts and accepts client connections. The other nodes go into slave mode and connect the the master and synchronize their persistent state /w it. The slave nodes do not accept client connections. All persistent operations are replicated to the connected slaves. If the master dies, the slaves with the latest update gets promoted to become the master. The failed node can then be brought back online and it will go into slave mode.
So one chosen master exist, it accepts client connections and the rest are replicated slave nodes who do not accept client connections. Fine.
So if the master dies it's all working fine - the master gets reelected, clients disconnect and they eventually connect to the new master. Awesome.
Now what happens if the master isn't dead from the perspective of Zookeeper, but it's just NOT ACCESSIBLE from clients. So a master is chosen, it's considered live(as i understand zookeeper's need to be able to connect to it to be considered available), but the actual clients can't connect to it?
Sure clients CAN connect to the other slave nodes, they just can't connect to the master. But the master won't ever be changed as it's live. Is that how it works?
Not sure i understood it right.
LevelDB support in ActiveMQ is deprecated and has been for quite some time (years) so I'd suggest not bothering with it as there is no support and plenty of open bugs that will not be fixed.
I'd suggest taking a look instead at ActiveMQ Artemis.
You understand it right, and it's a reasonable design.
Clients only commuicate to master, and slaves are just used for backup. If what you described really happens, maybe caused by network problem, then you should fix the network(or any other possible reasons).
i have the following scenario which i want to fulfill:
rabbit mq must be loadbalanced (is it something which is provided by rabbitmq out of the box OR something like haproxy load balancer would work great. Which one is well loadbalanced.)
CAN haproxy directly push messages to rabbitmq (lets say a POST request coming to http://localhost:3333/redirectToRabbit gets redirected to rabbit and optionally either the ACK or RESPONSE goes back to client. Also note haproxy would load balance the request)
with HA; what the best configuration ( exchange with durable queue, durable queue or something else. NOTE: How would the messages gets redirected to some other rabbitmq instance if one of the rabbitmq instance goes down -- persisted and auto redirection to available rabbitmq )
Assuming you setup a two-node RabbitMQ cluster. Before talking about ha proxy, you need to understand the ha policies and the behavior of ha queues first. Different ha options might cause completely different behaviors of RabbitMQ message replication and node failover. RabbitMQ is so flexible, so don't expect a golden way of configuration which could meet all scenarios.
Then, since you have two nodes which could accept connections, your client could either use a loadbalancer (such as ha proxy) or to use a client driver which supports connecting to multiple nodes of a cluster. Either way will work.
When using haproxy, you have one load balancer ip. Client connects only to this load balancer ip, the load balancer forward you connection to the underlying nodes. But as long as a connection created, the client connection instance keeps talking to one of the node. When one of the node is down, if no "Health Checking" options are configured in your load balancer, client might get random connection failures. When you have "Health Checking" options configured correctly, the load balancer knows which nodes are down, so that clients will only connect to healthy nodes, which solves the issue.
When not using a load balancer and only base on client driver to connect to all the nodes, the client driver should be able to handle connection failure or health check internally and do failover/retry, etc, to ensure connections go to healthy nodes.
I have a Redis Cluster that clients are connecting to via HAPRoxy with a Virtual IP. The Redis cluster has three nodes (with each node sharing the same server with a running sentinel instance).
My question is, when i clients gets a "MOVED" error/message from a cluster node upon sending a request, does it bypass the HAProxy the second time when it connects since it has been provided with an IP:port when the MOVEd message was issued? If not, how does the HAProxy know the second time to send it to the correct node?
I just need to understand how this works under the hood.
If you want to use HAProxy in front of Redis Cluster nodes, you will need to either:
Set up an HAProxy for each master/slave pair, and wire up something to update HAProxy when a failure happens, as well as probably intercept the topology related commands to insert the virtual IPs rather than the IPs the nodes themselves have and report via the topology commands/responses.
Customize HAProxy to teach it how to be the cluster-aware Redis client so the actual client doesn't know about cluster at all. This means teaching it the Redis protocol, storing the cluster's topology information, and selecting the node to query based on the key(s) being accessed by the consumer code.
With Redis Cluster the client must be able to access every node in the cluster. Of the two options above Option 2 is the "easier" one, but at this point I wouldn't recommend either.
Conceivably you could use the VIP as a "first place to get the topology info" IP but I suspect you'd have serious issues develop as that original IP would not be one of the ones properly being reported as a nod handling data. For that you could simply use round-robin DNS and avoid that problem, or use the built-in "here is a list of cluster IPs (or names?)" to the initial connection configuration.
Your simplest, and least likely to be problematic, route is to go "full native" and simply give full and direct access to every node in the cluster to your clients and not use HAProxy at all.
I'm trying to set up three brokers in a network for load balancing -- clients and producers can connect to any of these brokers.
Questions:
What is the recommended topology to use to network these brokers? More specifically, what is the networkConnector configuration to use on each of these brokers? should duplex setting be enabled? (I guess duplex setting depends on the topology we choose)
A->B->C->A or A<-->B<-->C<-->A
Client should use failover protocol to connect to these brokers, right? e.g. failover://(tcp://b1:6161, tcp://b2:6161, tcp://b3:6161)
Any duplicate message handling required on the client side in case of restarts? See http://forum.springsource.org/showthread.php?108461-Failover-issue-in-ActiveMQ -- not clear why duplicate message issue exists here
Ideally we want to set up topology as shown in this post http://edelsonmedia.com/?p=143 -- not clear how to set up networkConnector on masters and slaves.
1.) I can't actually recommend a topology. This choice depends on the number of hops (between the broker where the messages enters the cluster and the broker where the consumer conects to) you can accept. In a heave traffic scenario every hop adds to the network load.
In my company we use a hypercube network (every broker knows every other broaker) and it works great.
Generaly you should make sure that your node configurations are as similar as possible. Using duplex makes sure you have less connections to configure (since the connection from B to A is already part of the duplex connection from A to B) but it introduce a large number of differences into your config file.
Personaly i created my own start script for ActiveMQ that auto-generated the connection config based on the dns names of my cluster (mycluster-01 to 06).
2.) yes. You might want to add ?randomize=false if you want to make sure the client uses the first entry in the list.
3.) Duplicate entries can happen if there are failures during message transport or as race conditions during heavy load. In general one message only is owned by one broker.
4.) dont set up network connectors between masters and slaves (REALLY DONT). Use the pure Master Slave feature of activeMQ and configure the master for each slave (you don't have to configure anything on the masters). For the all Masters configure NetworkConnections to the other Masters with failover to their slaves)
http://activemq.apache.org/pure-master-slave.html