i have the following scenario which i want to fulfill:
rabbit mq must be loadbalanced (is it something which is provided by rabbitmq out of the box OR something like haproxy load balancer would work great. Which one is well loadbalanced.)
CAN haproxy directly push messages to rabbitmq (lets say a POST request coming to http://localhost:3333/redirectToRabbit gets redirected to rabbit and optionally either the ACK or RESPONSE goes back to client. Also note haproxy would load balance the request)
with HA; what the best configuration ( exchange with durable queue, durable queue or something else. NOTE: How would the messages gets redirected to some other rabbitmq instance if one of the rabbitmq instance goes down -- persisted and auto redirection to available rabbitmq )
Assuming you setup a two-node RabbitMQ cluster. Before talking about ha proxy, you need to understand the ha policies and the behavior of ha queues first. Different ha options might cause completely different behaviors of RabbitMQ message replication and node failover. RabbitMQ is so flexible, so don't expect a golden way of configuration which could meet all scenarios.
Then, since you have two nodes which could accept connections, your client could either use a loadbalancer (such as ha proxy) or to use a client driver which supports connecting to multiple nodes of a cluster. Either way will work.
When using haproxy, you have one load balancer ip. Client connects only to this load balancer ip, the load balancer forward you connection to the underlying nodes. But as long as a connection created, the client connection instance keeps talking to one of the node. When one of the node is down, if no "Health Checking" options are configured in your load balancer, client might get random connection failures. When you have "Health Checking" options configured correctly, the load balancer knows which nodes are down, so that clients will only connect to healthy nodes, which solves the issue.
When not using a load balancer and only base on client driver to connect to all the nodes, the client driver should be able to handle connection failure or health check internally and do failover/retry, etc, to ensure connections go to healthy nodes.
Related
There are three nodes in a RabbitMQ cluster as below.
Within RabbitMQ, there are two queues, q1 and q2.
The master replica of q1 and q2 are distributed on different nodes. Both queues are mirrored by other nodes.
There is a load balancer in front of three nodes.
AMQP(node port 5672) and Management HTTP API(node port 15672) are exposed by load balancer.
When application establishes a connection through load balancer, it could reach a random RabbitMQ node behind. And this is invisible to application.
Question:
Is it ok for application to consume both queues in a single AMQP channel over a single connection no matter which RabbitMQ node it reaches?
It is ok for application to call management HTTP API no matter which RabbitMQ node its request hits?
When RabbitMQ is set up as a cluster and you have your queues mirrored across them, it doesn't matter to which node you are connected. Because the AMQP connection for a queue will be automatically routed to the node containing the master queue and this handled by RabbitMQ internally. So, if a request to publish or consume on queue q1 comes, it will be routed to Node #1.
Answers to your question.
It is not advisable to consume more than one queues in a single AMQP connection. Exception from one consuming process may cause the connection to close which will interrupt the other one.
It is ok for application to call management HTTP API no matter which RabbutMQ node its request hits. Once management plugin in a RabbitMQ cluster is enabled, all the nodes will accept the Management HTTP API requests.
Reference: https://www.rabbitmq.com/clustering.html
I want to setup a rabbitmq cluster behind a load balancer and connect to it using spring amqp. Questions :
Does spring client need to know the address of each node in the RMQ cluster or it is sufficient for it to know just the address of load balancer.
If Spring client is only aware of the load balancer, how will it maintain connections/connection factory for each node in the cluster.
Is there any code sample, which shows how to make the spring client work with load balancer.
It only needs the load balancer; however, Spring AMQP maintains a long-lived shared connection, so a load balancer generally doesn't bring much value unless you have multiple applications.
With a single application (with one connection factory), you will only be connected to one broker.
Clarification
See the documentation.
Starting with version 1.3, the CachingConnectionFactory can be configured to cache connections as well as just channels. In this case, each call to createConnection() creates a new connection (or retrieves an idle one from the cache). Closing a connection returns it to the cache (if the cache size has not been reached). Channels created on such connections are cached too. The use of separate connections might be useful in some environments, such as consuming from an HA cluster, in conjunction with a load balancer, to connect to different cluster members. Set the cacheMode to CacheMode.CONNECTION.
By default all components (listener containers, RabbitTemplates) share a single connection to the broker.
Starting with version 2.0.2, RabbitTemplate has a property usePublisherConnection; if this is set to true, publishers will use a separate connection to the listener containers - this is generally recommended to avoid a blocked publisher connection preventing consumers from receiving messages.
As shown in the quote, the use of a single (or 2) connections is controlled by connection factory's cache mode.
Setting the cache mode to CONNECTION, means that each component (listener container consumer, RabbitTemplate) gets its own connection. In practice there will only be one or two publisher connections because publish operations are, generally, short lived and the connection is cached for reuse. You might get one or two more publisher connections if concurrent publish operations are performed.
I have a cluster of 2 RabbitMQ nodes (each running version 3.6.10 of RabbitMQ with MQTT plugin enabled) and an AWS classic load balancer in front of them. Server and clients exchange MQTT messages.
Clients (apps running on mobile devices and using Eclipse Paho client lib) connect to the load balancer which distributes connections in round-robin fashion.
When I bring down one node, say Node1, all clients that were connected to Node1 get a callback indicating connection to the broker is lost.
These clients try to reconnect but the connection attempt fails indicating the broker is not reachable.
From AWS console I can see that AWS ELB detects that Node1 is down and marks it as "OutOfService".
Connection requests from new clients are routed to the "InService" node Node2; however, connection requests from existing clients that were previously connected to Node1 always fail!
ELB is configured with idle timeout of 180 seconds. Enabling or disabling connection draining in ELB did not make any difference.
Is there any specific configuration to make ELB forget that the existing clients were connected to Node1 and allow them to connect to Node2?
I tried by adding following HA policy :
rabbitmqctl set_policy ha-mqtt "^mqtt" \ '{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}'
With this policy in place, all queues created for MQTT clients were mirrored. Now when Node1 is down, connection attempts from existing client IDs also get routed to the other active node!
This makes me wonder what is the relationship between client IDs from MQTT clients and their connection to broker nodes? I thought mirroring of queues is necessary only to retain and access messages that were not yet acknowledged when the queue master node goes down. But I see that the clients are not even able to establish a connection!
client must know all brokers using Failover Transport, right? Like that,
failover:(tcp://broker1:61616,tcp://broker2:61616,tcp://broker3:61616)
Is there optimization,so that the client does not have to know the existence of each broker ?
Put a TCP load balancer in front of the brokers. Only forward requests to the master broker. The LB can ping who's online or not by checking the "Slave" attribute of the broker via Jolokia/JMX.
A standalone approach would be to provide an URL to a comma separated list of broker URLs to try in case of failure. Can be done using the updateURIsURL option in the failover URI.
There is also some possibilities to auto-discover brokers using Multicast or by querying an LDAP directory, but that requires certain infrastructure in place. Read more about it here.
I'm trying to set up three brokers in a network for load balancing -- clients and producers can connect to any of these brokers.
Questions:
What is the recommended topology to use to network these brokers? More specifically, what is the networkConnector configuration to use on each of these brokers? should duplex setting be enabled? (I guess duplex setting depends on the topology we choose)
A->B->C->A or A<-->B<-->C<-->A
Client should use failover protocol to connect to these brokers, right? e.g. failover://(tcp://b1:6161, tcp://b2:6161, tcp://b3:6161)
Any duplicate message handling required on the client side in case of restarts? See http://forum.springsource.org/showthread.php?108461-Failover-issue-in-ActiveMQ -- not clear why duplicate message issue exists here
Ideally we want to set up topology as shown in this post http://edelsonmedia.com/?p=143 -- not clear how to set up networkConnector on masters and slaves.
1.) I can't actually recommend a topology. This choice depends on the number of hops (between the broker where the messages enters the cluster and the broker where the consumer conects to) you can accept. In a heave traffic scenario every hop adds to the network load.
In my company we use a hypercube network (every broker knows every other broaker) and it works great.
Generaly you should make sure that your node configurations are as similar as possible. Using duplex makes sure you have less connections to configure (since the connection from B to A is already part of the duplex connection from A to B) but it introduce a large number of differences into your config file.
Personaly i created my own start script for ActiveMQ that auto-generated the connection config based on the dns names of my cluster (mycluster-01 to 06).
2.) yes. You might want to add ?randomize=false if you want to make sure the client uses the first entry in the list.
3.) Duplicate entries can happen if there are failures during message transport or as race conditions during heavy load. In general one message only is owned by one broker.
4.) dont set up network connectors between masters and slaves (REALLY DONT). Use the pure Master Slave feature of activeMQ and configure the master for each slave (you don't have to configure anything on the masters). For the all Masters configure NetworkConnections to the other Masters with failover to their slaves)
http://activemq.apache.org/pure-master-slave.html