Why only can java provide support for failover protocol in activemq whereas not other languages.
My doubt is that in the failover protocol like failover://(tcp://host1:61616,tcp://host2:61616)?randomize=false also the client uses one of the the inner urls like tcp://host1:61616 and then how does the broker comes to know that the call was using some failover protocol or not and then how the broker decides that it needs to replicate the message ?
Please understand that failover protocol is meant for reconnect logic on client side only and AMQ broker isn't even aware if a client is using failover protocol or not.
From the official AMQ documentation:
The Failover transport layers reconnect logic on top of any of the
other transports.
The Failover configuration syntax allows you to specify any number of
composite uris. The Failover transport randomly chooses one of the
composite URI and attempts to establish a connection to it. If it does
not succeed or if it subsequently fails, a new connection is
established to one of the other uris in the list.
Not sure what you mean by replication here but as per the official doc
The Failover transport tracks transactions by default. The inflight
transactions are replayed on reconnection.
There are different scenarios to put up a HA solution with ActiveMQ.
If clients connect using the failover protocol to host1,host2, then the broker setup needs to be setup for HA as well.
One solution is to cluster host1 and host2 in an Active-Active solution. Then messages are always propagated when they are asked for - the queues are shared in the entire cluster among all amq brokers.
Otherwise, if the active-active solution is not prefered, then a master-slave solution can be setup where the two brokers, host1 and host2, share the data area (for instance using a Database for persistance or a shared SAN disk).
There are more combinations of setups, but the failover protocol assumes that the entire solution can handle that messages arrives to different brokers, if one goes down. As far as I know, there is no other magic in the failover protocol, from the broker perspective.
Related
client must know all brokers using Failover Transport, right? Like that,
failover:(tcp://broker1:61616,tcp://broker2:61616,tcp://broker3:61616)
Is there optimization,so that the client does not have to know the existence of each broker ?
Put a TCP load balancer in front of the brokers. Only forward requests to the master broker. The LB can ping who's online or not by checking the "Slave" attribute of the broker via Jolokia/JMX.
A standalone approach would be to provide an URL to a comma separated list of broker URLs to try in case of failure. Can be done using the updateURIsURL option in the failover URI.
There is also some possibilities to auto-discover brokers using Multicast or by querying an LDAP directory, but that requires certain infrastructure in place. Read more about it here.
i have the following scenario which i want to fulfill:
rabbit mq must be loadbalanced (is it something which is provided by rabbitmq out of the box OR something like haproxy load balancer would work great. Which one is well loadbalanced.)
CAN haproxy directly push messages to rabbitmq (lets say a POST request coming to http://localhost:3333/redirectToRabbit gets redirected to rabbit and optionally either the ACK or RESPONSE goes back to client. Also note haproxy would load balance the request)
with HA; what the best configuration ( exchange with durable queue, durable queue or something else. NOTE: How would the messages gets redirected to some other rabbitmq instance if one of the rabbitmq instance goes down -- persisted and auto redirection to available rabbitmq )
Assuming you setup a two-node RabbitMQ cluster. Before talking about ha proxy, you need to understand the ha policies and the behavior of ha queues first. Different ha options might cause completely different behaviors of RabbitMQ message replication and node failover. RabbitMQ is so flexible, so don't expect a golden way of configuration which could meet all scenarios.
Then, since you have two nodes which could accept connections, your client could either use a loadbalancer (such as ha proxy) or to use a client driver which supports connecting to multiple nodes of a cluster. Either way will work.
When using haproxy, you have one load balancer ip. Client connects only to this load balancer ip, the load balancer forward you connection to the underlying nodes. But as long as a connection created, the client connection instance keeps talking to one of the node. When one of the node is down, if no "Health Checking" options are configured in your load balancer, client might get random connection failures. When you have "Health Checking" options configured correctly, the load balancer knows which nodes are down, so that clients will only connect to healthy nodes, which solves the issue.
When not using a load balancer and only base on client driver to connect to all the nodes, the client driver should be able to handle connection failure or health check internally and do failover/retry, etc, to ensure connections go to healthy nodes.
Here is what I try to achieve with ActiveMQ:
I'd like to have 2 clusters of brokers: clusterA and clusterB. Data between these 2 clusters should be mirrored. So, when clusterA receives a message it will be stored at storageA and also this message should be forwarded to clusterB (if there is such demand) and stored in storageB. On the other hand if clusterB receives a message it should be forwarded to clusterA.
I'm wondering whether config like this considered to be valid according to described above:
<networkConnectors>
<networkConnector
uri="static:(failover(tcp://clusterB_broker1:port,tcp://clusterB_broker2:port,tcp://clusterB_broker3:port))"
name="bridge"
duplex="true"
conduitSubscriptions="true"
decreaseNetworkConsumerPriority="false"/>
</networkConnectors>
This is a valid configuration. It indicates (assuming that all ClusterA brokers are configured this way) that brokers in ClusterA will store and forward first to clusterB_broker1, and if it is down will instead store and forward to clusterB_broker2, and then to clusterB_broker3 if clusterB_broker2 is down. But depending on your intra-cluster broker configuration, it is not going to do what you want it to.
The broker configurations must be set up for failover themselves or else you will lose messages when clusterB_broker1 goes down. If clusterB brokers are not working together as described below, then when clusterB_broker1 goes down, any messages sent to it will not be present or accessible on the other clusterB brokers. New messages will forward to them.
How to do failover within the cluster depends on your ActiveMQ version.
The latest version (5.9.0) supports 3 failover (or master/slave) cluster configurations. For quick reference, they are:
Shared File System Master Slave
JDBC Master Slave
Replicated LevelDB Store
Earlier versions supported a master/slave configuration that had one master and one slave node where messages were forwarded to the slave broker. This setup was not well maintained, had bugs, and has been removed from ActiveMQ.
I was able to set up Network of Brokers with store and forward strategy and working fine. I was given bigger machines now and would like to set up Master/Slave pair within the network of brokers. I understand Masters don't need any config changes but Slaves should indicate its corresponding master with URI. However, I'm not very clear on what uri to specify in the client. I'm using 5.6 release.
For example: Two machines with MasterA, SlaveB on 1 machine, and MasterB, SlaveA on another machine. No Network connectors between Masters and Slaves but network connectors between MasterA and MasterB. I hope that I'm right till this point. What about client uri? I'm currently using nio protocol at the clients like failover:(nio:localhost1:61616,nio:localhost2:61616)?randomize=true. I specify randomize=true to balance the load between the brokers.
Please suggest what client URI should I use? Should I include all brokers URI or just masters URI? Can I still use nio protocol? I prefer to use randomize=true so that load will be balanced.
In the simplest case, the client uri should contain 4 brokers, both pairs of master/slave uris.
For the network connectors, they will need to be prepared to bridge master to master or master to slave, which ever is available.
There is a new masterslave: discovery agent in 5.6 that simplifies the configuration for a networkconnector.
http://activemq.apache.org/networks-of-brokers.html#NetworksofBrokers-MasterSlaveDiscovery
I'm trying to set up three brokers in a network for load balancing -- clients and producers can connect to any of these brokers.
Questions:
What is the recommended topology to use to network these brokers? More specifically, what is the networkConnector configuration to use on each of these brokers? should duplex setting be enabled? (I guess duplex setting depends on the topology we choose)
A->B->C->A or A<-->B<-->C<-->A
Client should use failover protocol to connect to these brokers, right? e.g. failover://(tcp://b1:6161, tcp://b2:6161, tcp://b3:6161)
Any duplicate message handling required on the client side in case of restarts? See http://forum.springsource.org/showthread.php?108461-Failover-issue-in-ActiveMQ -- not clear why duplicate message issue exists here
Ideally we want to set up topology as shown in this post http://edelsonmedia.com/?p=143 -- not clear how to set up networkConnector on masters and slaves.
1.) I can't actually recommend a topology. This choice depends on the number of hops (between the broker where the messages enters the cluster and the broker where the consumer conects to) you can accept. In a heave traffic scenario every hop adds to the network load.
In my company we use a hypercube network (every broker knows every other broaker) and it works great.
Generaly you should make sure that your node configurations are as similar as possible. Using duplex makes sure you have less connections to configure (since the connection from B to A is already part of the duplex connection from A to B) but it introduce a large number of differences into your config file.
Personaly i created my own start script for ActiveMQ that auto-generated the connection config based on the dns names of my cluster (mycluster-01 to 06).
2.) yes. You might want to add ?randomize=false if you want to make sure the client uses the first entry in the list.
3.) Duplicate entries can happen if there are failures during message transport or as race conditions during heavy load. In general one message only is owned by one broker.
4.) dont set up network connectors between masters and slaves (REALLY DONT). Use the pure Master Slave feature of activeMQ and configure the master for each slave (you don't have to configure anything on the masters). For the all Masters configure NetworkConnections to the other Masters with failover to their slaves)
http://activemq.apache.org/pure-master-slave.html