I am configuring HA for ActiveMQ "Classic" using shared JDBC master/slave with MySQL. I wanted to keep track of who is the current master broker, listening for connections. Because in case of fail-over of a master broker the slave will become the new master. How the master broker will be identified so that we can keep track of it?
The "master" broker is the one which accepts connections. The "slave" broker will be passive and therefore will not accept connections. Therefore, you can know which broker is the master by attempting to connect to both. Whichever broker accepts your connection is the master.
Related
I am using jboss amq7.1/apache amq, When using replication as the HA policy for my cluster, it is documented that all data synchronization is done over the network, All persistent data received by the master broker is synchronized to the slave when the master drops from the network. A slave broker first needs to synchronize all existing data from the master broker before becoming capable of replacing it.
Per my understanding, if master broker is crashed instead of shutdown by administrator, no persistent data can be synced, therefore messages persisted in journal of master will be lost if the disk used by journal is broken, am I right?
Your understanding is not correct.
All persistent data received by the master broker is replicated to the slave when the master broker receives it so that when the master broker drops from the network (e.g. due to a crash) the slave can replace the master.
Replicating the data from the master to the slave when the master drops from the network would completely defeat the purpose of high availability.
Actually, if HA is configured as Master/Slave, whether network or journal replicated, the receipt of a message to the broker is FIRST replicated and ONLY if successful, it will be confirmed as received to the client.
Here is what I try to achieve with ActiveMQ:
I'd like to have 2 clusters of brokers: clusterA and clusterB. Data between these 2 clusters should be mirrored. So, when clusterA receives a message it will be stored at storageA and also this message should be forwarded to clusterB (if there is such demand) and stored in storageB. On the other hand if clusterB receives a message it should be forwarded to clusterA.
I'm wondering whether config like this considered to be valid according to described above:
<networkConnectors>
<networkConnector
uri="static:(failover(tcp://clusterB_broker1:port,tcp://clusterB_broker2:port,tcp://clusterB_broker3:port))"
name="bridge"
duplex="true"
conduitSubscriptions="true"
decreaseNetworkConsumerPriority="false"/>
</networkConnectors>
This is a valid configuration. It indicates (assuming that all ClusterA brokers are configured this way) that brokers in ClusterA will store and forward first to clusterB_broker1, and if it is down will instead store and forward to clusterB_broker2, and then to clusterB_broker3 if clusterB_broker2 is down. But depending on your intra-cluster broker configuration, it is not going to do what you want it to.
The broker configurations must be set up for failover themselves or else you will lose messages when clusterB_broker1 goes down. If clusterB brokers are not working together as described below, then when clusterB_broker1 goes down, any messages sent to it will not be present or accessible on the other clusterB brokers. New messages will forward to them.
How to do failover within the cluster depends on your ActiveMQ version.
The latest version (5.9.0) supports 3 failover (or master/slave) cluster configurations. For quick reference, they are:
Shared File System Master Slave
JDBC Master Slave
Replicated LevelDB Store
Earlier versions supported a master/slave configuration that had one master and one slave node where messages were forwarded to the slave broker. This setup was not well maintained, had bugs, and has been removed from ActiveMQ.
Why only can java provide support for failover protocol in activemq whereas not other languages.
My doubt is that in the failover protocol like failover://(tcp://host1:61616,tcp://host2:61616)?randomize=false also the client uses one of the the inner urls like tcp://host1:61616 and then how does the broker comes to know that the call was using some failover protocol or not and then how the broker decides that it needs to replicate the message ?
Please understand that failover protocol is meant for reconnect logic on client side only and AMQ broker isn't even aware if a client is using failover protocol or not.
From the official AMQ documentation:
The Failover transport layers reconnect logic on top of any of the
other transports.
The Failover configuration syntax allows you to specify any number of
composite uris. The Failover transport randomly chooses one of the
composite URI and attempts to establish a connection to it. If it does
not succeed or if it subsequently fails, a new connection is
established to one of the other uris in the list.
Not sure what you mean by replication here but as per the official doc
The Failover transport tracks transactions by default. The inflight
transactions are replayed on reconnection.
There are different scenarios to put up a HA solution with ActiveMQ.
If clients connect using the failover protocol to host1,host2, then the broker setup needs to be setup for HA as well.
One solution is to cluster host1 and host2 in an Active-Active solution. Then messages are always propagated when they are asked for - the queues are shared in the entire cluster among all amq brokers.
Otherwise, if the active-active solution is not prefered, then a master-slave solution can be setup where the two brokers, host1 and host2, share the data area (for instance using a Database for persistance or a shared SAN disk).
There are more combinations of setups, but the failover protocol assumes that the entire solution can handle that messages arrives to different brokers, if one goes down. As far as I know, there is no other magic in the failover protocol, from the broker perspective.
I am using ActiveMQ version 5.4 and I have a pure master slave configuration. My slave is configured such that starts its network transports connectors in the event of a failure. My clients are configured using the failover protocol, just like the docs say:
failover://(tcp://masterhost:61616,tcp://slavehost:61616)?randomize=false
When my master dies, the clients successfully fail over to the slave perfectly. The problem is that after I recover (i.e. stop the slave, copy over the data, restart the master, then restart the slave), the clients are still trying to connect to the the slave (which does not have any open network connectors at that point). Thus, the clients never reconnect to the master after restarting it. Is this how it's supposed to work?
I've seen this as well. If you're using the PooledConnectionFactory, set an expiry timeout on the pooled connections via setExpiryTimeout. The API documentation here suggests that this will force reconnection to the master broker:
allow connections to expire, irrespective of load or idle time. This is useful with failover to force a reconnect from the pool, to reestablish load balancing or use of the master post recovery
I have a little problem here with my sample JMS layout.
I have two brokers (A, B) on two machines, which are linked via network connector. The idea is that the producer can send to any broker and the consumer can listen to any broker and the topic to send to/receive from is available globally.
The topic has two durable subscriber clients (one on each machine) that both will process all the messages in the topic. I want it to be a durable subscription so that the processes won't loose any workload if a process has to be restarted. Both subscriber clients are configured to have a failover broker url, so that they first try to connect to their localhost broker and if not available to the other. Failover of the clients seems to work, but I found a problem in the following situation:
Each broker 'A' and 'B' have a subscriber client connected The producer is sending to 'A'. Broker 'B' gets restarted. Client of 'B' registers connection loss and switches to 'A'. 'B' comes up again, and because it had itself registered as a durable subscriber to 'A' it gets the message feed. It has no active durable subscriber now ('A' has now three, including 'B') and piles up until it reaches its connection limits.
Is my configuration wrong? Is it possible what I've intended?
Cheers,
Kai
Are you running master-slave configuration?
Why do you want both brokers to have connected clients at the same time?
If you user failover connection string (identifying both brokers in it) your consumers/producers will use ActiveMQ failover implementation and will connect/reconnect to the active node when needed. I don't think having two active instances with active clients is a good idea - unless you are trying to duplicate your processes (in this case there will be no synchronization)
To make both nodes (master and slave) to always have the same durable data you need
to persist your messages to the same place accessible to both nodes. It can be JDBC adapter connected to a single instance of database (probably behind the cluster) or it can be NAS with shared network folder for KahaDB.