So the documentation to the "Replicated LevelDB Store" says:
The elected master broker node starts and accepts client connections. The other nodes go into slave mode and connect the the master and synchronize their persistent state /w it. The slave nodes do not accept client connections. All persistent operations are replicated to the connected slaves. If the master dies, the slaves with the latest update gets promoted to become the master. The failed node can then be brought back online and it will go into slave mode.
So one chosen master exist, it accepts client connections and the rest are replicated slave nodes who do not accept client connections. Fine.
So if the master dies it's all working fine - the master gets reelected, clients disconnect and they eventually connect to the new master. Awesome.
Now what happens if the master isn't dead from the perspective of Zookeeper, but it's just NOT ACCESSIBLE from clients. So a master is chosen, it's considered live(as i understand zookeeper's need to be able to connect to it to be considered available), but the actual clients can't connect to it?
Sure clients CAN connect to the other slave nodes, they just can't connect to the master. But the master won't ever be changed as it's live. Is that how it works?
Not sure i understood it right.
LevelDB support in ActiveMQ is deprecated and has been for quite some time (years) so I'd suggest not bothering with it as there is no support and plenty of open bugs that will not be fixed.
I'd suggest taking a look instead at ActiveMQ Artemis.
You understand it right, and it's a reasonable design.
Clients only commuicate to master, and slaves are just used for backup. If what you described really happens, maybe caused by network problem, then you should fix the network(or any other possible reasons).
Related
I am using jboss amq7.1/apache amq, When using replication as the HA policy for my cluster, it is documented that all data synchronization is done over the network, All persistent data received by the master broker is synchronized to the slave when the master drops from the network. A slave broker first needs to synchronize all existing data from the master broker before becoming capable of replacing it.
Per my understanding, if master broker is crashed instead of shutdown by administrator, no persistent data can be synced, therefore messages persisted in journal of master will be lost if the disk used by journal is broken, am I right?
Your understanding is not correct.
All persistent data received by the master broker is replicated to the slave when the master broker receives it so that when the master broker drops from the network (e.g. due to a crash) the slave can replace the master.
Replicating the data from the master to the slave when the master drops from the network would completely defeat the purpose of high availability.
Actually, if HA is configured as Master/Slave, whether network or journal replicated, the receipt of a message to the broker is FIRST replicated and ONLY if successful, it will be confirmed as received to the client.
Hoping someone might be able to give me some guidance on this. I have setup a RabbitMQ cluster on 3 nodes.
- Node1 (Master), Node2 and Node3
The part I can't figure out is I cannot find any information about what happens if Master goes down, what is the client pointing to?
If Node1 goes down, then surely the clients which are configured to point to Node1 will not be able to automatically fail-over to whatever becomes the new master?
My idea was to setup a load balancer which detects which node is master and only ever redirects requests to that node?
Any thoughts or ideas?
Thanks :)
Spring amqp helps in solving this easily. It provides the failover mechanism out of the box. You can just configure the address of the nodes with comma separated and pass it to addresses parameter while creating the connection factory.
For eg:
<rabbit:connection-factory id="rabbitConnectionFactory" addresses="192.168.1.2:5672,192.168.1.3:5672" username="${rabbitmq.username}"
connection-factory="clientConnectionFactory"
cache-mode="${rabbit.cacheMode}"
channel-cache-size="${rabbit.channelCacheSize}"
password="${rabbitmq.password}" virtual-host="${rabbitmq.vhost}"/>
This way, when the first node goes down, the connection will be made to the second node.
If clients are only capable of connecting to one node, and that node goes down, clients will lose connectivity. You need (as you surmise) some kind of setup whereby either:
you place a load-balancer in front of the nodes, and the clients connect only to the load-balancer; OR
the clients can connect to any of the available nodes, trying each in turn.
Whichever solution, there is no need to direct all client traffic to the master node. Clients nodes can connect to either a master or slave broker.
At CloudAMQP we have one single URL to the cluster that clients connect to and we recommend to always configure clients to automatically reconnect on connection loss.
We have fail over and load balancing between nodes, and it's done over DNS, with a low TTL (30s).
https://www.cloudamqp.com/blog/2015-12-29-cloudamqp-plan-setup-pause-minority-mirrored-nodes-and-the-cap-theorem.html
Starting with Redis 2.8,redis add a function named "Partial resynchronization".I read this official document,but i don't understand.who can help me?
It is about master-slave replication.
The normal behavior of a Redis slave (slave of command, or configuration) is to connect to the master, ask the master to accumulate master-slave traffic, request a complete dump on filesystem to the master, download this dump on the slave, load the dump, and finally play the accumulated traffic until the slave catches up with the master.
This mechanism is quite robust but not very efficient to cover transient connection drops between the slave and the master. If the master-slave link is down for a couple of seconds, the slave will request a full resynchronization (involving a dump, etc ...), even if only a few commands have been missed.
Starting with 2.8, Redis includes a partial replication mechanism so a slave can reconnect to the master, and if some conditions are met (like a transient connection drop), asks the master to resynchronize without having to dump the whole memory instance.
In order to support this feature, the master has to buffer and keep a backlog of commands, so they can be served to the slaves at any time if needed. If the slave is too late behind the master, the backlog may not contain anymore the required data. In that case, a normal full synchronization is done, as in previous versions.
Assuming I have a Master-Slave deployment of Redis (1 master, 1 slave) and a client (webapp) that will manage Publish-Subscribe.
Can I Publish messages to the slave and will they be "seen" by the master?
Or should I use only the Master for Publish and the Slave for Subscribe commands?
I've been looking around but couldn't find the answer. Anyone knows?
EDIT: As #jameshfisher pointed out, the link below is regarding Redis Cluster. The comment from #lionello seems to be the correct answer:
Publishing to a slave will not propagate to the master, only the other way around.
The answer is on the cluster-spec docs:
Publish/Subscribe
In a Redis Cluster clients can subscribe to every node, and can also publish to every other node. The cluster will make sure that published messages are forwarded as needed.
The current implementation will simply broadcast each published message to all other nodes, but at some point this will be optimized either using Bloom filters or other algorithms.
For the typical data you store in Redis, you should only write to the master.
From http://redis.io/topics/replication:
...writes [to slaves] will be discarded if the slave and the master will [sic] resynchronize, or if the slave is restarted...
In fact, starting from v2.6, you can put slaves in slave-read-only mode which would prevent the mistake of writing data to a slave.
The documentation does go on to mention a potential use case for writing data to slaves:
...often there is ephemeral data that is unimportant that can be stored
into slaves. For instance clients may take information about
reachability of master in the slave instance to coordinate a fail over
strategy.
I am using ActiveMQ version 5.4 and I have a pure master slave configuration. My slave is configured such that starts its network transports connectors in the event of a failure. My clients are configured using the failover protocol, just like the docs say:
failover://(tcp://masterhost:61616,tcp://slavehost:61616)?randomize=false
When my master dies, the clients successfully fail over to the slave perfectly. The problem is that after I recover (i.e. stop the slave, copy over the data, restart the master, then restart the slave), the clients are still trying to connect to the the slave (which does not have any open network connectors at that point). Thus, the clients never reconnect to the master after restarting it. Is this how it's supposed to work?
I've seen this as well. If you're using the PooledConnectionFactory, set an expiry timeout on the pooled connections via setExpiryTimeout. The API documentation here suggests that this will force reconnection to the master broker:
allow connections to expire, irrespective of load or idle time. This is useful with failover to force a reconnect from the pool, to reestablish load balancing or use of the master post recovery