According to this page:Curator Leader Latch, when the leader lose connection with zookeeper server, there will not be a leader until the connection is re-established. What does it mean? If the leader never reconnected to the server again, then other participants will never become leader?
The default behavior is that the instance suffering from the connection is LOST or SUSPENDED will not become leader. One of the other instances on another JVMs may become a leader.
Related
In a RabbitMQ Quorum Queue (using raft) cluster of say 4 nodes (N1-N4),
Can I have a consumer that can read only from N1/N2? In this case, will a message produced in N3, be delivered to a consume via N1/N2?
As per the documentation from the below post:
https://www.cloudamqp.com/blog/2019-04-03-quorum-queues-internals-a-deep-dive.html
With Raft, all reads and writes go through a leader whose job it is to
replicate the writes to its followers. When a client attempts to
read/write to a follower, it is told who the leader is and told to
send all writes to that node. The leader will only confirm the write
to the client once a quorum of nodes has confirmed they have written
the data to disk. A quorum is simply a majority of node
If this is the case, How can scaling be achieved if it's just the leader node that's gonna do all the work?
First of all, RabbitMQ clusters should have an odd number of nodes, so that a majority can always be established in the event of a network partition.
Consumers can always read from any node in a RabbitMQ cluster. If a queue master/mirror is not running on the node to which the consumer is connected, the communication will be forwarded to another node.
How can scaling be achieved if it's just the leader node that's gonna
do all the work?
"scaling" is so non-specific a word that I hesitate to answer this. But I assume you're asking what happens with multiple quorum queues. The answer is that each queue has its own leader, and these leaders are distributed around the cluster.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I set up a basic test topology with Petabridge Lighthouse and two simple test actors that communicate with each other. This works well so far, but there is one problem: Lighthouse (or the underlying Akka.Cluster) makes one of my actors the leader, and when not shutting the node down gracefully (e.g. when something crashes badly or I simply hit "Stop" in VS) the Lighthouse is not usable any more. Tons of exceptions scroll by and it must be restarted.
Is it possible to configure Akka.Cluster .net in a way that the rest of the topology elects a new leader and carries on?
There are 2 things to point here. One is that if you have a serious risk of your lighthouse node going down, you probably should have more that one -
akka.cluster.seed-nodes setting can take multiple addresses, the only requirement here is that all nodes, including lighthouses, must have them specified in the same order. This way if one lighthouse is going down, another one still can take its role.
Other thing is that when a node becomes unreachable (either because the process crashed on network connection is unavailable), by default akka.net cluster won't down that node. You need to tell it, how it should behave, when such thing happens:
At any point you can configure your own IDowningProvider interface, that will be triggered after certain period of node inactivity will be reached. Then you can manually decide what to do. To use it add fully qualified type name to followin setting: akka.cluster.downing-provider = "MyNamespace.MyDowningProvider, MyAssembly". Example downing provider implementation can be seen here.
You can specify akka.cluster.auto-down-unreachable-after = 10s (or other time value) to specify some timeout given for an unreachable node to join - if it won't join before the timeout triggers, it will be kicked out from the cluster. Only risk here is when cluster split brain happens: under certain situations a network failure between machines can split your cluster in two, if that happens with auto-down set up, two halves of the cluster may consider each other dead. In this case you could end up having two separate clusters instead of one.
Starting from the next release (Akka.Cluster 1.3.3) a new Split Brain Resolver feature will be available. It will allow you to configure more advanced strategies on how to behave in case of network partitions and machine crashes.
I'm trying to provide disaster recovery between two data centers for a RabbitMQ. The secondary datacenter is passive until the primary DC goes down.
Federation of queues is inappropriate because it wouldn't move messages until the consumers in the secondary DC go active. That shouldn’t happen unless the primary DC is unavailable at which point those messages are inaccessible.
I’ve considered creating an extra queue in the primary DC that would receive a copy of each message and then use Federation or Shovel to copy those messages to the secondary. The issue then becomes removing the duplicate message from the secondary DC when the “original” in the primary DC is processed.
Mirroring the queue to a node in the secondary DC would be work, except that RabbitMQ won’t cluster over a WAN due to latency.
Has anyone else faced this scenario? Thanks.
you quite eloquently explain the issues with using Federation and Shovels to try and solve DR with RabbitMq. Rabbit isn't really designed to move data efficiently over a WAN.
Moving data across a WAN always presents problems for a messaging solutions. For instance IBM MQ has multi-instance queue managers for HA, but needs a SAN for DR which becomes expensive both in product and processing time.
Another free product like Rabbit that you could use is Solace. It has HA and DR replication built into it. It can manage active/passive passive DR scenario you describe by moving each message across the WAN asynchronously in near realtime. As soon as you're ready to move application traffic to the backup DC, you can activate the backup instance and start consuming messages. It automatically "removes the duplicate message" as it is consumed from the active side.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Hallo i am mohamad a student in masters degree
I want to ask a question about Zookeeper.
I read that the write operation in zookeeper to be done first the server connected to the cliend must contact the leader and then the leader makes a vote and when he gets more than the half of servers then he replies to the server which is connected to the cliend inorder to go on with this operartion .
But my question is that for what is the voting procccess? i mean they vote for what?
and the second question is that how do they vote do they send messages or how do they notify the leader. and the third question is that why do they need voting, i read that there is a versioned number inorder to check the update data so why do they need voting ???
Please can anybody reply for me as fast as he can .
Thanks in advance
This is the fastest I can. You need to get the book Distributed Algorithms by Nancy Lynch for better understanding of how these systems works.
Background -
The algorithm paradigm is called Paxos though Zookeeper has its own implementation which is a bit different. Zookeeper commits data in a two phased commit. All the communication happens using Atomic Broadcast Protocol over a FIFO channel to preserve ordering.
What is the voting processes - Voting process is for finding a leader not for two phased commit. No ballots nothing. Node with highest zxid is the leader.
Voting is for leader election. Two phased commit is for write operations. For more info check out the zookeeper docs and more importantly dist algos book to understand why these things behave the way they are :).
--Sai
ZooKeeper follows a simple client-server model where clients are nodes (i.e., machines) that make use of the service, and servers are nodes that provide the service. A collection of ZooKeeper servers forms a ZooKeeper ensemble. At any given time, one ZooKeeper client is connected to one ZooKeeper server. Each ZooKeeper server can handle a large number of client connections at the same time. Each client periodically sends pings to the ZooKeeper server it is connected to let it know that it is alive and connected. The ZooKeeper server in question responds with an acknowledgment of the ping, indicating the server is alive as well. When the client doesn't receive an acknowledgment from the server within the specified time, the client connects to another server in the ensemble, and the client session is transparently transferred over to the new ZooKeeper server. Check this to understand the zookeeper architecture
When a client requests to read the contents of a particular znode, the read takes place at the server that the client is connected to. Consequently, since only one server from the ensemble is involved, reads are quick and scalable. However, for writes to be completed successfully, a strict majority of the nodes of the ZooKeeper ensemble are required to be available. When the ZooKeeper service is brought up, one node from the ensemble is elected as the leader. When a client issues a write request, the connected server passes on the request to the leader. This leader then issues the same write request to all the nodes of the ensemble. If a strict majority of the nodes (also known as a quorum) respond successfully to this write request, the write request is considered to have succeeded. A successful return code is then returned to the client who initiated the write request. If a quorum of nodes are not available in an ensemble, the ZooKeeper service is nonfunctional. Check this to understand the voting process for write operations
For the service to be reliable and scalable, it is replicated over a set of machines. ZooKeeper uses a version of the famous Paxos algorithm, to keep replicas consistent.
Zookeeper gives the following consistency guarantees
Sequential Consistency Updates from a client will be applied in the order that they were sent.
Atomicity Updates either succeed or fail -- there are no partial results.
Single System Image A client will see the same view of the service regardless of the server that it connects to.
Reliability Once an update has been applied, it will persist from that time forward until a client overwrites the update. This guarantee has two corollaries:
Timeliness The clients view of the system is guaranteed to be up-to-date within a certain time bound (on the order of tens of seconds). Either system changes will be seen by a client within this bound, or the client will detect a service outage.
Here are the answers to your questions
Question1: The voting is for whether the write operation should be committed or not.
Question2: The communication between clients and servers in the zookeeper ensemble happen over a TCP connection through the exchange of messages using ZAB protocol
Question3: For the service to be reliable and fault tolerant, the data has to be replicated to a quorum of servers.
I have implemented a wcf service and now, my client wants it to have three copies of it, working independently on different machines. A master-slave approach. I need to find a solution that will enable behavior:
the first service that is instantiated "asks" the other two "if they are alive?" - if no, then it becomes a master and it is the one that is active on the net. The other two, once instantiated see that there is already a master alive, so they became slaves and start sleeping. There needs to be some mechanisms to periodically check if master is not dead and if so, choses the next copy that is alive to became a master (until it becomes dead respectively)
This i think should be a kind of an architectural pattern, so I would be more than glad to be given any advices.
thanks
I would suggest looking at the WCF peer channel (System.Net.PeerToPeer) to facilitate each node knowing about the other nodes. Here is a link that offers a decent introduction.
As for determining which node should be the master, the trick will be negotiating which node should be the master if two or more nodes come online at about the same time. Once the nodes become aware of each other, there needs to be some deterministic mechanism for establishing the master. For example, you could use the earliest creation time, the lowest value of the last octet of each node's IP address, or anything really. You'll just need to define some scheme that allows the nodes to negotiate this automatically.
Finally, as for checking if the master is still alive, I would suggest using the event-based mechanism described here. The master could send out periodic health-and-status events that the other nodes would register for. I'd put a try/catch/finally block at the code entry point so that if the master were to crash, it could publish one final MasterClosing event to let the slaves know it's going away. What this does not account for is a system crash, e.g., power failure, etc. To handle this, provide a timeout in the slaves so that when the timeout expires, the slaves can query the master to see if it's still there. If not, the slaves can negotiate between themselves using your deterministic algorithm about who should be the next master.