I want to configure mirrored queues in my applications so that if any node is down, it can survive failure and reconnect to the mirrored nodes.
Mirroring of queues is done. And i have added ip address and port in application.properties file to get connection on failures.
application.properties
spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
spring.rabbitmq.addresses=172.20.97.39:48442,172.20.97.144:57037,127.0.0.1:3962
When i stop the master node, i expect the queuing to continue on other nodes. What is happening right now is i am getting failed message stating not able to enqueue.
Can someone please help me to know are there other code related changes i'll have to do to configure mirrored queues or something which i am missing.
Any help will be appreciated.
Thanks in advance.
Related
I have a rabbitMQ cluster with two nodes configured to be synchronized. Each queue is mirrored and persistent.
Each time I need to reboot a node of my cluster, some old messages are replayed.
I don’t understand why because one of the two nodes is still alive and they are "normally" synchronized.
Have you any idea to help me to investigate this problem?
Could you check if you have some messages that are not acknowledged?
If you do (this would mean a consumer never acknowledges it), it could explain the behavior:
Message consumed but never acknowledged
Node reboots
The connections of the consumers connected to that node get closed
Any of the unacked messages that have been consumed in the related channel are put back in the queue
I have rabbitMQ broker running on two nodes as a cluster. I have observed that if node, where queue have been created, goes down, then queue would not be available on other node. If I try to publish a message from other node it fails. Even if I remove the failed node from cluster(using forget cluster command) and try to publish message from other node, the behavior is same.
I dont want to enable mirroring of the queue for the simple reason that it would replicate the messages which would be additional load on inter-network.
Is there way available in rabbitMQ to achieve this?
The behaviour you are experiencing is the default behaviour of rabbitmq and its exactly what i supposed to happen.The node where you created the queue becomes the producer now and if this node goes down then any connection available to it or queues or exchanges associated with it will not work at all. There are two options to resolve this issue.
One option is that there is one separate queue for every node and any node that wants to listen to receive messages from a particular node can subscribe to that particular queue's exchange. This seems to be not a very good idea since you need to manage a lot of things for it.
Second option is to always declare a queue before you publish so if your queue is not available then a new queue would take its place and all the nodes subscribe to would be able to listen and any producer node will be able to post that queue. This option will resolve the problems of node getting down or node not available. from the docs
before sending we need to make sure the recipient queue exists. If we send a message to non-existing location, RabbitMQ will just drop the message. Let's create a hello queue to which the message will be delivered:
RabbitMQ lets you import and export definitions. Definitions are json files which contain all broker objects (queues, exchanges, bindings, users, virtual hosts, permissions and parameters). They do not include the messages of queues.
You can export definitions of the node who owns the queue and import them to the slave node of the cluster periodically. You have to enable the management plugin for this task.
More information here: https://www.rabbitmq.com/management.html#configuration
Hoping someone might be able to give me some guidance on this. I have setup a RabbitMQ cluster on 3 nodes.
- Node1 (Master), Node2 and Node3
The part I can't figure out is I cannot find any information about what happens if Master goes down, what is the client pointing to?
If Node1 goes down, then surely the clients which are configured to point to Node1 will not be able to automatically fail-over to whatever becomes the new master?
My idea was to setup a load balancer which detects which node is master and only ever redirects requests to that node?
Any thoughts or ideas?
Thanks :)
Spring amqp helps in solving this easily. It provides the failover mechanism out of the box. You can just configure the address of the nodes with comma separated and pass it to addresses parameter while creating the connection factory.
For eg:
<rabbit:connection-factory id="rabbitConnectionFactory" addresses="192.168.1.2:5672,192.168.1.3:5672" username="${rabbitmq.username}"
connection-factory="clientConnectionFactory"
cache-mode="${rabbit.cacheMode}"
channel-cache-size="${rabbit.channelCacheSize}"
password="${rabbitmq.password}" virtual-host="${rabbitmq.vhost}"/>
This way, when the first node goes down, the connection will be made to the second node.
If clients are only capable of connecting to one node, and that node goes down, clients will lose connectivity. You need (as you surmise) some kind of setup whereby either:
you place a load-balancer in front of the nodes, and the clients connect only to the load-balancer; OR
the clients can connect to any of the available nodes, trying each in turn.
Whichever solution, there is no need to direct all client traffic to the master node. Clients nodes can connect to either a master or slave broker.
At CloudAMQP we have one single URL to the cluster that clients connect to and we recommend to always configure clients to automatically reconnect on connection loss.
We have fail over and load balancing between nodes, and it's done over DNS, with a low TTL (30s).
https://www.cloudamqp.com/blog/2015-12-29-cloudamqp-plan-setup-pause-minority-mirrored-nodes-and-the-cap-theorem.html
I have a redis master-slave setup and the configuration of the slave is set to slave_read_only:1, but when I enter a PUBLISH command on the slave node it does not fail. I would expect an error, but it just takes the command and nothing else happens. The message is not propagated to the master either.
The question is, why is that? Did I mis-configure redis? Is that a feature? To what purpose? Or is it just a bug?
The problem arises in a setup where automatic failover occurs. A master may become a slave and clients of that slave may publish messages without realizing that it is no master any more. Do I have to check before each message is sent if the redis node is still master?
I use redis 3.0.5
You didn't misconfigure - this is the defined behavior as PUBLISH isn't considered a write command.
Also note, that when replicating published events are replicated from master to slaves (downstream, as usual), so if you're publishing to a slave only clients connected to it or to its slaves and subscribed to the relevant channel will get the message.
I am new to RabbitMq. I am not able to understand the concept here. Please find the scenario.
I have two machines (RMQ1, RMQ2) where I have installed rabbitmq in both the machines which are running. Again I clustered RMQ2 to join RMQ1
cmd:/> rabbitmqctl join_cluster rabbit#RMQ1
If you see the status of the machines here it is as below
In RMQ1
c:/> rabbitmqctl cluster_status
Cluster status of node rabbit#RMQ1...
[{nodes,[{disc,[rabbit#RMQ1,rabbit#RMQ2]}]},
{running_nodes,[rabbit#RMQ1,rabbit#RMQ2]}]
In RMQ2
c:\> rabbitmqctl cluster_status
Cluster status of node rabbit#RMQ2 ...
[{nodes,[{disc,[rabbit#RMQ1,rabbit#RMQ2]}]},
{running_nodes,[rabbit#RMQ1,rabbit#RMQ2]}]
The in order to publish and subscribe message I am connecting to RMQ1. Now I see the whenever I sent or message to RMQ1, I see message mirrored in both RMQ1 and RMQ2. This I understand clearly that as both the nodes are in same cluster they are getting mirrored across nodes.
Let say I bring down the RMQ2, I still see message getting published to RMQ1.
But when I bring down the RMQ1, I cannot publish the message anymore. From this I understand that RMQ1 is master and RMQ2 is slave.
Now I have below questions, without changing the code :
How do I make the RMQ2 take up the job of accepting the message.
What is the meaning of Highly Available Queues.
How should be the strategy for implementing this kind scenario.
Please help
Question #2 is best answered first, since it will clear up a lot of things for you.
What is the meaning of highly available queues?
A good source of information for this is the Rabbit doc on high availability. It's very important to understand that mirroring (which is how you achieve high availability in Rabbit) and clustering are not the same thing. You need to create a cluster in order to mirror, but mirroring doesn't happen automatically just because you create a cluster.
When you cluster Rabbit, the nodes in the cluster share exchanges, bindings, permissions, and other resources. This allows you to manage the cluster as a single logical broker and utilize it for scenarios such as load-balancing. However, even though queues in a cluster are accessible from any machine in the cluster, each queue and its messages are still actually located only on the single node where the queue was declared.
This is why, in your case, bringing down RMQ1 will make the queues and messages unavailable. If that's the node you always connect to, then that's where those queues reside. They simply do not exist on RMQ2.
In addition, even if there are queues and messages on RMQ2, you will not be able to access them unless you specifically connect to RMQ2 after you detect that your connection to RMQ1 has been lost. Rabbit will not automatically connect you to some surviving node in a cluster.
By the way, if you look at a cluster in the RabbitMQ management console, what you see might make you think that the messages and queues are replicated. They are not. You are looking at the cluster in the management console. So regardless of which node you connect to in the console, you will see a cluster-wide view.
So with this background now you know the answer to your other two questions:
What should be the strategy for implementing high availability? / how to make RMQ2 accept messages?
From your description, you are looking for the failover that high availability is intended to provide. You need to enable this on your cluster. This is done through a policy, and there are various ways to do it, but the easiest way is in the management console on the Admin tab in the Policies section:
The previously cited doc has more detail on what it means to configure high availability in Rabbit.
What this will give you is mirroring of queues and messages across your cluster. That way, if RMQ1 fails then RMQ2 will still have your queues and messages since they are mirrored across both nodes.
An important note is that Rabbit will not automatically detect a loss of connection to RMQ1 and connect you to RMQ2. Your client needs to do this. I see you tagged your question with EasyNetQ. EasyNetQ provides this "failover connect" type of feature for you. You just need to supply both node hosts in the connection string. The EasyNetQ doc on clustering has details. Note that EasyNetQ even lets you inject a simple load balancing strategy in this case as well.