I have created a rabbitmq cluster with 3 nodes, if i run rabbitmqctl cluster_status on all 3 nodes, all of them report the 3 running nodes but each node reports a different cluster name, each node uses it's own name for the name of the cluster, is this normal;
Related
I have 3 redis servers running with 3 sentinels on each host
3 redis-3 sentinels(total 3 hosts)
Can I run sentinel on a separate host or it should always run along with redis-server?
3 redis on 3 hosts
3 sentinels on 3 other hosts(total 6 hosts)
Is it possible to monitor all the 3 redis servers with only one redis sentinel? 3 redis on 3 host
1 sentinel on 1 host(total 3 or 4 hosts)
You can run sentinel on a separate hosts or on the same hosts.
The benefit of running it in separate hosts is that the sentinel instances will not be affected by load on the Redis instances.
The benefit of running it on the same hosts is mainly cost.
It might be possible but doesn't make any sense.
The benefit of Redis sentinel deployment over Redis single node deployment is that it adds high availability (HA).
It means that in case of master failure one of the slaves will be promoted to master and the cluster will continue to function.
If you have only single sentinel instance, you don't have HA since failure in sentinel instance will cause the cluster to fail.
Therefore to achieve HA you must have at least 3 sentinel instances running on different physical nodes.
If you don't need HA, just run Redis single instance without sentinel.
I've just set up a Redis cluster, which is working fine. The structure is the following:
Server 1 | Server 2 | Server 3
Master A | Master B | Master C
Slave B | Slave C | Slave A
Master A <-> Slave A
Master B <-> Slave B
Master C <-> Slave C
I'm wondering what's best practice to add a whole new server to this cluster.
Server 4
Master ?
Slave ?
Of course, I can add new nodes via ./redis-trib.rb add-node. Is there a generic way?
You can use the CLUSTER MEET command to add a new node into a cluster. This command forces two nodes take each other as a trusted node, and the newly added node will be introduced to other nodes in the cluster by the gossip protocol.
The newly added node will be an empty master. In order to turn it into a slave of another master node. You can use the CLUSTER REPLICATE command.
In your case, you can do the following steps:
Log in SLAVE-D
Join SLAVE-D into the cluster by sending the MEET command: CLUSTER MEET MASTER-A-IP MATER-A-PORT.
Join MASTER-D into the cluster by sending the MEET command: CLUSTER MEET MASTER-D-IP MATER-D-PORT
Make SLAVE-D a slave of MASTER-D by sending the REPLICATE command: CLUSTER REPLICATE MASTER-D-node-id
I am trying to implement a Redis cluster with 6 machine.
I have a vagrant cluster of six machines:
192.168.56.101
192.168.56.102
192.168.56.103
192.168.56.104
192.168.56.105
192.168.56.106
all running redis-server
I edited /etc/redis/redis.conf file of all the above servers adding this
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-slave-validity-factor 0
appendonly yes
I then ran this on one of the six machines;
./redis-trib.rb create --replicas 1 192.168.56.101:6379 192.168.56.102:6379 192.168.56.103:6379 192.168.56.104:6379 192.168.56.105:6379 192.168.56.106:6379
A Redis cluster is up and running. I checked manually by setting value in one machine it shows up on other machine.
$ redis-cli -p 6379 cluster nodes
3c6ffdddfec4e726f29d06a6da550f94d976f859 192.168.56.105:6379 master - 0 1450088598212 5 connected
47d04bc98ab42fc793f9f382855e5c54ab8f2e20 192.168.56.102:6379 slave caf2cec45114dc8f4cbc6d96c6dbb20b62a39f90 0 1450088598716 7 connected
040d4bb6a00569fc44eec05440a5fe0796952ccf 192.168.56.101:6379 myself,slave 5318e48e9ef0fc68d2dc723a336b791fc43e23c8 0 0 4 connected
caf2cec45114dc8f4cbc6d96c6dbb20b62a39f90 192.168.56.104:6379 master - 0 1450088599720 7 connected 0-10922
d78293d0821de3ab3d2bca82b24525e976e7ab63 192.168.56.106:6379 slave 5318e48e9ef0fc68d2dc723a336b791fc43e23c8 0 1450088599316 8 connected
5318e48e9ef0fc68d2dc723a336b791fc43e23c8 192.168.56.103:6379 master - 0 1450088599218 8 connected 10923-16383
My problem is that when I shutdown or stop redis-server on any one machine which is master the whole cluster goes down, but if all the three slaves die the cluster still works properly.
What should I do so that a slave turns a master if a master fails(Fault tolerance)?
I am under the assumption that redis handles all those things and I need not worry about it after deploying the cluster. Am I right or would I have to do thing myself?
Another question is lets say I have six machine of 16GB RAM. How much total data I would be able to handle on this Redis cluster with three masters and three slaves?
Thank you.
the setting cluster-slave-validity-factor 0 may be the culprit here.
from redis.conf
# A slave of a failing master will avoid to start a failover if its data
# looks too old.
In your setup the slave of the terminated master considers itself unfit to be elected master since the time it last contacted master is greater than the computed value of:
(node-timeout * slave-validity-factor) + repl-ping-slave-period
Therefore, even with a redundant slave, the cluster state is changed to DOWN and becomes unavailable.
You can try with a different value, example, the suggested default
cluster-slave-validity-factor 10
This will ensure that the cluster is able to tolerate one random redis instance failure. (it can be slave or a master instance)
For your second question: Six machines of 16GB RAM each will be able to function as a Redis Cluster of 3 Master instances and 3 Slave instances. So theoretical maximum is 16GB x 3 data. Such a cluster can tolerate a maximum of ONE node failure if cluster-require-full-coverage is turned on. else it may be able to still serve data in the shards that are still available in the functioning instances.
I have a rabbitmq cluster with 2 nodes. Node A and B. Node A is up and running. Everytime I run the following commadn on node A I get:
./rabbitmqctl cluster_status
Cluster status of node rabbit#A ...
[{nodes,[{disc,[rabbit#A,rabbit#B]}]},
{running_nodes,[rabbit#A]},
{partitions,[]}]
...done.
Interestingly node B is up and running. Everytime I have it join the other node (A) to get it clustered it states:
rabbitmqctl join_cluster rabbit#A
...done (already_member).
rabbitmqctl cluster_status
Cluster status of node rabbit#B ...
[{nodes,[{disc,[rabbit#B]}]}]
...done.
So somehow node A cannot see B. And on be the "already_member" does not seem to be reflected from the cluster_status command...
I can check the queues on both nodes and they are different. Node A has dozens of queues and node B none, therefore it is clear the cluster is not established. Both node A and B can ping each other and nothing gets reported in the rabbitmq's logs
Any idea how this is not working ?
In case of cluster I will suggest you to go for Load Balancer . Make sure you already set HA Policy for your cluster.
To set HA Policy
$ rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'
More here RabbitMQ Cluster
I was able to solve this same problem. Given NodeA is the parent and NodeB is trying to join the cluster.
Stop NodeB app rabbitmqctl stop_app
On NodeA forget cluster node rabbitmqctl forget_cluster_node rabbit#NodeB
Reset NodeB rabbitmqctl reset
Join NodeB to the cluster rabbitmqctl join_cluster rabbit#NodeA
Start NodeB rabbitmqctl start_app
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 7 years ago.
Improve this question
Is it possible to create a redis cluster with 2 nodes , one acting as a master and other one as slave.
I get the following error if I try with 2 nodes (one as master and other as slave)
>>> Creating cluster
Connecting to node 127.0.0.1:6379: OK
Connecting to node 192.168.40.159:6379: OK
*** ERROR: Invalid configuration for cluster creation.
*** Redis Cluster requires at least 3 master nodes.
*** This is not possible with 2 nodes and 1 replicas per node.
*** At least 6 nodes are required.
Yes. The requirement of at least 3 master nodes is set by the ruby script, but not a hard limit in cluster.
The first thing you need to do is send a cluster command with 16385 arguments like
cluster addslots 0 1 2 3 ... 16384
to the cluster. Since there is too many arguments to manually type them in a redis-cli, I suggest write a program to do that, in which you open a TCP socket connecting to the redis node, convert the previous command into a redis command string and write it to the socket.
The single node cluster will be online after few seconds you send the command. Then connect to the other node with redis-cli, type the following command to make it a slave
cluster meet MASTER_HOST MASTER_PORT
cluster replicate MASTER_ID
where MASTER_HOST:MASTER_PORT is the address of the previous node, and MASTER_ID is the ID of that node, which you could retrieve it via a cluster nodes command.
For convenience I've written a python tool for those kinds of redis cluster management, you could install it with
pip install redis-trib
For more detail please go to https://github.com/HunanTV/redis-trib.py/
Redis-Cluster is not a fit for your use case.
For your use case, you need to configure one server (the master), then configure a second server and add the "slaveof" directive - pointing it to the master. How you handle failover is up to your scenario but I would recommend the use of redis-sentinel.
For a more detailed walkthrough, see the Redis Replication page
Nope, it is not possible to create a redis cluster with 1 master node, as suggested here setting up a redis cluster requires atleast 3 master nodes.
'redis-trib.rb create' command requires at least 3 nodes.
this way can make 1 master, 1 slave redis cluster.
Using redis-trib
$ redis-server 5001/redis.conf
$ redis-trib.rb fix 127.0.0.1:5001
so many messages ...
$ redis-server 5002/redis.conf
$ redis-trib.rb add-node --slave 127.0.0.1:5002 127.0.0.1:5001
>>> Adding node 127.0.0.1:5002 to cluster 127.0.0.1:5001<br>
Connecting to node 127.0.0.1:5001: OK<br>
>>> Performing Cluster Check (using node 127.0.0.1:5001)<br>
M: 015bec64d631990b83ad63736d906cda257a762c 127.0.0.1:5001<br>
slots:0-16383 (16384 slots) master<br>
0 additional replica(s)<br>
[OK] All nodes agree about slots configuration.<br>
>>> Check for open slots...<br>
>>> Check slots coverage...<br>
[OK] All 16384 slots covered.<br>
Automatically selected master 127.0.0.1:5001<br>
Connecting to node 127.0.0.1:5002: OK<br>
>>> Send CLUSTER MEET to node 127.0.0.1:5002 to make it join the cluster.<br>
Waiting for the cluster to join.<br>
>>> Configure node as replica of 127.0.0.1:5001.<br>
[OK] New node added correctly.
Using cluster commands
$ redis-server 5001/redis.conf
Using Ruby : addslots
$ echo '(0..16383).each{|x| puts "cluster addslots "+x.to_s}' | ruby | redis-cli -c -p 5001 > /dev/null
$ redis-server 5002/redis.conf
$ redis-cli -c -p 5002
127.0.0.1:5002> cluster meet 127.0.0.1 5001
OK
127.0.0.1:5002> cluster replicate 7c38d2e5e76fc4857fe238e34b4096fc9c9f12a5
node-id of 5001
OK