I Create a redis cluster:
M: dbaf0a596c4f5a2f4ac1d9de2ed5c117f201d26d localhost:9501
slots:5461-10922 (5462 slots) master
2 additional replica(s)
S: a375ec8221f9872f8f287bca8a67fcef701cef72 localhost:9000
slots: (0 slots) slave
replicates dbaf0a596c4f5a2f4ac1d9de2ed5c117f201d26d
S: a319a75ac0ee58484e345d7996a6712335ebddf5 localhost:9001
slots: (0 slots) slave
replicates dbaf0a596c4f5a2f4ac1d9de2ed5c117f201d26d
M: 05f7d7cc87b417ad4bdf61b3877a674d54c02ef7 localhost:9002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 932dfde44c77d341b873bf205c3d069b236c424a localhost:9502
slots: (0 slots) slave
replicates 05f7d7cc87b417ad4bdf61b3877a674d54c02ef7
M: 69ec05cbf01c81474ea1ccf8848d3f336c99200b localhost:9500
slots:0-5460 (5461 slots) master
0 additional replica(s)
//cluster keyslot b (in slot 3300)
//cluster keyslot g581 (in slot 3300)
redis-cli -h localhost -p 9500 -c
1. localhost:9500> set b 1 //b store in 9500 node
2. localhost:9500> cluter setslot 3300 migrating 9002_NODE_ID
3. localhost:9500> set g581 2 //(error) ASK 3300 localhost:9002
redis-cli -h localhost -p 9002 -c
localhost:9002> set g581 2 //Redirected to slot [3300] located at localhost:9500`
now Can I write data into slot 3300
When you want to transfer redis slot in a cluster, steps below are requried:
use two "cluster setslot" commands to set source node migrating and destination node importing
CLUSTER SETSLOT IMPORTING
CLUSTER SETSLOT MIGRATING
use CLUSTER GETKEYSINSLOT to get all keys in this slot, and for each slot, use MIGRATE to transfer data.
use CLUSTER SETSLOT <slot> NODE <node_id> to transfer the slot to new node officially.
So it's not a good option to handle redis slot yourself, better use redis-trib to do the managing.
Related
I have 3 EC2 Instances with Redis Running like this:
Server 001: 10.0.1.203, Port: 6379
Server 002: 10.0.1.202, Port: 6380
Server 003: 10.0.1.190, Port: 6381
Config file for each one:
# bind 127.0.0.1
protected-mode no
port PORT
pidfile /var/run/redis_PORT.pid
cluster-enabled yes
cluster-config-file nodes-PORT.conf
cluster-node-timeout 15000
I can connect via redis to each one on each server.
But when I run the cluster creation the script never ends on Server 001.
root#ip-10-0-1-203:~/redis-stable# redis-cli --cluster create 10.0.1.203:6379 10.0.1.202:6380 10.0.1.190:6381
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: 4c0b7609e5d906ff58d67ab446bbd9e20833e0db 10.0.1.203:6379
slots:[0-5460] (5461 slots) master
M: a5dbd72815a1875b58a0cc0fd6a52dc0b76735b7 10.0.1.202:6380
slots:[5461-10922] (5462 slots) master
M: 14d39c0876a982cadd50f301a3d35715171279c0 10.0.1.190:6381
slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
....................................................................................................................................................................................................................................................................................................................................
Server 002 logs:
44119:M 02 Nov 2020 13:30:03.477 * Ready to accept connections
44119:M 02 Nov 2020 13:30:45.362 # configEpoch set to 0 via CLUSTER RESET HARD
44119:M 02 Nov 2020 13:30:45.362 * Node hard reset, now I'm a5dbd72815a1875b58a0cc0fd6a52dc0b76735b7
44119:M 02 Nov 2020 13:30:59.352 # configEpoch set to 2 via CLUSTER SET-CONFIG-EPOCH
Server 003 logs:
44033:M 02 Nov 2020 13:30:50.695 # configEpoch set to 0 via CLUSTER RESET HARD
44033:M 02 Nov 2020 13:30:50.695 * Node hard reset, now I'm 14d39c0876a982cadd50f301a3d35715171279c0
44033:M 02 Nov 2020 13:30:59.346 # configEpoch set to 3 via CLUSTER SET-CONFIG-EPOCH
Am missing something on the configuration?
Probably the Redis Cluster port is not accessible on the EC2 instances.
From the Redis Cluster Specification:
Every Redis Cluster node has an additional TCP port for receiving incoming connections from other Redis Cluster nodes. This port is at a fixed offset from the normal TCP port used to receive incoming connections from clients. To obtain the Redis Cluster port, 10000 should be added to the normal commands port. For example, if a Redis node is listening for client connections on port 6379, the Cluster bus port 16379 will also be opened.
I have Redis cluster with 6 nodes (3 masters, 3 slaves). I trying to add additional master and slave:
redis-cli --cluster add-node 100.96.1.11:6379 100.96.2.14:6379
Ive got
redis-cli --cluster add-node 100.96.1.11:6379 100.96.2.14:6379
>>> Adding node 100.96.1.11:6379 to cluster 100.96.2.14:6379
>>> Performing Cluster Check (using node 100.96.2.14:6379)
M: 9d91c8d60c4c644a7846829f91f1af8563bf99f9 100.96.2.14:6379
slots: (0 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.
Meanwhile check cluster is ok:
redis-cli --cluster check 100.96.1.5:6379
100.96.1.5:6379 (d475435a...) -> 36265 keys | 5462 slots | 1 slaves.
100.96.1.7:6379 (fd44a251...) -> 36205 keys | 5461 slots | 1 slaves.
100.96.2.13:6379 (00cc3a8b...) -> 36069 keys | 5461 slots | 1 slaves.
[OK] 108539 keys in 3 masters.
6.62 keys per slot on average.
>>> Performing Cluster Check (using node 100.96.1.5:6379)
M: d475435a6e70a5603f046f743f7774e6eead6afe 100.96.1.5:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 2c66b6ab6ebacd6054371bdc5d39aa08c48f76b5 100.96.1.10:6379
slots: (0 slots) slave
replicates 00cc3a8ba8cae1b539e172be01333b6f5fb2631a
M: fd44a251ce75c92049d1a5c748a8e52c2b0763e0 100.96.1.7:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: 00cc3a8ba8cae1b539e172be01333b6f5fb2631a 100.96.2.13:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: edcacf3cd1de6c5d9a3108211c7b84676b2cddca 100.96.2.9:6379
slots: (0 slots) slave
replicates fd44a251ce75c92049d1a5c748a8e52c2b0763e0
S: e6cea582a6d6a9c726d36d2899c9fbf4a7e08909 100.96.2.6:6379
slots: (0 slots) slave
replicates d475435a6e70a5603f046f743f7774e6eead6afe
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
What's wrong?
To add a new node to the Redis cluster we should use command:
redis-cli --cluster add-node <new-node-ip:port> <existing-node-ip:port>
In my case there are two IPs of new nodes.
So, if we need to add two new nodes as Master and Slave, we should run this command twice, but use --cluster-slave flag for the slave:
redis-cli --cluster add-node <new-node-ip:port> <existing-node-ip:port>
redis-cli --cluster add-node <new-node-ip:port> <existing-node-ip:port> --cluster-slave
After that we should reshard Redis cluster:
redis-cli --cluster reshard <existing-node-ip:port>
I am trying to connect a cluster of 8 replicas on one address to an existing cluster on another address.
The replica servers are all running in cluster mode.
When I try to do either:
./redis-trib.rb add-node --slave REPLICA_IP:6380 MASTER_IP:6380
or
./redis-cli --cluster add-node REPLICA_IP:6380 MASTER_IP:6380 --cluster-slave
I get the same result;
Waiting for the cluster to join...........................
which hangs indefinitely.
The two servers can definitely see each other and I can connect to any relevant redis-node (replica or master) from either server. The discovery/communion ports (16830, etc.) are all open and contactable as well. The output of these commands also suggests that the cluster has been found as it shows each of the nodes and their correct node ids.
here is the full output of either add-node command:
>>> Adding node REPLICA_IP:6380 to cluster MASTER_IP:6380
>>> Performing Cluster Check (using node MASTER_IP:6380)
M: 043a5fa4fdca929d3d87f953906dc7c1f030926c MASTER_IP:6380
slots:[0-2047] (2048 slots) master
M: e104777d31630eef11a01e41c7d3a6c98e14ab64 MASTER_IP:6386
slots:[12288-14335] (2048 slots) master
M: 9c807d6f57a9634adcdf75fa1943c32c985bda1c MASTER_IP:6384
slots:[8192-10239] (2048 slots) master
M: 0f7ec07deff97ca23fe67109da2365d916ff1a67 MASTER_IP:6383
slots:[6144-8191] (2048 slots) master
M: 974e8b4051b7a8e33db62ba7ad62c7e54abe699d MASTER_IP:6382
slots:[4096-6143] (2048 slots) master
M: b647bb9d732ff2ee83b097ffb8b49fb2bccd366f MASTER_IP:6387
slots:[14336-16383] (2048 slots) master
M: a86ac1d5e783bed133b153e471fdd970c17c6af5 MASTER_IP:6381
slots:[2048-4095] (2048 slots) master
M: 6f859b03f86eded0188ba493063c5c2114d7c11f MASTER_IP:6385
slots:[10240-12287] (2048 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master MASTER_IP:6380
>>> Send CLUSTER MEET to node REPLICA_IP:6380 to make it join the cluster.
Waiting for the cluster to join
............................
If I run CLUSTER MEET manually and then CLUSTER NODES I can temporarily see another node in state 'handshake' with status 'disconnected' and then it disappears. It shows a node id that is not the same as it actually is.
I figured it out:
Using tcpdump I confirmed that both servers were talking to each other on both the redis server ports and the handshake ports repeatedly while the add-slave command hung forever.
but in the redis configs for each node I had:
bind 0.0.0.0
but on both the masters and the replicas the config must read:
bind SERVER_IP
in order for CLUSTER MEET to work properly.
In my case each node had same MMID so it was waiting forever.
What I was doing, I configured an EC2 ami, and launched 3 servers from
AMI and using user-data I did reconfigured the redis cluster via shell
script and restarted the server, each server got same ID as from which
server I had created AMI.
M: b29aff425cdfa94272cdce1816939a9692c71e12 10.0.134.109:6379
slots:[0-5460] (5461 slots) master
M: b29aff425cdfa94272cdce1816939a9692c71e12 10.0.175.235:6379
slots:[5461-10922] (5462 slots) master
M: b29aff425cdfa94272cdce1816939a9692c71e12 10.0.155.10:6379
slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
So on each node I did CLUSTER RESET HARD; it works
https://redis.io/commands/cluster-reset
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: 36a129fab85d2aed310bfd7cc141035de420fa92 10.0.134.109:6379
slots:[0-5460] (5461 slots) master
M: 773bc76e903da27efbd965bca26366fa20878397 10.0.175.235:6379
slots:[5461-10922] (5462 slots) master
M: 10a79173d1f7a9c568bdfa3b955b6e133d2dceaa 10.0.155.10:6379
slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 10.0.134.109:6379)
M: 36a129fab85d2aed310bfd7cc141035de420fa92 10.0.134.109:6379
slots:[0-5460] (5461 slots) master
M: 773bc76e903da27efbd965bca26366fa20878397 10.0.175.235:6379
slots:[5461-10922] (5462 slots) master
M: 10a79173d1f7a9c568bdfa3b955b6e133d2dceaa 10.0.155.10:6379
slots:[10923-16383] (5461 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
If there is no firewall problem between the nodes, you may check bind setting in redis.conf.
You should bind the redis service on LAN IP, of course, but one more thing:
Delete 127.0.0.1 or move 127.0.0.1 to the end after LAN IP!
Just like this: bind 10.2.1.x 127.0.0.1 or bind 10.2.1.x
source
I have setup up Redis master slave configuration having one master (6379 port) and 3 slaves (6380,6381,6382) running in the same machine. Looks like cluster is setup properly as I can see the following output on running info command:
# Replication
role:master
connected_slaves:3
slave0:ip=127.0.0.1,port=6380,state=online,offset=29,lag=1
slave1:ip=127.0.0.1,port=6381,state=online,offset=29,lag=1
slave2:ip=127.0.0.1,port=6382,state=online,offset=29,lag=1
master_repl_offset:43
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:42
But wherever I try to add new key in master, I get the following error:
(error) CLUSTERDOWN Hash slot not served
Using redis-3.0.7 in Mac OS X Yosemite.
I had the same issue, turned out I forgot to run the create cluster:
cd /path/to/utils/create-cluster
./create-cluster create
http://redis.io/topics/cluster-tutorial#creating-a-redis-cluster-using-the-create-cluster-script
To fix slots issue while insertion:
redis-cli --cluster fix localhost:6379
You can use ruby script buddled with redis for creating clusters as mentioned below :
/usr/local/redis-3.2.11/src/redis-trib.rb create --replicas 1 192.168.142.128:7001 192.168.142.128:7002 192.168.142.128:7003 192.168.142.128:7004 192.168.142.128:7005 192.168.142.128:7006
There is a hash slot not allotted to any master. Check the hash slots by looking at the column 9 in the output of following command (column 9 will be empty if no hash slots for that node):
redis-cli -h masterIP -p masterPORT CLUSTER NODES
The hash slots can be allotted by using the following command.
redis-cli -h masterIP -p masterPORT CLUSTER ADDSLOTS SLOTNNUMBER
But this has to be done for every slot number individually without missing. Use a bat script to include it in for loop. something like,
for /L %a in (0,1,5400) Do redis-cli -h 127.0.0.1 -p 7001 cluster addslots %a
Also, this command works before assigning slaves to master. After this ADDSLOTS step and completing the setup, the SET and GET worked fine. Remember to use -c along with redis-cli before SET to enable cluster support.
The issue comes when one or more Redis nodes gets corrupted and can no longer serve its configured hash slots.
You will have to bootstrap the cluster again to make sure the nodes agree on the hash slots to serve.
If the Redis nodes contain data or a key in the database 0, you will have to clear this data before rerunning the bootstrap.
Given a redis cluster with six nodes (3M/3S) on ports 7000-7005 with master nodes on ports 7000-7002 and slave nodes on the rest, master node 7000 is shut down, so node 7003 becomes the new master:
$ redis-cli -p 7003 cluster nodes
2a23385e94f8a27e54ac3b89ed3cabe394826111 127.0.0.1:7004 slave 1108ef4cf01ace085b6d0f8fd5ce5021db86bdc7 0 1452648964358 5 connected
5799de96ff71e9e49fd58691ce4b42c07d2a0ede 127.0.0.1:7000 master,fail - 1452648178668 1452648177319 1 disconnected
dad18a1628ded44369c924786f3c920fc83b59c6 127.0.0.1:7002 master - 0 1452648964881 3 connected 10923-16383
dfcb7b6cd920c074cafee643d2c631b3c81402a5 127.0.0.1:7003 myself,master - 0 0 7 connected 0-5460
1108ef4cf01ace085b6d0f8fd5ce5021db86bdc7 127.0.0.1:7001 master - 0 1452648965403 2 connected 5461-10922
bf60041a282929cf94a4c9eaa203a381ff6ffc33 127.0.0.1:7005 slave dad18a1628ded44369c924786f3c920fc83b59c6 0 1452648965926 6 connected
How does one go about [automatically] reconnecting/restarting node 7000 as a slave instance of 7003?
Redis Cluster: Re-adding a failed over node has detail explanation about what happens.
Basically, the node will become a slave of the slave (which is now a master) that replaced it during the failover.
Have you seen the Redis Sentinel Documentation?
Redis Sentinel provides high availability for Redis. In practical
terms this means that using Sentinel you can create a Redis deployment
that resists without human intervention to certain kind of failures.