Redis cluster is created with internal ip addresses.
M: 24ff344338e4abb4f7f2e888ee9d57843fc46e62 10.0.9.19:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 9aa005c2d914db12a394af0cb8a0d8e218730099 10.0.9.15:6379
slots: (0 slots) slave
replicates f67bbf56d98c2bff9eba343356e2b52bd5e59b12
S: aedd33304e59cbe7091fb36befdb230f3956f03e 10.0.9.16:6379
slots: (0 slots) slave
replicates 9cd64e70f9fd7fffb44b79186e09e3872ea3ebb4
M: f67bbf56d98c2bff9eba343356e2b52bd5e59b12 10.0.9.20:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 2af640e7072b255786b47337e7ca171e0506f5f9 10.0.9.14:6379
slots: (0 slots) slave
replicates 24ff344338e4abb4f7f2e888ee9d57843fc46e62
M: 9cd64e70f9fd7fffb44b79186e09e3872ea3ebb4 10.0.9.18:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
There is only one public ip for external connection, 54.174.xxx.xxx.
After connecting to the cluster, there is problem for redirecting.
52.71.xxx.xxx:6379> lrange mylist 0 -1
-> Redirected to slot [5282] located at 10.0.9.18:6379
Could not connect to Redis at 10.0.9.18:6379: Connection timed out
Could not connect to Redis at 10.0.9.18:6379: Connection timed out
(254.54s)
not connected>
It look like redis is asking the client to connect to another node for the data. Is it possible to let the current connected node to fetch data on behave? or Any another solution?
As least, I know Cassandra will fetch data from another node to return instead of redirecting.
Related
<redisson.version>3.16.3</redisson.version>
readMode: SLAVE
Not sure why am I seeing this error.
caused by: org.redisson.client.RedisException: ERR WAIT cannot be used with slave instances. Please also note that since Redis 4.0 if a slave is configured to be writable (which is not the default) writes to slaves are just local and are not propagated.. channel: [id: 0x90c58500, L:/172.16.189.108:43192 - R:10.112.94.132/10.112.94.132:15003] command: (WAIT), promise: RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise#35eeb35e(incomplete)], params: [1, 1000]
at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:370)
at org.redisson.client.handler.CommandDecoder.decodeCommandBatch(CommandDecoder.java:271)
at org.redisson.client.handler.CommandDecoder.decodeCommand(CommandDecoder.java:210)
at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:137)
at org.redisson.client.handler.CommandDecoder.decode(CommandDecoder.java:113)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
I have a Redis v3.0.2 Master and Slave
on the slave I issue slaveof (masterIP) 6379
Sync happens, logs looking fine on master and slave. Key counts look sane.
After the sync completes and the slave loads the database, No more operations happen.
running monitor on the master give me hundreds of sets / sec.
the slave only sees a few deletes and an occasional PING
Slave Log:
2734:S 16 Aug 07:23:29.460 * MASTER <-> SLAVE sync: Loading DB in memory
2734:S 16 Aug 07:25:16.531 * MASTER <-> SLAVE sync: Finished with success
Slave Monitor:
~
[119](root#[slave])[0]:\> redis-cli
127.0.0.1:6379> monitor
OK
1534405063.907020 [0 [master]:6379] "PING"
1534405065.409863 [0 [master]:6379] "DEL" "pmlock12"
1534405065.709784 [0 [master]:6379] "DEL" "pmlock22"
1534405065.909400 [0 [master]:6379] "DEL" "pmlock27"
Master Log
2951:C 16 Aug 07:20:57.908 * RDB: 279 MB of memory used by copy-on-write
2745:M 16 Aug 07:20:58.297 * Background saving terminated with success
2745:M 16 Aug 07:22:59.369 * Synchronization with slave 10.168.230.15:6379 succeeded
Master Monitor:
1534405287.136316 [0 [src]:54660] "SET" "CMP36" "{\"m_cur...
1534405252.002731 [0 [src]:45742] "SET" "PM14" "H4sIAAAAAAAAAO1cW4...
Master Info
[209](root#master)[0]:\> redis-cli info replication
# Replication
role:master
connected_slaves:1
slave0:ip=[slave],port=6379,state=online,offset=1747897005,lag=0
master_repl_offset:1748304094
repl_backlog_active:1
repl_backlog_size:104857600
repl_backlog_first_byte_offset:1643446495
repl_backlog_histlen:104857600
I've rebooted the master and slave, i just can't get the master to send through anything but ping and delete. I'm not well versed on redis so i'm sure i'm just missing something.
If I configured activemq.xml as follows, i can't connect 123.57.167.211:8161. why ?
replicatedLevelDB directory="${activemq.data}/leveldb"
replicas="3" bind="tcp://0.0.0.0:0"
zkAddress="10.172.5.40:2181,10.170.253.39:2181,10.165.112.136:2181"
zkPassword="password"
zkPath="/activemq40/leveldb-stores"
sync="local_disk"
hostname="123.57.167.211"
when replicas="3", you need to start at least 2 activemqs for the cluster to run.
One of them will be the master, you can connect only to the master's 8161, as only the master is actually running.
I want setup redis cluster with 6 nodes (node1, node2, node3, node4, node5, node6), which has 3 masters and 3 slaves. Each node has this configuration file
redis.conf
port 6379
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 10000
appendonly yes
I get error when create the cluster. Create command:
redis-trib.rb create --replicas 1 node1:6379 node2:6379 node3:6379 node4:6379 node5:6379 node6:6379
Error:
>>> Creating cluster
Connecting to node node1:6379: OK
Connecting to node node2:6379: OK
Connecting to node node3:6379: OK
Connecting to node node4:6379: OK
Connecting to node node5:6379: OK
Connecting to node node6:6379: OK
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
node6:6379
node5:6379
node4:6379
Adding replica node3:6379 to node6:6379
Adding replica node2:6379 to node5:6379
Adding replica node1:6379 to node4:6379
S: 1f13819038ba983bb8355f54cb8cec19d2b29e01 node1:6379
replicates 534745088c8b403b81d7e48a22d2e317fb420a38
S: 711461862393664b46d73db6561631f40de29561 node2:6379
replicates f503fe6fd52c73e446267795111ae6ea95495829
S: 204fa4e23b08e2c6ad80b0aca271fc380bc6885d node3:6379
replicates fe6a8e88afdb2796c09fcc873b37ba90c2ba6d79
M: 534745088c8b403b81d7e48a22d2e317fb420a38 node4:6379
slots:10923-16383 (5461 slots) master
M: f503fe6fd52c73e446267795111ae6ea95495829 node5:6379
slots:5461-10922 (5462 slots) master
M: fe6a8e88afdb2796c09fcc873b37ba90c2ba6d79 node6:6379
slots:0-5460,6918 (5462 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
/var/lib/gems/1.8/gems/redis-3.2.2/lib/redis/client.rb:114:in `call': ERR Slot 16011 is already busy (Redis::CommandError)
from /var/lib/gems/1.8/gems/redis-3.2.2/lib/redis.rb:2646:in `method_missing'
from /var/lib/gems/1.8/gems/redis-3.2.2/lib/redis.rb:57:in `synchronize'
from /usr/lib/ruby/1.8/monitor.rb:242:in `mon_synchronize'
from /var/lib/gems/1.8/gems/redis-3.2.2/lib/redis.rb:57:in `synchronize'
from /var/lib/gems/1.8/gems/redis-3.2.2/lib/redis.rb:2645:in `method_missing'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:205:in `flush_node_config'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:667:in `flush_nodes_config'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:666:in `each'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:666:in `flush_nodes_config'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:1007:in `create_cluster_cmd'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:1388:in `send'
from /home/hadoop/projects/ramin/redis-3.0.5/src/redis-trib.rb:1388
I also did these, but also got same error message
use ip address instead of hostname
remove nodes.conf in each nodes
How said #thepirat000 (in all nodes did FLUSHALL and then CLUSTER RESET SOFT) i also changed hostname to ip address
I have 2 servers and 3 instances of redis3 in each of them. I have a cluster-nodes directory, where I have all the data of each instance. Here it is.
cluster-nodes/
|-- 7777
| |-- db01
| | -- nodes-7777.conf
| -- redis.conf
|-- 7778
| |-- db02
| | -- nodes-7778.conf
| -- redis.conf
-- 7779
|-- db03
| -- nodes-7779.conf
-- redis.conf
Here is my config file redis.conf under the 7777 directory
pidfile /var/run/redis/redis-7777.pid
port 7777
dir /opt/redis/cluster-nodes/7777/db01/
cluster-enabled yes
cluster-config-file nodes-7777.conf
cluster-node-timeout 15000
When I try to start redis I get
./redis-trib.rb create --replicas 1 127.0.0.1:7777 127.0.0.1:7778 127.0.0.1:7779 192.168.56.41:7777 192.168.56.41:7778 192.168.56.41:7779
>>> Creating cluster
Connecting to node 127.0.0.1:7777: OK
Connecting to node 127.0.0.1:7778: OK
Connecting to node 127.0.0.1:7779: OK
Connecting to node 192.168.56.41:7777: OK
Connecting to node 192.168.56.41:7778: OK
Connecting to node 192.168.56.41:7779: OK
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7777
192.168.56.41:7777
127.0.0.1:7778
Adding replica 192.168.56.41:7778 to 127.0.0.1:7777
Adding replica 127.0.0.1:7779 to 192.168.56.41:7777
Adding replica 192.168.56.41:7779 to 127.0.0.1:7778
M: 209d68fae9c64855d34972f660232eb96370a669 127.0.0.1:7777
slots:0-5460 (5461 slots) master
M: 62e2b167a287b94b5154f7b9b0f226345baa81b7 127.0.0.1:7778
slots:10923-16383 (5461 slots) master
S: 36ed59deceb01788db76abc0c2f22925a27295fc 127.0.0.1:7779
replicates 2760b5fcc99c6563a7cf8deea159efb012309238
M: 2760b5fcc99c6563a7cf8deea159efb012309238 192.168.56.41:7777
slots:5461-10922 (5462 slots) master
S: 16bf95ba9cb743c2a3caecaab5c2fd5121d80557 192.168.56.41:7778
replicates 209d68fae9c64855d34972f660232eb96370a669
S: 30e7a5b4a94b5ff3a09f4809d6fd62edb2279b0e 192.168.56.41:7779
replicates 62e2b167a287b94b5154f7b9b0f226345baa81b7
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.....................................................................................................................................................................................................................................................................^C./redis-trib.rb:534:in `sleep': Interrupt
from ./redis-trib.rb:534:in `wait_cluster_join'
from ./redis-trib.rb:1007:in `create_cluster_cmd'
from ./redis-trib.rb:1373:in `<main>'
Here is the output from cluster nodes on the first server
62e2b167a287b94b5154f7b9b0f226345baa81b7 127.0.0.1:7778 master - 0 1435144555558 2 connected 10923-16383
36ed59deceb01788db76abc0c2f22925a27295fc 127.0.0.1:7779 master - 0 1435144554554 3 connected
209d68fae9c64855d34972f660232eb96370a669 127.0.0.1:7777 myself,master - 0 0 1 connected 0-5460
And this is from the second
16bf95ba9cb743c2a3caecaab5c2fd5121d80557 127.0.0.1:7778 master - 0 1435144648065 5 connected
30e7a5b4a94b5ff3a09f4809d6fd62edb2279b0e 127.0.0.1:7779 master - 0 1435144647057 6 connected
2760b5fcc99c6563a7cf8deea159efb012309238 127.0.0.1:7777 myself,master - 0 0 4 connected 5461-10922
It seems that all of them are started as masters? Is there something wrong in my configs?
Thank you.
p.s. when I try the same configs and start all instances in one server, everything works fine.
The problem in my case was that I was starting the service with localhost address,
./redis-trib.rb create --replicas 1 127.0.0.1:7777 127.0.0.1:7778 127.0.0.1:7779 192.168.56.41:7777 192.168.56.41:7778 192.168.56.41:7779
In order to fix that 127.0.0.1 should be substituted with ip address of the local node, i.e.
./redis-trib.rb create --replicas 1 192.168.56.40:7777 192.168.56.40:7778 192.168.56.40:7779 192.168.56.41:7777 192.168.56.41:7778 192.168.56.41:7779
Please check the 17777 17778...port，cluster need those port to communication。