I want to create an active-active replication for keydb, I used the official docs https://docs.keydb.dev/docs/active-rep/ , however, I'm not getting the expected results, even though I'm not getting errors.
Config A:
port 6379
requirepass mypassword123
masterauth mypassword123
active-replica yes
replicaof 10.0.11.205 6379
Config B:
port 6379
requirepass mypassword123
masterauth mypassword123
active-replica yes
replicaof 10.0.11.208 6379
Related
I have Master-Slave(3 nodes) setup using Redis Sentinel but when I try to do failover, I am seeing the below error
127.0.0.1:26379> sentinel failover mymaster
(error) NOGOODSLAVE No suitable replica to promote
Below is the Redis server configurations for master and slave nodes
#redis.conf of master node
bind 127.0.0.1 192.26.x.1
protected-mode no
daemonize yes
logfile /opt/softwares/redis-6.0.16/log/redis-server.log
#redis.conf of slave node 1
bind 127.0.0.1 192.26.x.2
protected-mode no
daemonize yes
logfile /opt/softwares/redis-6.0.16/log/redis-server.log
replicaof 192.26.x.1 6379
#redis.conf of slave node 2
bind 127.0.0.1 192.26.x.3
protected-mode no
daemonize yes
logfile /opt/softwares/redis-6.0.16/log/redis-server.log
replicaof 192.26.x.1 6379
Below is my sentinel nodes config
# Same for all sentinel nodes
bind 127.0.0.1 192.26.x.1
protected-mode no
port 26379
daemonize yes
pidfile "/var/run/redis-sentinel.pid"
logfile "/var/log/redis-sentinel.log"
sentinel monitor mymaster 192.26.x.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
When I queried the slave's status, I can see master-related config are not defined.
127.0.0.1:26379> SENTINEL masters
31) "master-link-down-time"
32) "0"
33) "master-link-status"
34) "err"
35) "master-host"
36) "?"
37) "master-port"
38) "0"
I am starting the Redis-servers in all nodes then Redis-sentinel. Not sure the ordering of starting nodes matters.
Please let me know whether I am missing some configurations or doing something wrong. The Redis version that I am using is 6.0.16.
Thanks in advance.
Changing the order of IP addresses in a bind in all configuration files solved the issue i.e.
bind <public_ip> <localhost>
I setup a cluster of 3 redis-sentinel (3.2.6-1) on three instance of redis-server (3.2.6-1).
I checked the firewall for the 6379 and 26379 TCP port and it's all good.
The configuration for my redis-sentinel is something like that:
port 26379
dir "/tmp"
sentinel myid 0559ec26112bebce70bbfa5849f77338453315b
sentinel monitor rback 10.3.0.43 6379 2
sentinel down-after-milliseconds rback 5000
sentinel failover-timeout rback 10000
daemonize yes
pidfile "/var/run/redis/redis-sentinel.pid"
loglevel notice
logfile "/var/log/redis/redis-sentinel.log"
When I start the redis-server and redis-sentinel instances, I can query on the port 26379 port sentinel master rback and see the options:
9) "flags"
10) "master"
...
31) "num-slaves"
32) "2"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
In the logs of the redis-sentinel, I see this:
26851:X 12 Jun 15:22:35.092 * +sentinel sentinel 4b22b6ff1b983432028f8cdb0db75cd553bec4b3 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:40.105 * +sentinel sentinel 8fc263bf82226364917478541c13f2c7f5b746e6 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:40.168 # +sdown sentinel 4b22b6ff1b983432028f8cdb0db75cd553bec4b3 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:45.120 # +sdown sentinel 8fc263bf82226364917478541c13f2c7f5b746e6 XXXXX 26379 # redis-back XXXXX 6379
And if I run the sleep command or crash the master redis, I see each sentinel logging a +sdown command, but never promote it to +odown and promoting a new master.
How can I debug this?
Thanks
Add Information:
I run a tcpdump and analyse the traffic with wireshark, and found out that the sentinel is connecting to the other sentinel and try to communicate with it, but receive a "DENIED Redis is running in protected mode...". Even though the redis-servers are not running in protected mode.
The problem is the communication between the sentinel.
Redis adds with 3.2 version a "protected-mode" configuration flag on the sentinel.conf too.
The sentinel will receive an error message "Denied Redis is running in protected mode..." if the sentinel doesn't have the flag.
I found this information here:
https://newbiedba.wordpress.com/2016/07/01/redis-3-2-sentinel-with-protected-mode/
I'm testing redis failover with this simple setup:
3 Ubuntu server 16.04
redis and redis-sentinel are configured on each box.
Master ip : 192.168.0.18
Resque ip : 192.168.0.16
Resque2 ip : 192.168.0.13
Data replication works well but I can't get failover to work.
When I start redis-sentinel I always get a +sdown message after 60 seconds:
14913:X 17 Jul 10:40:03.505 # +monitor master mymaster 192.168.0.18 6379 quorum 2
14913:X 17 Jul 10:41:03.525 # +sdown master mymaster 192.168.0.18 6379
this is the configuration file for redis-sentinel:
bind 192.168.0.18
port 16379
sentinel monitor mymaster 192.168.0.18 6379 2
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 6000
loglevel verbose
logfile "/var/log/redis/sentinel.log"
repl-ping-slave-period 5
slave-serve-stale-data no
repl-backlog-size 8mb
min-slaves-to-write 1
min-slaves-max-lag 10
the bind directive uses the proper IP for each box.
I followed the redis tutorial here: https://redis.io/topics/sentinel but I can't get the failover to work.
Redis server version : 3.2.9
The issue is all about how redis-sentinel works because sentinel can not handle password protected redis-server.
In your redis-server configuration file (/etc/redis/redis.conf) do not use "requirepass" directive if you want to use redis-sentinel.
I'm trying to figure out how to troubleshoot my redis master / slave replication. It has "just stopped" working.
Setup Information
Let's say my master's IP address is 10.1.2.3
Here's what I've checked so far:
I've restarted redis on both the master and slave... but anytime I run INFO REPLICATION on the slave it shows the link as "down"
Ran netstat -lnp on both the master and slave. Here's the output from the master:
masterdb:~# netstat -lnp | grep 6379
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 21611/redis-server
tcp 0 0 10.1.2.3:6379 0.0.0.0:* LISTEN 21611/redis-server
And from the slave machine:
slavedb:~# netstat -lnp | grep 6379
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 5577/redis-server
tcp 0 0 :::6379 :::* LISTEN 5577/redis-server
slavedb:~#
I've checked the logs on both the master and the slave and I don't see any error messages. But I see timeout messages on the slave... which I think I've seen before, even when replication was working. The log looks like this on the slave:
5577:S 26 Oct 13:17:19.510 * MASTER <-> SLAVE sync started
5577:S 26 Oct 13:18:20.597 # Timeout connecting to the MASTER...
5577:S 26 Oct 13:18:20.597 * Connecting to MASTER 10.1.2.3:6379
5577:S 26 Oct 13:18:20.597 * MASTER <-> SLAVE sync started
5577:S 26 Oct 13:19:21.685 # Timeout connecting to the MASTER...
When i start the redis-cli on the slave and re-issue the slaveof command, i get this message:
127.0.0.1:6379> slaveof 10.1.2.3 6379
OK Already connected to specified master
127.0.0.1:6379>
I also tried the following commands on the master :
127.0.0.1:6379> save
OK
127.0.0.1:6379> bgsave
Background saving started
127.0.0.1:6379>
But that didn't resolve anything on the slave. It still says the link is down when I check the INFO on REPLCIATION:
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.1.2.3
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:1
master_link_down_since_seconds:1477488462
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
127.0.0.1:6379>
I'm not sure what else to check.
sounds like a networking issue. Try to
telnet masterip 6379 from the slave next time it happens to ensure it is a network issue.
I was testing Redis Sentinel's failover ability. It worked, and Sentinel added some lines to the conf files. It auto-discovered the other sentinels and slave replicas, but it added some weird ids.
Can anyone tell me what those ids represent? Since they come right after known-sentinel, I assume they are the id of those sentinels but I can't be sure.
# Generated by CONFIG REWRITE
sentinel known-slave redis_master 127.0.0.1 6379
sentinel known-slave redis_master 127.0.0.1 6381
sentinel known-sentinel redis_master 127.0.0.1 26380
26f81b692201f11f0f16747b007da9d4f079d9d3 # this
sentinel known-sentinel redis_master 127.0.0.1 26381
0b613c6146bbf261f08c1b13f1d1b2dbc2f99413 # and this?
It's the run_id of sentinel. Remember sentinel is a special redis instance. Log into the sentinel and using "info server" to see its information, which includes the run_id. e.g.
redis-cli -h sentinel_host -p sentinel_port
info server
If you have multiple sentinels, you can use
sentinel sentinels mymaster(or redis_master in your situation)
to list all other sentinels' infomation.