Redis sentinel slave error link state on slave - redis

Question

I try to make replication using redis sentinel, on dev server wiht redis 3 all stuff works fine, but on production when i use redis 5 I had problem. First think i start to replicate using replicaof in config slave, then I configure sentinel
sentinel down-after-milliseconds mymaster 15000
sentinel failover-timeout mymaster 20000
sentinel was discover the master, but did't slave of redis, then I try manual add slave to sentinel
sentinel known-slave mymaster SLAVE-IP 6379
after change i restart sentinel, then change slave to master and make old master broken becouse
master-link-status = err and
SENTINEL failover mymaster
(error) NOGOODSLAVE No suitable slave to promote
without sentinel replication between redis works fine
redis-slave config
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile "/var/run/redis/redis-server.pid"
loglevel notice
logfile "/var/log/redis/redis-server.log"
databases 16
always-show-logo yes
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/var/lib/redis"
replica-serve-stale-data yes
replica-read-only no
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
maxmemory 2000mb
maxmemory-policy allkeys-lru
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
redis master config
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 60
daemonize yes
supervised no
pidfile "/var/run/redis/redis-server.pid"
loglevel notice
logfile "/var/log/redis/redis-server.log"
databases 16
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/var/lib/redis"
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
maxmemory 4000mb
maxmemory-policy allkeys-lru
appendonly no
appendfilename "appendonly.aof"
appendfsync no
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

Answer 1

Had similar issue. What I did was to send a SENTINEL RESET command to all Sentinel instances. One after the other, waiting at least 30 seconds between instances. That fixed the "master-link-status" and let the failover happen.
$ src/redis-cli -h 192.168.1.153 -p 26379
> SENTINEL RESET mymaster
(wait 30 seconds)
$ src/redis-cli -h 192.168.1.154 -p 26379
> SENTINEL RESET mymaster
(wait 30 seconds)
$ src/redis-cli -h 192.168.1.155 -p 26379
> SENTINEL RESET mymaster
> sentinel slaves mymaster
2) "192.168.1.155:6379"
31) "master-link-status"
32) "ok"
2) "192.168.1.154:6379"
31) "master-link-status"
32) "ok"
-> so that seems fine.
192.168.1.155:26379> sentinel failover mymaster
OK
-> finally!