I have created sentinel File as per below: sentinel.conf
sentinel monitor myHAsetup 10.147.55.132 6379 1
sentinel down-after-mlliseconds myHAsetup 6001
sentinel failover-timeout myHAsetup 60000
sentinel parallel-syncs myHAsetup 1
bind 10.147.55.132
While running it, getting below error
[root#harish bin]# ./redis-server /etc/redis/sentinel.conf --sentinel
* FATAL CONFIG FILE ERROR *
Reading the configuration file, at line 2
'sentinel down-after-mlliseconds myHAsetup 6001'
Unrecognized sentinel configuration statement.
Related
I have a redis sentinel structure with 3 redis instances and 3 redis-sentinel instances. When I make them up for the first time, slaves constantly changes their master to 127.0.0.1 and then back to real master and then to 127.0.0.1 and so on. When I force the master to be down for 20 sec, another instance becomes master and everything becomes stable. Here are my config. What can cause this and how can I solve this problem? Logs are constantly saying fix-slave-config when sentinels are unstable.
initial master config
daemonize yes
port 6380
bind 0.0.0.0
supervised systemd
pidfile "/run/redis/redis-sentinel.log"
logfile "/var/log/redis/sentinel.log"
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel auth-pass mymaster ***
sentinel down-after-milliseconds mymaster 4000
sentinel failover-timeout mymaster 2000
sentinel parallel-syncs mymaster 1
sentinel announce-ip IP_OF_CURRENT_MACHINE
initial slaves' config
daemonize yes
port 6380
bind 0.0.0.0
supervised systemd
pidfile "/run/redis/redis-sentinel.log"
logfile "/var/log/redis/sentinel.log"
sentinel monitor mymaster MASTER_IP 6379 2
sentinel auth-pass mymaster ***
sentinel down-after-milliseconds mymaster 4000
sentinel failover-timeout mymaster 2000
sentinel parallel-syncs mymaster 1
sentinel announce-ip IP_OF_CURRENT_MACHINE
What I tried:
quorum= 1,2
down-after-milliseconds 4000,10000
failover-timeout 2000,6000
I have Master-Slave(3 nodes) setup using Redis Sentinel but when I try to do failover, I am seeing the below error
127.0.0.1:26379> sentinel failover mymaster
(error) NOGOODSLAVE No suitable replica to promote
Below is the Redis server configurations for master and slave nodes
#redis.conf of master node
bind 127.0.0.1 192.26.x.1
protected-mode no
daemonize yes
logfile /opt/softwares/redis-6.0.16/log/redis-server.log
#redis.conf of slave node 1
bind 127.0.0.1 192.26.x.2
protected-mode no
daemonize yes
logfile /opt/softwares/redis-6.0.16/log/redis-server.log
replicaof 192.26.x.1 6379
#redis.conf of slave node 2
bind 127.0.0.1 192.26.x.3
protected-mode no
daemonize yes
logfile /opt/softwares/redis-6.0.16/log/redis-server.log
replicaof 192.26.x.1 6379
Below is my sentinel nodes config
# Same for all sentinel nodes
bind 127.0.0.1 192.26.x.1
protected-mode no
port 26379
daemonize yes
pidfile "/var/run/redis-sentinel.pid"
logfile "/var/log/redis-sentinel.log"
sentinel monitor mymaster 192.26.x.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
When I queried the slave's status, I can see master-related config are not defined.
127.0.0.1:26379> SENTINEL masters
31) "master-link-down-time"
32) "0"
33) "master-link-status"
34) "err"
35) "master-host"
36) "?"
37) "master-port"
38) "0"
I am starting the Redis-servers in all nodes then Redis-sentinel. Not sure the ordering of starting nodes matters.
Please let me know whether I am missing some configurations or doing something wrong. The Redis version that I am using is 6.0.16.
Thanks in advance.
Changing the order of IP addresses in a bind in all configuration files solved the issue i.e.
bind <public_ip> <localhost>
I setup a cluster of 3 redis-sentinel (3.2.6-1) on three instance of redis-server (3.2.6-1).
I checked the firewall for the 6379 and 26379 TCP port and it's all good.
The configuration for my redis-sentinel is something like that:
port 26379
dir "/tmp"
sentinel myid 0559ec26112bebce70bbfa5849f77338453315b
sentinel monitor rback 10.3.0.43 6379 2
sentinel down-after-milliseconds rback 5000
sentinel failover-timeout rback 10000
daemonize yes
pidfile "/var/run/redis/redis-sentinel.pid"
loglevel notice
logfile "/var/log/redis/redis-sentinel.log"
When I start the redis-server and redis-sentinel instances, I can query on the port 26379 port sentinel master rback and see the options:
9) "flags"
10) "master"
...
31) "num-slaves"
32) "2"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
In the logs of the redis-sentinel, I see this:
26851:X 12 Jun 15:22:35.092 * +sentinel sentinel 4b22b6ff1b983432028f8cdb0db75cd553bec4b3 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:40.105 * +sentinel sentinel 8fc263bf82226364917478541c13f2c7f5b746e6 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:40.168 # +sdown sentinel 4b22b6ff1b983432028f8cdb0db75cd553bec4b3 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:45.120 # +sdown sentinel 8fc263bf82226364917478541c13f2c7f5b746e6 XXXXX 26379 # redis-back XXXXX 6379
And if I run the sleep command or crash the master redis, I see each sentinel logging a +sdown command, but never promote it to +odown and promoting a new master.
How can I debug this?
Thanks
Add Information:
I run a tcpdump and analyse the traffic with wireshark, and found out that the sentinel is connecting to the other sentinel and try to communicate with it, but receive a "DENIED Redis is running in protected mode...". Even though the redis-servers are not running in protected mode.
The problem is the communication between the sentinel.
Redis adds with 3.2 version a "protected-mode" configuration flag on the sentinel.conf too.
The sentinel will receive an error message "Denied Redis is running in protected mode..." if the sentinel doesn't have the flag.
I found this information here:
https://newbiedba.wordpress.com/2016/07/01/redis-3-2-sentinel-with-protected-mode/
I'm testing redis failover with this simple setup:
3 Ubuntu server 16.04
redis and redis-sentinel are configured on each box.
Master ip : 192.168.0.18
Resque ip : 192.168.0.16
Resque2 ip : 192.168.0.13
Data replication works well but I can't get failover to work.
When I start redis-sentinel I always get a +sdown message after 60 seconds:
14913:X 17 Jul 10:40:03.505 # +monitor master mymaster 192.168.0.18 6379 quorum 2
14913:X 17 Jul 10:41:03.525 # +sdown master mymaster 192.168.0.18 6379
this is the configuration file for redis-sentinel:
bind 192.168.0.18
port 16379
sentinel monitor mymaster 192.168.0.18 6379 2
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 6000
loglevel verbose
logfile "/var/log/redis/sentinel.log"
repl-ping-slave-period 5
slave-serve-stale-data no
repl-backlog-size 8mb
min-slaves-to-write 1
min-slaves-max-lag 10
the bind directive uses the proper IP for each box.
I followed the redis tutorial here: https://redis.io/topics/sentinel but I can't get the failover to work.
Redis server version : 3.2.9
The issue is all about how redis-sentinel works because sentinel can not handle password protected redis-server.
In your redis-server configuration file (/etc/redis/redis.conf) do not use "requirepass" directive if you want to use redis-sentinel.
I was testing Redis Sentinel's failover ability. It worked, and Sentinel added some lines to the conf files. It auto-discovered the other sentinels and slave replicas, but it added some weird ids.
Can anyone tell me what those ids represent? Since they come right after known-sentinel, I assume they are the id of those sentinels but I can't be sure.
# Generated by CONFIG REWRITE
sentinel known-slave redis_master 127.0.0.1 6379
sentinel known-slave redis_master 127.0.0.1 6381
sentinel known-sentinel redis_master 127.0.0.1 26380
26f81b692201f11f0f16747b007da9d4f079d9d3 # this
sentinel known-sentinel redis_master 127.0.0.1 26381
0b613c6146bbf261f08c1b13f1d1b2dbc2f99413 # and this?
It's the run_id of sentinel. Remember sentinel is a special redis instance. Log into the sentinel and using "info server" to see its information, which includes the run_id. e.g.
redis-cli -h sentinel_host -p sentinel_port
info server
If you have multiple sentinels, you can use
sentinel sentinels mymaster(or redis_master in your situation)
to list all other sentinels' infomation.