Does anyone know how redis sentinel determines the value for
myid in the config file?
I've created an image for deploying redis on my network but the Redis Sentinel values for myid are all duplicated, which is causing lots of log spam showing:
[991] 30 Nov 20:56:48.639 * -dup-sentinel master mymaster 10.254.18.162 6379 #duplicate of 10.254.18.160:26379 or e2aa60902f872679221db15c4b414c7d5b3eca7c
[991] 30 Nov 20:56:48.639 * +sentinel sentinel 10.254.18.160:26379 10.254.18.160 26379 # mymaster 10.254.18.162 6379
Attempts at manually changing the value fail b/c the file rewrites itself with the old value.
Ok well what I had to do was stop redis-sentinel, delete most of the config, including the myid line, and then start redis-sentinel back up. It generated a new id.
Related
I setup a cluster of 3 redis-sentinel (3.2.6-1) on three instance of redis-server (3.2.6-1).
I checked the firewall for the 6379 and 26379 TCP port and it's all good.
The configuration for my redis-sentinel is something like that:
port 26379
dir "/tmp"
sentinel myid 0559ec26112bebce70bbfa5849f77338453315b
sentinel monitor rback 10.3.0.43 6379 2
sentinel down-after-milliseconds rback 5000
sentinel failover-timeout rback 10000
daemonize yes
pidfile "/var/run/redis/redis-sentinel.pid"
loglevel notice
logfile "/var/log/redis/redis-sentinel.log"
When I start the redis-server and redis-sentinel instances, I can query on the port 26379 port sentinel master rback and see the options:
9) "flags"
10) "master"
...
31) "num-slaves"
32) "2"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
In the logs of the redis-sentinel, I see this:
26851:X 12 Jun 15:22:35.092 * +sentinel sentinel 4b22b6ff1b983432028f8cdb0db75cd553bec4b3 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:40.105 * +sentinel sentinel 8fc263bf82226364917478541c13f2c7f5b746e6 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:40.168 # +sdown sentinel 4b22b6ff1b983432028f8cdb0db75cd553bec4b3 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:45.120 # +sdown sentinel 8fc263bf82226364917478541c13f2c7f5b746e6 XXXXX 26379 # redis-back XXXXX 6379
And if I run the sleep command or crash the master redis, I see each sentinel logging a +sdown command, but never promote it to +odown and promoting a new master.
How can I debug this?
Thanks
Add Information:
I run a tcpdump and analyse the traffic with wireshark, and found out that the sentinel is connecting to the other sentinel and try to communicate with it, but receive a "DENIED Redis is running in protected mode...". Even though the redis-servers are not running in protected mode.
The problem is the communication between the sentinel.
Redis adds with 3.2 version a "protected-mode" configuration flag on the sentinel.conf too.
The sentinel will receive an error message "Denied Redis is running in protected mode..." if the sentinel doesn't have the flag.
I found this information here:
https://newbiedba.wordpress.com/2016/07/01/redis-3-2-sentinel-with-protected-mode/
I'm testing redis failover with this simple setup:
3 Ubuntu server 16.04
redis and redis-sentinel are configured on each box.
Master ip : 192.168.0.18
Resque ip : 192.168.0.16
Resque2 ip : 192.168.0.13
Data replication works well but I can't get failover to work.
When I start redis-sentinel I always get a +sdown message after 60 seconds:
14913:X 17 Jul 10:40:03.505 # +monitor master mymaster 192.168.0.18 6379 quorum 2
14913:X 17 Jul 10:41:03.525 # +sdown master mymaster 192.168.0.18 6379
this is the configuration file for redis-sentinel:
bind 192.168.0.18
port 16379
sentinel monitor mymaster 192.168.0.18 6379 2
sentinel down-after-milliseconds mymaster 60000
sentinel failover-timeout mymaster 6000
loglevel verbose
logfile "/var/log/redis/sentinel.log"
repl-ping-slave-period 5
slave-serve-stale-data no
repl-backlog-size 8mb
min-slaves-to-write 1
min-slaves-max-lag 10
the bind directive uses the proper IP for each box.
I followed the redis tutorial here: https://redis.io/topics/sentinel but I can't get the failover to work.
Redis server version : 3.2.9
The issue is all about how redis-sentinel works because sentinel can not handle password protected redis-server.
In your redis-server configuration file (/etc/redis/redis.conf) do not use "requirepass" directive if you want to use redis-sentinel.
After sentinel failover. My slave prometed as a new master. But my old master couldn't convert as a slave. Sentinel's log is below getting forever :)
26378:X 23 May 17:55:00.429 * +convert-to-slave slave 10.0.22.43:6379 10.0.22.43 6379 # master01 10.0.22.44 6379
I tested this redis v3.2.0 and v3.0.7
same error. I am missing something.
I was testing Redis Sentinel's failover ability. It worked, and Sentinel added some lines to the conf files. It auto-discovered the other sentinels and slave replicas, but it added some weird ids.
Can anyone tell me what those ids represent? Since they come right after known-sentinel, I assume they are the id of those sentinels but I can't be sure.
# Generated by CONFIG REWRITE
sentinel known-slave redis_master 127.0.0.1 6379
sentinel known-slave redis_master 127.0.0.1 6381
sentinel known-sentinel redis_master 127.0.0.1 26380
26f81b692201f11f0f16747b007da9d4f079d9d3 # this
sentinel known-sentinel redis_master 127.0.0.1 26381
0b613c6146bbf261f08c1b13f1d1b2dbc2f99413 # and this?
It's the run_id of sentinel. Remember sentinel is a special redis instance. Log into the sentinel and using "info server" to see its information, which includes the run_id. e.g.
redis-cli -h sentinel_host -p sentinel_port
info server
If you have multiple sentinels, you can use
sentinel sentinels mymaster(or redis_master in your situation)
to list all other sentinels' infomation.
I am working with two redis instances monitored by one sentinel.
when the master goes down and there is a "+sdown", I run a notification-script where I promote the slave to master using the following command on its redis-client:
SLAVEOF NO ONE .
It works fine.
My question is, it takes around 10 secs for the slave to become master and the application to continue working again.
How can i reduce this time stamp?
below is the sentinel config::::
sentinel monitor mymaster 127.0.0.1 6379 1
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 900000
sentinel can-failover mymaster yes
sentinel parallel-syncs mymaster 1
sentinel notification-script mymaster /etc/init.d/config/script.sh
This is not a direct answer to your question, but a description of a different setup that avoids the need for having a quick response (in some scenario's).
When we model a use case in UML, we never put the redis sentinel in the default flow. The sentinel is our guard for situations where unknown errors occor: the exceptional flow.
If we know beforehand if the client needs to connect to a different redis instance, we simply instruct the client to do so, using redis pub/sub (combined with low resolution polling, since pub/sub traffic is not guaranteed-delivery).
Kind regards, TW
Try reduce sentinel down-after-milliseconds mymaster and failover-timeout mymaster