I am trying to get a list of all sentinels which are currently monitoring the redis master.
I know that if I have one sentinel I can use sentinel sentinels mymaster but if I don't have any of the Sentinel's addresses how can I get them?
There is no direct command to get the list of sentinels from a master/slave node. To get the sentinels' list, you need to subscribe to any node's pub/sub (master or slave doesn't matter) "__sentinel__:hello" channel and wait for the messages. Messages passing through that hello channel are from sentinels that are listening that cluster. If you parse those, you get the sentinels' addresses. The messages are in form: "sentinel_ip,sentinel_port,sentinel_runid,sentinel_current_epoch,master_name,master_ip,master_port,master_config_epoch" (e.g. 127.0.0.1,26380,07fabf3cbac43bcc955588b1023f95498b58f8f2,16,mymaster,127.0.0.1,6381,16). See: https://redis.io/topics/sentinel#sentinels-and-slaves-auto-discovery about sentinel details. If you need more to know about how sentinel works, take a look at https://github.com/antirez/redis/blob/unstable/src/server.c
In order to explore more about this instance, you may want to try the following two commands:
SENTINEL slaves mymaster
SENTINEL sentinels mymaster
https://redis.io/topics/sentinel#asking-sentinel-about-the-state-of-a-master
Related
When I check my slaves
redis-cli -p 26379 sentinel slaves mycluster
I get some ips that have already been nuked.
Is there away to take them out of sentinel?
You can send SENTINEL RESET master command to every sentinel to remove these slaves.
Question Background:
I deploy a redis cluster in k8s cluster and use Redis-Sentinel to implement ha for redis cluster. My redis cluster structure likes below:
One master
One slave
three sentinel （serve a specific redis cluster)
When i login the container of the one of sentinels, i execute a command:
sentinel sentinels mymaster
Luckly, i get a desirable output. These are two sentinel's infos. After a period of time, i execute "sentinels mymaster" command again, i found that there is a additional sentinel and don't find this instance through IP address or runId。
I know that sentinel discover other sentinels and master and slave through sub the channel of sentinel:hello in redis master.
Question:
how to check the message published from redis sentinel to redis master? I have opened log for master and set the log level to debug.
You can see the Sentinel's activity (when it discovers a sentinel, a replica, failsover to a new master, etc.) in the sentinel log file, not the master. If a sentinel is running on a host, it will be in the same directory the master or replica log file is. For me on CentOS it's /var/log/redis/sentinel.log.
I have a redis sentinel configuration with one master, two slaves and 3 sentinels running. I noticed that at some point the sentinels may switch the master electing one of the slaves as master. This is causing problems to an application which is connecting to the master node as a standalone client(I'm working on changing the code to use sentinels). I wanted to know if it is possible to switch the master by connecting to the sentinel client i.e. through 'redis-cli'
Can somebody let me know if there is a command that I can use to switch the master IP?
The client applications should use a client library that supports sentinel in the case where a redis master goes down and the sentinels select a new master. Not sure how beneficial it is to have sentinel setup if your client applications are not taking advantage of it. A client application that supports sentinel will query sentinel for the master ip and should be somewhat tolerant to faults occurring with the master connection. You can trigger a manual failover like the other answer states:
redis-cli -h {sentinel-ip} -p {26379 or sentinel port} sentinel failover {mastername}
But you will not be able to pick which node it fails over to. You can control a configuration value slave_priority in the redis.conf file so that it prefers a node over the rest. A description of the slave priority can be found here: https://redis.io/topics/sentinel
You can manually trigger a failover by running:
redis-cli -a {password} -p {sentinel_port} SENTINEL failover {cluster_name}
If you are using Lettuce Client you can use masterSlaveStatefulConnection and pass the sentinel URI it will perform auto discovery in the background and will refresh the master node internally.
https://github.com/lettuce-io/lettuce-core/wiki/Master-Replica
Hi i have create a cluster Redis with sentinel composed by 3 aws instances, i have configured sentinel to have an HA redis cluster and work, but if i simulate a crash of master (shutdown of master instance), sentinel installed on slaves, not locate sentinel of master and the election fail.
My sentinel configuration is:
sentinel monitor master ip-master 6379 2
sentinel down-after-milliseconds master 5000
sentinel failover-timeout master 10000
sentinel parallel-syncs master 1
Same file to all instaces
There are issues when running sentinel on the same node as the master and attempting to trigger a failover. Try it w/o running Sentinel on the master. Ultimately this means not running Sentinel on the same nodes as the Redis instances.
In your case your dead-node simulation is showing why you should not run Sentinel on the same node as Redis: If the node dies you lose one of your sentinels. In theory it should still work but as you and others have seen it isn't certain to work. I have some theories why but I've not yet confirmed them.
In a sense Sentinel is partly a monitoring system. Running a monitoring solution on the same nodes as are being monitored is generally unadvisable anyway, so you should be using off-node sentinels anyway. As Sentinel is resource efficient you don't necessarily need dedicated machines or large VMs. Indeed if you have a static set of application servers (where your client code runs), you should run Sentinel there, keeping in mind you want 3 minimum and a quorum of 50%+1.
recent redis version introduced the "protected-mode" option, which defaults to yes.
with protected-mode set to yes, redis instances, without a password set will not allow remote clients to execute commands.
this also affects sentinels master election.
try it with setting "protected-mode no" in the sentinels. this will allow them to talk to each other.
If you don't want to set protected-mode as no. you'd better set masterauth myredis in redis.conf and use sentinel auth-pass mymaster myredis in sentinel.conf
I am working with two redis instances monitored by one sentinel.
when the master goes down and there is a "+sdown", I run a notification-script where I promote the slave to master using the following command on its redis-client:
SLAVEOF NO ONE .
It works fine.
My question is, it takes around 10 secs for the slave to become master and the application to continue working again.
How can i reduce this time stamp?
below is the sentinel config::::
sentinel monitor mymaster 127.0.0.1 6379 1
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 900000
sentinel can-failover mymaster yes
sentinel parallel-syncs mymaster 1
sentinel notification-script mymaster /etc/init.d/config/script.sh
This is not a direct answer to your question, but a description of a different setup that avoids the need for having a quick response (in some scenario's).
When we model a use case in UML, we never put the redis sentinel in the default flow. The sentinel is our guard for situations where unknown errors occor: the exceptional flow.
If we know beforehand if the client needs to connect to a different redis instance, we simply instruct the client to do so, using redis pub/sub (combined with low resolution polling, since pub/sub traffic is not guaranteed-delivery).
Kind regards, TW
Try reduce sentinel down-after-milliseconds mymaster and failover-timeout mymaster