I'm trying to setup a typical redis sentinel configuration, with three machines that will run three redis servers and three redis sentinels. The Master/Slave part of the redis servers are working OK, but the sentinels are not working. When I start two sentinels, the sentinel with the master detects the slaves, but mark them as down after the specified amount of time. I'm running Redis 3.0.5 64-bit in debian jessie machines.
8319:X 22 Dec 14:06:17.855 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
8319:X 22 Dec 14:06:17.855 # Sentinel runid is cdd5bbd5b84c876982dbca9d45ecc4bf8500e7a2
8319:X 22 Dec 14:06:17.855 # +monitor master mymaster xxxxxxxx0 6379 quorum 2
8319:X 22 Dec 14:06:18.857 * +slave slave xxxxxxxx2:6379 xxxxxxx2 6379 # mymaster xxxxxxx0 6379
8319:X 22 Dec 14:06:18.858 * +slave slave xxxxxx1:6380 xxxxxxx1 6380 # mymaster xxxxxxx0 6379
8319:X 22 Dec 14:07:18.862 # +sdown slave xxxxxxxx1:6380 xxxxxxx1 6380 # mymaster xxxxxx0 6379
8319:X 22 Dec 14:07:18.862 # +sdown slave xxxxxx2:6379 xxxxxxx2 6379 # mymaster xxxxxx0 6379
Sentinel config file:
daemonize yes
pidfile "/var/run/redis/redis-sentinel.pid"
logfile "/var/log/redis/redis-sentinel.log"
bind 127.0.0.1 xxxxxxx0
port 26379
sentinel monitor mymaster xxxxxxx0 6379 2
sentinel down-after-milliseconds mymaster 60000
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
dir "/var/lib/redis"
Of course, there is connectivity between these machines, as the slaves are working OK:
7553:S 22 Dec 13:46:33.285 * Connecting to MASTER xxxxxxxx0:6379 <br/>
7553:S 22 Dec 13:46:33.286 * MASTER <-> SLAVE sync started
7553:S 22 Dec 13:46:33.286 * Non blocking connect for SYNC fired the event.
7553:S 22 Dec 13:46:33.287 * Master replied to PING, replication can continue...
7553:S 22 Dec 13:46:33.288 * Partial resynchronization not possible (no cached master)
7553:S 22 Dec 13:46:33.291 * Full resync from master: f637ca8fe003acd09c6d021aed3f89a0d9994c9b:98290
7553:S 22 Dec 13:46:33.350 * MASTER <-> SLAVE sync: receiving 18 bytes from master
7553:S 22 Dec 13:46:33.350 * MASTER <-> SLAVE sync: Flushing old data
7553:S 22 Dec 13:46:33.350 * MASTER <-> SLAVE sync: Loading DB in memory
7553:S 22 Dec 13:46:33.350 * MASTER <-> SLAVE sync: Finished with success
7553:S 22 Dec 14:01:33.072 * 1 changes in 900 seconds. Saving...
I can answer myself. The problem was that the first IP that appeared in the sentinel conf was the localhost ip. It needs to be the binding IP. Just in case it serves anyone.
Related
I have 1 master and 2 slave redis servers. I configured sentinel.conf in all the 3 servers as same(sentinel port-26379).
When I start the Sentinel I got an error message saying the following:
16765:X 22 May 2019 12:19:53.215 # Configuration loaded
16766:X 22 May 2019 12:19:53.217 * Increased maximum number of open files to 10032 (it was originally set to 1024).
16766:X 22 May 2019 12:19:53.217 # Could not create server TCP listening socket slave-1-ip:26379: bind: Cannot assign requested address
16766:X 22 May 2019 12:19:53.217 # Could not create server TCP listening socket slave-2-ip:26379: bind: Cannot assign requested address
16766:X 22 May 2019 12:19:53.218 * Running mode=sentinel, port=26379.
16766:X 22 May 2019 12:19:53.218 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
16766:X 22 May 2019 12:19:53.218 # Sentinel ID is f3738395bf4099ca3faebc7ea7caaa45e93b40a3
16766:X 22 May 2019 12:19:53.218 # +monitor master redis-01 masterip 6379 quorum 2
16766:X 22 May 2019 12:21:23.219 # +sdown master redis-01 masterip 6379
16766:X 22 May 2019 12:58:00.013 # +sdown slave slave-1-ip:6379 slave-1-ip 6379 # redis-01 masterip 6379
My sentinel.conf file:
protected-mode no
port 26379
daemonize yes
sentinel monitor redis-01 master-ip 6379 2
# sentinel announce-ip 1.2.3.4
sentinel down-after-milliseconds redis-01 90000
sentinel failover-timeout redis-01 90000
sentinel parallel-syncs redis-01 2
# Generated by CONFIG REWRITE
protected-mode no
sentinel auth-pass redis-01 password
sentinel config-epoch redis-01 0
sentinel leader-epoch redis-01 0
sentinel known-replica redis-01 slave-1-ip 6379
sentinel known-replica redis-01 slave-2-ip 6379
sentinel current-epoch 0
What am I missing?
What should I change to make sentinel working perfect?
UPDATE:
I changed the sentinel ports to a different port on all hosts(26379, 26380, 26381 for host1, host2 and host3 respectively, previously it's same on all 26379).
It's working perfectly fine now;
But why can't I run sentinel on the same port on all the hosts? while I can run redis server same on all hosts(6379)?
I have setup a redis master-slave(s) cluster with sentinel monitoring for HA on linux CentOS7.0 (redis v4.0.2).
Sentinel is working well as, when I shutdown one of the three nodes, another node is elected as the new master.
Now I try to setup a reconfig script to notify clients of the new master.
I created a readable and executable (chmod a+rx) script in /usr/opt/notify_master.sh then I added such a line in my 3 sentinel nodes in /etc/sentinel.conf:
sentinel client-reconfig-script mymaster /usr/opt/notify_master.sh
Looking at sentinel config with a sentinel master mymaster command, I can confirm that client-reconfig-script is well configured:
10.0.0.41:26379> sentinel master mymaster
...
41) "client-reconfig-script"
42) "/usr/opt/notify_master.sh"
However, when a failover occurs, my reconfig script is not triggered. And I wonder why. Here is the sentinel log:
3314:X 02 Apr 10:14:07.069 # +sdown master mymaster 10.0.0.41 6379
3314:X 02 Apr 10:14:07.159 # +new-epoch 254
3314:X 02 Apr 10:14:07.161 # +vote-for-leader 8ed286e02e81d6946e0d007f569e164a9404c03f 254
3314:X 02 Apr 10:14:07.679 # +config-update-from sentinel 8ed286e02e81d6946e0d007f569e164a9404c03f 10.0.0.40 26379 # mymaster 10.0.0.41 6379
3314:X 02 Apr 10:14:07.679 # +switch-master mymaster 10.0.0.41 6379 10.0.0.42 6379
3314:X 02 Apr 10:14:07.680 * +slave slave 10.0.0.40:6379 10.0.0.40 6379 # mymaster 10.0.0.42 6379
3314:X 02 Apr 10:14:07.680 * +slave slave 10.0.0.41:6379 10.0.0.41 6379 # mymaster 10.0.0.42 6379
3314:X 02 Apr 10:14:37.742 # +sdown slave 10.0.0.41:6379 10.0.0.41 6379 # mymaster 10.0.0.42 6379
3314:X 02 Apr 10:48:36.099 # -sdown slave 10.0.0.41:6379 10.0.0.41 6379 # mymaster 10.0.0.42 6379
3314:X 02 Apr 10:48:46.056 * +convert-to-slave slave 10.0.0.41:6379 10.0.0.41 6379 # mymaster 10.0.0.42 6379
Here is the notify_master.sh script (which does a VIP drift)
#!/bin/bash
MASTER_IP=$6
LOCAL_IP=`ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:"`
VIP='10.0.0.31'
NETMASK='24'
INTERFACE='eno16777736'
if [ ${MASTER_IP} = ${LOCAL_IP} ];then
sudo /usr/sbin/ip addr add ${VIP}/${NETMASK} dev ${INTERFACE}
sudo /usr/sbin/arping -q -c 3 -A ${VIP} -I ${INTERFACE}
exit 0
else
sudo /usr/sbin/ip addr del ${VIP}/${NETMASK} dev ${INTERFACE}
exit 0
fi
exit 1
This script works mannauly.
I have setup a redis master-slave(s) cluster with sentinel monitoring for HA on linux debian (using stretch backports: redis v4.0.2).
Sentinel is working well as, when I shutdown one of the three nodes, another node is elected as the new master.
Now I try to setup a reconfig script to notify clients of the new master.
I created a readable and executable (chmod a+rx) script in /var/redis/test.sh then I added such a line in my 3 sentinel nodes in /etc/redis/sentinel.conf:
sentinel client-reconfig-script mymaster /var/redis/test.sh
Looking at sentinel config with a sentinel master mymaster command, I can confirm that client-reconfig-script is well configured:
10.2.0.6:26379> sentinel master mymaster
...
43) "client-reconfig-script"
44) "/var/redis/test.sh"
However, when a failover occurs, my reconfig script is not triggered. And I wonder why. Here is the sentinel log:
29765:X 16 Oct 23:03:11.724 # Executing user requested FAILOVER of 'mymaster'
29765:X 16 Oct 23:03:11.724 # +new-epoch 480
29765:X 16 Oct 23:03:11.724 # +try-failover master mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:11.777 # +vote-for-leader 5a0661a5982701465a387b4872cfa4c576edbd38 480
29765:X 16 Oct 23:03:11.777 # +elected-leader master mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:11.777 # +failover-state-select-slave master mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:11.854 # +selected-slave slave 10.2.0.8:6379 10.2.0.8 6379 # mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:11.854 * +failover-state-send-slaveof-noone slave 10.2.0.8:6379 10.2.0.8 6379 # mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:11.910 * +failover-state-wait-promotion slave 10.2.0.8:6379 10.2.0.8 6379 # mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:12.838 # +promoted-slave slave 10.2.0.8:6379 10.2.0.8 6379 # mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:12.838 # +failover-state-reconf-slaves master mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:12.893 * +slave-reconf-sent slave 10.2.0.6:6379 10.2.0.6 6379 # mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:13.865 * +slave-reconf-inprog slave 10.2.0.6:6379 10.2.0.6 6379 # mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:13.865 * +slave-reconf-done slave 10.2.0.6:6379 10.2.0.6 6379 # mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:13.937 # +failover-end master mymaster 10.2.0.7 6379
29765:X 16 Oct 23:03:13.937 # +switch-master mymaster 10.2.0.7 6379 10.2.0.8 6379
29765:X 16 Oct 23:03:13.937 * +slave slave 10.2.0.6:6379 10.2.0.6 6379 # mymaster 10.2.0.8 6379
29765:X 16 Oct 23:03:13.937 * +slave slave 10.2.0.7:6379 10.2.0.7 6379 # mymaster 10.2.0.8 6379
May I have a missing configuration option?
additional information: I installed a similar architecture a few weeks ago (redis 4.0.1) and it worked (I mean it was firing my reconfig script), but I did not keep the configuration, so I may have missed something. Or... could it be a bug introduced in v4.0.2?!
The 'chroot-like environment' for me was the systemd setup that comes with the default apt install redis-sentinel.
Changing the options in /etc/systemd/system/sentinel.service
PrivateTmp=no
ReadWriteDirectories=-/tmp
will make writing a test file to /tmp work as expected.
Sending emails from the command line involves switching most of the other options off (or swap it to run as root...)
I finally solved my problem.
The "reconfig.sh" script WAS fired by the failover, but I didn't realize it was because:
sentinel logging (even in debug mode) is not very clear about the reconfig script execution
reconfig script seems to be run in a chroot-like environment that made my tests non-relevant!
Here is the sentinel log when a client-reconfig-script is triggered ("script-child" lines):
32711:X 18 Oct 16:06:42.615 # +failover-state-reconf-slaves master mymaster 10.2.0.6 6379
32711:X 18 Oct 16:06:42.671 * +slave-reconf-sent slave 10.2.0.8:6379 10.2.0.8 6379 # mymaster 10.2.0.6 6379
32711:X 18 Oct 16:06:42.671 . +script-child 397
32711:X 18 Oct 16:06:42.813 . -script-child 397 0 0
Then my reconfig.sh looked like this:
#!/bin/bash
touch /tmp/reconfig
exit 0
=> Don't expect to find a /tmp/reconfig file when this script is called by Sentinel!
However, I still do not know exactly how it works internally...
If run redis as the user 'root', the client-reconfig-script will be triggered
.
I'm having trouble with resurrecting a master node with Sentinel. Specifically, slaves are promoted properly when the master is lost, but the master upon reboot is never demoted. However, if I restart Sentinel immediately the master node is demoted. Is my configuration bad, or am I missing something basic?
EDIT: Xpost with https://groups.google.com/forum/#!topic/redis-db/4AnGNssqYTw
I setup a few VMs as follows, all with Redis 3.1.999:
192.168.0.101 - Redis Slave
192.168.0.102 - Redis Slave
192.168.0.103 - Redis Master
192.168.0.201 - Sentinel
192.168.0.202 - Sentinel
My Sentinel configuration, for both sentinels:
loglevel verbose
logfile "/tmp/sentinel.log"
sentinel monitor redisA01 192.168.0.101 6379 2
sentinel down-after-milliseconds redisA01 30000
sentinel failover-timeout redisA01 120000
I stop redis on the master node; as expected Sentinel catches it and promotes a slave to master.
3425:X 08 Sep 23:47:43.839 # +sdown master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:43.896 # +odown master redisA01 192.168.0.103 6379 #quorum 2/2
3425:X 08 Sep 23:47:43.896 # +new-epoch 53
3425:X 08 Sep 23:47:43.896 # +try-failover master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:43.898 # +vote-for-leader 71de0d8f6250e436e1f76800cbe8cbae56c1be7c 53
3425:X 08 Sep 23:47:43.901 # 192.168.0.201:26379 voted for 71de0d8f6250e436e1f76800cbe8cbae56c1be7c 53
3425:X 08 Sep 23:47:43.975 # +elected-leader master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:43.976 # +failover-state-select-slave master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:44.077 # +selected-slave slave 192.168.0.102:6379 192.168.0.102 6379 # redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:44.078 * +failover-state-send-slaveof-noone slave 192.168.0.102:6379 192.168.0.102 6379 # redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:44.977 * +failover-state-wait-promotion slave 192.168.0.102:6379 192.168.0.102 6379 # redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:44.980 - -role-change slave 192.168.0.102:6379 192.168.0.102 6379 # redisA01 192.168.0.103 6379 new reported role is master
3425:X 08 Sep 23:47:44.981 # +promoted-slave slave 192.168.0.102:6379 192.168.0.102 6379 # redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:44.981 # +failover-state-reconf-slaves master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:45.068 * +slave-reconf-sent slave 192.168.0.101:6379 192.168.0.101 6379 # redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:46.031 * +slave-reconf-inprog slave 192.168.0.101:6379 192.168.0.101 6379 # redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:46.032 * +slave-reconf-done slave 192.168.0.101:6379 192.168.0.101 6379 # redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:46.101 # -odown master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:46.101 # +failover-end master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:46.102 # +switch-master redisA01 192.168.0.103 6379 192.168.0.102 6379
3425:X 08 Sep 23:47:46.103 * +slave slave 192.168.0.101:6379 192.168.0.101 6379 # redisA01 192.168.0.102 6379
3425:X 08 Sep 23:47:46.103 * +slave slave 192.168.0.103:6379 192.168.0.103 6379 # redisA01 192.168.0.102 6379
I wait a few minutes and restart Redis on the former master node. Unexpectedly (to me) the node is not demoted to slave.
3425:X 08 Sep 23:48:16.105 # +sdown slave 192.168.0.103:6379 192.168.0.103 6379 # redisA01 192.168.0.102 6379
3425:X 08 Sep 23:50:09.131 # -sdown slave 192.168.0.103:6379 192.168.0.103 6379 # redisA01 192.168.0.102 6379
After waiting a few more minutes, I restart one of the sentinels. Immediately it detects the dangling former master node and demotes it.
3425:signal-handler (1441758237) Received SIGTERM scheduling shutdown...
...
3670:X 09 Sep 00:23:57.687 # Sentinel ID is 71de0d8f6250e436e1f76800cbe8cbae56c1be7c
3670:X 09 Sep 00:23:57.687 # +monitor master redisA01 192.168.0.102 6379 quorum 2
3670:X 09 Sep 00:23:57.690 - -role-change slave 192.168.0.103:6379 192.168.0.103 6379 # redisA01 192.168.0.102 6379 new reported role is master
3670:X 09 Sep 00:23:58.708 - Accepted 192.168.0.201:49731
3670:X 09 Sep 00:24:07.778 * +convert-to-slave slave 192.168.0.103:6379 192.168.0.103 6379 # redisA01 192.168.0.102 6379
3670:X 09 Sep 00:24:17.801 - +role-change slave 192.168.0.103:6379 192.168.0.103 6379 # redisA01 192.168.0.102 6379 new reported role is slave
I would check for multiple processes on the master, and for possible circular replication. I you look at the end of the first log batch you will see it detects the 103 IP as a slave already via the +slave entry. I would try to look at why upon promotion the new master already shows the old master as a slave.
Upon restart the reconfiguration is happening, according to the logs provided, before slave rediscovery whereupon it detects the slave reporting itself as master.
Try it again, it directly interrogate each node before restarting sentinel to see what they each have for master and slaves. That might illuminate the underlying issue.
Edit: your sentinel configuration described is incorrect. You list the master as 103 in your listing, but the sentinel config file you posted indicates 101, which is a slave according to your listing.
Also, add a third sentinel. Two makes it easy to have split brain, which you may well what you are seeing.
for now we're trying to play with redis 2.8.7 as cache storage (from the .NET web application using booksleeve client).
It seems to be very interesting and exciting task at the moment, redis documentation is very nice, but due to lack of real practical experience I do have couple of questions about how expected configuration should be done properly.
I took next articles as main configuration source:
Installing redis with autostart capability (using an init script, so that after a restart everything will start again properly) : http://redis.io/topics/quickstart
Deployment of the redis into azure: http://haishibai.blogspot.com/2014/01/walkthrough-setting-up-redis-cluster-on.html
Initial idea/assumption - is to have 1 redis master and 2 slave instances running with linux Ubuntu. In order to provide high availability of the instances - I've decided to use sentinel. So my expected configuration looks like this at the moment:
MasterInstance: VM1 (linux, Ubuntu), port : 6379 (autostarted when linux is restarted)
Slave1: VM2 (linux, ubuntu), port : 6380 (autostarted when linux is restarted) : slaveOf MasterID 6379
Slave2: VM3 (linux, ubuntu), port : 6379 (autostarted when linux is restarted) : slaveOf MasterIP 6379
After VMs started, I can see that I've got 2 slaves successfully connected and syncing with Master:
Trace sample from the master:
[1120] 25 Mar 14:11:18.629 - 1 clients connected (0 slaves), 793352 bytes in use
[1120] 25 Mar 14:11:18.634 * Slave asks for synchronization
[1120] 25 Mar 14:11:18.634 * Full resync requested by slave.
[1120] 25 Mar 14:11:18.634 * Starting BGSAVE for SYNC
[1120] 25 Mar 14:11:18.634 * Background saving started by pid 1227
[1227] 25 Mar 14:11:18.810 * DB saved on disk
[1227] 25 Mar 14:11:18.810 * RDB: 0 MB of memory used by copy-on-write
[1120] 25 Mar 14:11:18.836 * Background saving terminated with success
[1120] 25 Mar 14:11:18.837 * Synchronization with slave succeeded
[1120] 25 Mar 14:11:23.829 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:23.829 - DB 2: 4 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:23.829 - 0 clients connected (1 slaves), 1841992 bytes in use
[1120] 25 Mar 14:11:29.011 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:29.011 - DB 2: 4 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:29.011 - 0 clients connected (1 slaves), 1841992 bytes in use
[1120] 25 Mar 14:11:29.826 - Accepted 168.62.36.189:1024
[1120] 25 Mar 14:11:29.828 * Slave asks for synchronization
[1120] 25 Mar 14:11:29.828 * Full resync requested by slave.
[1120] 25 Mar 14:11:29.828 * Starting BGSAVE for SYNC
[1120] 25 Mar 14:11:29.828 * Background saving started by pid 1321
[1321] 25 Mar 14:11:29.871 * DB saved on disk
[1321] 25 Mar 14:11:29.871 * RDB: 0 MB of memory used by copy-on-write
[1120] 25 Mar 14:11:29.943 * Background saving terminated with success
[1120] 25 Mar 14:11:29.946 * Synchronization with slave succeeded
[1120] 25 Mar 14:11:34.195 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:34.195 - DB 2: 4 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:34.195 - 0 clients connected (2 slaves), 1862920 bytes in use
now I need to setup sentinel instances ...
I copied sentinel.conf from the initial redis-stable package into 3 VM runnung redis (1 master and both slaves)
Inside each config I've done next modifications:
sentinel monitor mymaster MasterPublicIP 6379 2
on each VM started sentinel using next command line:
redis-server /etc/redis/sentinel.conf -- sentinel
After that I've got the response that sentinel successfully started ... on all VMs...
After I started all 3 sentinel instances I've got next trace sample (sentinel.conf files were updated with information about slaves and other sentinel instances):
[1743] 25 Mar 16:35:46.450 # Sentinel runid is 05380d689af9cca1e826ce9c85c2d68c65780878
[1743] 25 Mar 16:35:46.450 # +monitor master mymaster MasterIP 6379 quorum 2
[1743] 25 Mar 16:36:11.578 * -dup-sentinel master mymaster MasterIP 6379 #duplicate of 10.119.112.41:26379 or 83666bdd03fd064bcf2ec41ec2134d4e1e239842
[1743] 25 Mar 16:36:11.578 * +sentinel sentinel 10.119.112.41:26379 10.119.112.41 26379 # mymaster 168.62.41.1 6379
[1743] 25 Mar 16:36:16.468 # +sdown sentinel 10.175.220.134:26379 10.175.220.134 26379 # mymaster 168.62.41.1 6379
[1743] 25 Mar 16:36:40.876 * -dup-sentinel master mymaster MasterIP 6379 #duplicate of 10.175.220.134:26379 or fe9edeb321e04070c6ac6e28f52c05317a593ffd
[1743] 25 Mar 16:36:40.876 * +sentinel sentinel 10.175.220.134:26379 10.175.220.134 26379 # mymaster 168.62.41.1 6379
[1743] 25 Mar 16:37:10.962 # +sdown sentinel 10.175.220.134:26379 10.175.220.134 26379 # mymaster 168.62.41.1 6379
based on the trace sample, I have next questions. It will be great, if someone can clarify them:
Why do I see -dup-sentinel master mymaster configuration here ... Is it because I added 3 sentinels for the same master instance (maybe I need to register 1 sentinel per instance of redis - so 1 sentinel is going to be mapped to the master and 2 other sentinels - to the 2 slaves)?
how to start sentinels in the way redis servers is started (automatically even then VM is restarted)? - do I need to perform same actions and register them as ordinary redis-server instances?
Is it ok to have sentinel instance to be hosted in the same VM as redis-server?
After that I started new putty connection and started redis-cli to work with sentinel APIs, but received next response on my command below:
127.0.0.1:6379> SENTINEL masters
(error) ERR unknown command 'SENTINEL'
I guess I've done something stupid here... :(
What I've done wrong and how to test sentinel APIs from the terminal connection?
Thank you in advance for any help.
I guess "SENTINEL masters" should be run on the Redis sentinel
redis-cli -p 26379 (which the default sentinel port)
then issue
127.0.0.1:26379> SENTINEL masters
and you will get something
1) "name"
2) "mymaster"
3) "ip"
4) "127.0.0.1"
5) "port"
6) "6379"
.
.
.
To start sentinels automatically even then VM is restarted
first set daemonize yes into sentinel.conf
and modify the init script here (https://github.com/antirez/redis/blob/unstable/utils/redis_init_script) to reflect the sentinel port and .conf location.
$EXEC $CONF --sentinel # starting in Sentinel mode
and the rest is like you did for redis server.
First, you don't run Sentinel on the master. Sentinel is designed to detect when the master fails. If you run Sentinel on the same system as the master, you will lose a Sentinel when you lose the system. For the same reasons you shouldn't use the slaves as your additional test points.
You want to run Sentinel from where the clients run - to ensure you are testing for network outages.
Next, you mention you added slave information to your sentinel configs. You don't configure slaves in sentinel - it discovers them through the master. I suspect you've added additional sentinel monitor commands for each slave - this would indeed cause duplicate monitoring attempts.
Third, as #yofpro mentioned, to run sentinel commands you need to connect to sentinel -not Redis master or slaves.