I'm using a redis setup with a master, two slaves and a single instance of redis sentinel ( i know this is not recomended, but we are just doing a proof of concept).
The master listens on 6379.
When we failover the master, we query redis sentinel and now my slave is the master, so let's say that 6380 is now the master.
127.0.0.1:46379> sentinel masters
1) 1) "name"
2) "redis6379-master"
3) "ip"
4) "127.0.0.1"
5) "port"
6) "6380"
But when I try to write to the instance on port 6380, it says it can't write to it because it's a slave.
So when I query that instance, this is what I get (partially truncated for simplicity):
# Replication
role:slave
master_host:127.0.0.1
master_port:6380
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:1
slave_repl_offset:1
master_sync_left_bytes:-1
master_sync_last_io_seconds_ago:10
master_link_down_since_seconds:531
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
It seems like sentinel isn't being able to reconfigure the slave to act has a master? Any clues?
Thanks,
Pedro
Related
When I check my slaves
redis-cli -p 26379 sentinel slaves mycluster
I get some ips that have already been nuked.
Is there away to take them out of sentinel?
You can send SENTINEL RESET master command to every sentinel to remove these slaves.
We have an application with Redis Sentinel(3 sentinel 1 master 1 slave). We don't need replication at all. How can I turn off replication?
The command SLAVEOF NO ONE, when sent to a slave, will turn off the replication. The slave will become a master.
If you disable replication, you probably don't need Redis Sentinel either.
Hi i have create a cluster Redis with sentinel composed by 3 aws instances, i have configured sentinel to have an HA redis cluster and work, but if i simulate a crash of master (shutdown of master instance), sentinel installed on slaves, not locate sentinel of master and the election fail.
My sentinel configuration is:
sentinel monitor master ip-master 6379 2
sentinel down-after-milliseconds master 5000
sentinel failover-timeout master 10000
sentinel parallel-syncs master 1
Same file to all instaces
There are issues when running sentinel on the same node as the master and attempting to trigger a failover. Try it w/o running Sentinel on the master. Ultimately this means not running Sentinel on the same nodes as the Redis instances.
In your case your dead-node simulation is showing why you should not run Sentinel on the same node as Redis: If the node dies you lose one of your sentinels. In theory it should still work but as you and others have seen it isn't certain to work. I have some theories why but I've not yet confirmed them.
In a sense Sentinel is partly a monitoring system. Running a monitoring solution on the same nodes as are being monitored is generally unadvisable anyway, so you should be using off-node sentinels anyway. As Sentinel is resource efficient you don't necessarily need dedicated machines or large VMs. Indeed if you have a static set of application servers (where your client code runs), you should run Sentinel there, keeping in mind you want 3 minimum and a quorum of 50%+1.
recent redis version introduced the "protected-mode" option, which defaults to yes.
with protected-mode set to yes, redis instances, without a password set will not allow remote clients to execute commands.
this also affects sentinels master election.
try it with setting "protected-mode no" in the sentinels. this will allow them to talk to each other.
If you don't want to set protected-mode as no. you'd better set masterauth myredis in redis.conf and use sentinel auth-pass mymaster myredis in sentinel.conf
On my slave redis server, after executing 'slaveof master_hostname 7804', it keeps logging the following messages:
Connecting to MASTER master_hostname:7804
Unable to connect to MASTER: Invalid argument
On the Linux server that hosts my slave redis server, I can connect to my master redis server without any problem:
$ redis-cli -h master_hostname -p 7804
The output of 'info' on master:
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
The output of 'info' on slave:
# Replication
role:slave
master_host:master_hostname
master_port:7804
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:1
master_link_down_since_seconds:1428997735
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
Both redis servers are 3.0.0. If I use the IP of the master redis server instead of its hostname, I meet the same problem.
Any idea?
I am having the exactly same issue here. I'm using Redis 3.0.0 in both machines (master and slave).
I've tried sysctl vm.overcommit_memory=1, but still not working.
In my case, I'm using CentOS 6.6.
EDIT:
I've found the solution! In slave config, if you have configured bind 127.0.0.1, then you need to change to bind 192.168.1.54 (where 192.168.1.54 is the slave IP). Remember that master is in the same network 192.168.1.*.
I had same issue with Redis 3 in MASTER-SLAVE setup.
I changed bind 127.0.0.1 10.234.1.10 to bind 10.234.1.10 127.0.0.1 and it works!
So, it looks, that "127.0.0.1" cannot be first in the list.
I'm successfully using Redis for Windows (2.6.8-pre2) in a master slave setup. However, I need to provide some automated failover capability, and it appears the sentinel is the most popular choice. When I run redis in sentinel mode the sentinel connects, but it always thinks the master is down. Also, when I run the sentinel master command it reports that there are 0 slaves (not true) and that there are no other sentinels (again, not true). So it's like it connects to the master, but not correctly.
Has anyone else seen this issue on Windows and, more importantly, is anyone successfully using sentinel in a windows environment? Any help or direction at all is appreciated!
I recommend use this:
1 master node redis server 1 slave node redis server
List item 3 redis sentinels with a quorum of 2
It's so important have more than have 3 sentinels to get a odd quorum.
I made this configuration in Windows 7 and it's working well.
Example of sentinel conf:
port 20001
logfile "sentinel1.log"
sentinel monitor shard1 127.0.0.1 16379 2
sentinel down-after-milliseconds shard1 5000
sentinel failover-timeout shard1 30000