I want my redis-cli client to subscribe to sentinel notifications like "+odown", "+sdown", "+switch-master", here is what i tried:
$ redis-cli
127.0.0.1:6379> subscribe "+odown"
nothing... I triggered a failover, still nothing.
To make sure my connection is good, I tried psubscribe * and triggered another failover, I only see a bunch of:
3) "__sentinel__:hello"
4) "127.0.0.1,5001,ece169c208d80543b07a18ead87c53eb4bd87e4a,30,mymaster,127.0.0.1,6380,30"
1) "pmessage"
2) "*"
3) "__sentinel__:hello"
4) "127.0.0.1,5001,ece169c208d80543b07a18ead87c53eb4bd87e4a,30,mymaster,127.0.0.1,6380,30"
1) "pmessage"
2) "*"
3) "__sentinel__:hello"
4) "127.0.0.1,5000,060d95f0255ca3a70fa33b5caa5dbba787c3af55,30,mymaster,127.0.0.1,6380,30"
What can I do to subscribe to those internal system events?
sentinelEvent(REDIS_WARNING,"+sdown",ri,"%#");
will pub events to +sdown channel.
Do it on a redis sentinel instance.
$ ps -ef | grep redis
root 13961 1 0 21:30 ? 00:00:00 redis-server *:26379 [sentinel]
$ redis-cli -p 26379
127.0.0.1:26379> subscribe +sdown
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "+sdown"
3) (integer) 1
1) "message"
2) "+sdown"
3) "master mymaster 127.0.0.1 6379"
Related
We have a Redis cluster (3 master, 3 slave) and are seeing a large number of keys on one of the master nodes (and related slaves) which appear to be empty and cannot be deleted.
If I connect to the master node which has a large number of entries (as determined by DBSIZE), I can SCAN for and see the keys:
--> redis-cli -h 192.168.100.81 -p 6381
192.168.100.81:6381> scan 0 match mykey-* count 10
1) "2359296"
2) 1) "mykey-1be333a7"
2) "mykey-e85a9d31"
3) "mykey-d9162eff"
4) "mykey-41d12fd8"
5) "mykey-a6e755d3"
6) "mykey-c2aa1eaa"
7) "mykey-c0597cac"
8) "mykey-10e69376"
9) "mykey-7263aef0"
10) "mykey-7fa9de50"
However, if I try and GET the value of a key, it shows it has moved:
192.168.100.81:6381> get mykey-1be333a7
(error) MOVED 8301 192.168.3.107:6380
If I connect to the node to which the key has moved, I am not able to GET or DEL the value:
--> redis-cli -h 192.168.3.107 -p 6380
192.168.3.107:6380> get mykey-1be333a7
(nil)
192.168.3.107:6380> del mykey-1be333a7
(integer) 0
I am also unable to GET or DEL the value using the cluster (-c) flag for redis-cli:
--> redis-cli -h 192.168.100.81 -p 6381 -c get mykey-1be333a7
(nil)
--> redis-cli -h 192.168.100.81 -p 6381 -c del mykey-1be333a7
(integer) 0
--> redis-cli -h 192.168.3.107 -p 6380 -c get mykey-1be333a7
(nil)
--> redis-cli -h 192.168.3.107 -p 6380 -c del mykey-1be333a7
(integer) 0
What can I do to remove these types of keys?
Interesting question!
How to reproduce the problem
The following is a scenario that will reproduce your problem:
You create a standalone Redis instance, and set some data into it. However, one day, you configure this standalone Redis as a member of a Redis Cluster, without flushing the old data or moving the old data to the right node of the cluster. Let's call these data as dirty data.
In this scenario, you can SCAN all keys on this instance, including the dirty data. However, you cannot read or write these dirty data, since your Redis is in cluster mode, it will redirect your request to the right node, which doesn't have such data.
How to solve the problem
In order to remove these dirty data, you should reconfigure your Redis instance into standalone mode, i.e. cluster-enabled no, and delete dirty data in standalone mode.
Then you can make it join the cluster again.
I want some clients to be unable to subscribe to some topics in Redis pubsub. Is this possible with redis acl?
Yes, this is how you can use ACL to limit a client only to the "foo" topic:
$ redis-cli
127.0.0.1:6379> ACL SETUSER limitedpubsub on nopass -#all +subscribe|foo
OK
127.0.0.1:6379> AUTH limitedpubsub ""
OK
127.0.0.1:6379> SUBSCRIBE bar
Reading messages... (press Ctrl-C to quit)
(error) NOPERM this user has no permissions to run the 'subscribe' command or its subcommand
^C
$ redis-cli
127.0.0.1:6379> AUTH limitedpubsub ""
OK
127.0.0.1:6379> SUBSCRIBE foo
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "foo"
3) (integer) 1
I setup a cluster of 3 redis-sentinel (3.2.6-1) on three instance of redis-server (3.2.6-1).
I checked the firewall for the 6379 and 26379 TCP port and it's all good.
The configuration for my redis-sentinel is something like that:
port 26379
dir "/tmp"
sentinel myid 0559ec26112bebce70bbfa5849f77338453315b
sentinel monitor rback 10.3.0.43 6379 2
sentinel down-after-milliseconds rback 5000
sentinel failover-timeout rback 10000
daemonize yes
pidfile "/var/run/redis/redis-sentinel.pid"
loglevel notice
logfile "/var/log/redis/redis-sentinel.log"
When I start the redis-server and redis-sentinel instances, I can query on the port 26379 port sentinel master rback and see the options:
9) "flags"
10) "master"
...
31) "num-slaves"
32) "2"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
In the logs of the redis-sentinel, I see this:
26851:X 12 Jun 15:22:35.092 * +sentinel sentinel 4b22b6ff1b983432028f8cdb0db75cd553bec4b3 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:40.105 * +sentinel sentinel 8fc263bf82226364917478541c13f2c7f5b746e6 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:40.168 # +sdown sentinel 4b22b6ff1b983432028f8cdb0db75cd553bec4b3 XXXXX 26379 # redis-back XXXXX 6379
26851:X 12 Jun 15:22:45.120 # +sdown sentinel 8fc263bf82226364917478541c13f2c7f5b746e6 XXXXX 26379 # redis-back XXXXX 6379
And if I run the sleep command or crash the master redis, I see each sentinel logging a +sdown command, but never promote it to +odown and promoting a new master.
How can I debug this?
Thanks
Add Information:
I run a tcpdump and analyse the traffic with wireshark, and found out that the sentinel is connecting to the other sentinel and try to communicate with it, but receive a "DENIED Redis is running in protected mode...". Even though the redis-servers are not running in protected mode.
The problem is the communication between the sentinel.
Redis adds with 3.2 version a "protected-mode" configuration flag on the sentinel.conf too.
The sentinel will receive an error message "Denied Redis is running in protected mode..." if the sentinel doesn't have the flag.
I found this information here:
https://newbiedba.wordpress.com/2016/07/01/redis-3-2-sentinel-with-protected-mode/
I'm trying to figure out how to troubleshoot my redis master / slave replication. It has "just stopped" working.
Setup Information
Let's say my master's IP address is 10.1.2.3
Here's what I've checked so far:
I've restarted redis on both the master and slave... but anytime I run INFO REPLICATION on the slave it shows the link as "down"
Ran netstat -lnp on both the master and slave. Here's the output from the master:
masterdb:~# netstat -lnp | grep 6379
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 21611/redis-server
tcp 0 0 10.1.2.3:6379 0.0.0.0:* LISTEN 21611/redis-server
And from the slave machine:
slavedb:~# netstat -lnp | grep 6379
tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 5577/redis-server
tcp 0 0 :::6379 :::* LISTEN 5577/redis-server
slavedb:~#
I've checked the logs on both the master and the slave and I don't see any error messages. But I see timeout messages on the slave... which I think I've seen before, even when replication was working. The log looks like this on the slave:
5577:S 26 Oct 13:17:19.510 * MASTER <-> SLAVE sync started
5577:S 26 Oct 13:18:20.597 # Timeout connecting to the MASTER...
5577:S 26 Oct 13:18:20.597 * Connecting to MASTER 10.1.2.3:6379
5577:S 26 Oct 13:18:20.597 * MASTER <-> SLAVE sync started
5577:S 26 Oct 13:19:21.685 # Timeout connecting to the MASTER...
When i start the redis-cli on the slave and re-issue the slaveof command, i get this message:
127.0.0.1:6379> slaveof 10.1.2.3 6379
OK Already connected to specified master
127.0.0.1:6379>
I also tried the following commands on the master :
127.0.0.1:6379> save
OK
127.0.0.1:6379> bgsave
Background saving started
127.0.0.1:6379>
But that didn't resolve anything on the slave. It still says the link is down when I check the INFO on REPLCIATION:
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.1.2.3
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:1
master_link_down_since_seconds:1477488462
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
127.0.0.1:6379>
I'm not sure what else to check.
sounds like a networking issue. Try to
telnet masterip 6379 from the slave next time it happens to ensure it is a network issue.
I have a health check I'm trying to use that executes the redis-cli command from the redis servers to the redis-sentinels remotely.
redis-cli -h 10.10.10.10 -p 26379 SENTINEL MASTER testing
There is a logic that sorts out whether there is a quorum and it all works fine unless a sentinel's network interface is unavailable. The redis-cli command hangs indefinitely in this case and the health check fails even though there are two healthy sentinels with a quorum.
I can't seem to find a way to set a timeout for the redis-cli on the client side to prevent it from hanging. Is there a way with redis-cli to do this or will I have to go outside the command to ensure it doesn't hang indefinitely?
I decided to use the timeout command to wrap the redis-cli command. It seems to work very well for my purposes!
timeout 3 redis-cli -h 10.10.10.10 -p 26379 SENTINEL MASTER testing