We have a Redis cluster (3 master, 3 slave) and are seeing a large number of keys on one of the master nodes (and related slaves) which appear to be empty and cannot be deleted.
If I connect to the master node which has a large number of entries (as determined by DBSIZE), I can SCAN for and see the keys:
--> redis-cli -h 192.168.100.81 -p 6381
192.168.100.81:6381> scan 0 match mykey-* count 10
1) "2359296"
2) 1) "mykey-1be333a7"
2) "mykey-e85a9d31"
3) "mykey-d9162eff"
4) "mykey-41d12fd8"
5) "mykey-a6e755d3"
6) "mykey-c2aa1eaa"
7) "mykey-c0597cac"
8) "mykey-10e69376"
9) "mykey-7263aef0"
10) "mykey-7fa9de50"
However, if I try and GET the value of a key, it shows it has moved:
192.168.100.81:6381> get mykey-1be333a7
(error) MOVED 8301 192.168.3.107:6380
If I connect to the node to which the key has moved, I am not able to GET or DEL the value:
--> redis-cli -h 192.168.3.107 -p 6380
192.168.3.107:6380> get mykey-1be333a7
(nil)
192.168.3.107:6380> del mykey-1be333a7
(integer) 0
I am also unable to GET or DEL the value using the cluster (-c) flag for redis-cli:
--> redis-cli -h 192.168.100.81 -p 6381 -c get mykey-1be333a7
(nil)
--> redis-cli -h 192.168.100.81 -p 6381 -c del mykey-1be333a7
(integer) 0
--> redis-cli -h 192.168.3.107 -p 6380 -c get mykey-1be333a7
(nil)
--> redis-cli -h 192.168.3.107 -p 6380 -c del mykey-1be333a7
(integer) 0
What can I do to remove these types of keys?
Interesting question!
How to reproduce the problem
The following is a scenario that will reproduce your problem:
You create a standalone Redis instance, and set some data into it. However, one day, you configure this standalone Redis as a member of a Redis Cluster, without flushing the old data or moving the old data to the right node of the cluster. Let's call these data as dirty data.
In this scenario, you can SCAN all keys on this instance, including the dirty data. However, you cannot read or write these dirty data, since your Redis is in cluster mode, it will redirect your request to the right node, which doesn't have such data.
How to solve the problem
In order to remove these dirty data, you should reconfigure your Redis instance into standalone mode, i.e. cluster-enabled no, and delete dirty data in standalone mode.
Then you can make it join the cluster again.
Related
I am trying to use redis-cli --pipe to bulk upload some commands to my AWS Elasticache for redis cluster. The commands come from parsing a file via a custom awk command, which helps generate some HSET commands. The awk command is in a custom shell script. When my Elasticache for redis server had cluster-mode disabled, doing something like the following worked like a charm:
sh script_containing_awk.sh $FILE_TO_PARSE | redis-cli -h <Primary_endpoint> -p <port> --tls --cacert <path/to/cert> --pipe
Due to an internal project requirement, the Elasticache for Redis server has been re-created with cluster-mode enabled, and hence I am adding the -c flag to the above command to specify as such.
I see the following results when trying to work with my Elasticache for Redis server with cluster-mode enabled:
I can connect to the cluster via the configuration endpoint no problem!
Single command uploads work (i.e: redis-cli -h <config_endpoint> -p <port> -c --tls --cacert <path/to/certs> SET key value)
It would be extremely convenient to just pipe output from my script to the cli:
sh script_containing_awk.sh $FILE_TO_PARSE | redis-cli -h <config_endpoint> -p <port> -c --tls --cacert <path/to/cert> --pipe
but adding the --pipe flag results in "MOVED" errors.
I have tried modifying the script to include {} (ex: HSET {user1}:hash field1 val1 field2 val2 ... brackets to try to force keys to the same CLUSTER SLOTS, but I still get the "MOVED" errors and I am attempting to bulk upload millions of keys so I don't think they would all fit in the same slot anyway.
Does anyone have experience getting --pipe to work with cluster-mode enabled Redis/Elasticache?
Thanks!
I am sure you understand that the core difference between Cluster Mode Disabled and Cluster Mode Enabled is that there is a split in your total Key slots.
Just to put in context;
CMD - Let's say we have 4 node cluster with 1 Primary and 3 Replicas.
if we have 100 key slots -
All the 100 key slots will be there in all the nodes. 3 of them will serve Read only commands and 1 of the node will serve all the commands.
CME - Let's say we have 4 nodes split in 2 shards - 1 replica and 1 primary each.
We can look at them as logical sub-clusters ie. they will have different sets of key-slots. Ideally a 50-50 split.
Now, the MOVED message is not necessarily an error.
When you connect to the configuration endpoint, by default you are being connected with one of the primary nodes (chosen at random, at first).
when you make a command, the client sends that command and the primary node decides if it has the correct hash-slot to serve that command.
As explained here, if the node does not have the hash-slot that your client is looking for, it will redirect you with a MOVED message.
So, I would assume MOVED messages are somewhat expected with CME clusters.
When I scan the Redis using below command:
redis-cli -h <redis_master_ip> -p 6379 --scan --pattern '*'
it returns keys which belong to this node, but it also returns many keys which belong to another redis node. Therefore if I run below command:
redis-cli -h <redis_master_ip> -p 6379 object freq <some_keys_from_scan>
I get error like Error: MOVED 90 <another_redis_master_ip>:6379
Due to the same reason, I get the same error when running:
redis-cli -h <redis_master_ip> -p 6379 --hotkeys
Note both the <redis_master_ip> and <another_redis_master_ip> are a part of redis cluster.
The document https://redis.io/commands/scan defines scan as: "iterates the set of keys in the currently selected Redis database". My understanding is it should only scan the keys belong to the current node. my redis cluster is 6.0.10.
Does anybody know why executing scan return the keys of another node? I am only interested in getting the keys of this node.
I see another link mentioned the same issue but no solution yet: https://github.com/redis/redis/issues/4810
My redis cluster is borked. One node seems to think it's a 3-node cluster with no replication, another node seems to think it's alone, and the third node agrees with the first. I want to have a 3-node cluster with 2 replicated nodes (no splitting).
I don't want to have to figure out how to un-bork my redis cluster, instead: I want to start over.
The existing question How do I delete everything in Redis? doesn't include clusters, for which everything is way more complicated.
Desired result: Clean installation. No cluster config.
Unfortunately simply reinstalling the whole program appears insufficient. How do I tell it to really --yes-I-am-really-sure forget everything about itself and all the other nodes it knows? In particular:
Remove all non-config-file-cluster-configuration.
The command CLUSTER FORGET gets me partway there, but nodes can't forget themselves? Why not? Anyway, somehow convincing them to do that might be enough.
The difficult part is in realizing these three things:
There's both redis-cli CLUSTER and redis-cli --cluster, and the cluster commands are randomly divided between the two.
When usingredis-cli --cluster for something involving two nodes, the -a password parameter must be provided twice, both before and after the --cluster part. The one provided after is for the target node, the one provided before is for the 'config' node.
Commands using redis CLUSTER only apply to one node, so must be repeated for each node. (They cannot access the other nodes)
Knowing this, we can proceed wiping the necessary things:
redis-cli -h host01 -a password CLUSTER NODES
prints a list of nodes with long identifiers. Proceed with
# forget about the other nodes in the cluster on each node
redis-cli -h host01 -a password1 CLUSTER FORGET <host02 key>
redis-cli -h host01 -a password1 CLUSTER FORGET <host03 key>
redis-cli -h host02 -a password2 CLUSTER FORGET <host01 key>
redis-cli -h host02 -a password2 CLUSTER FORGET <host03 key>
redis-cli -h host03 -a password3 CLUSTER FORGET <host01 key>
redis-cli -h host03 -a password3 CLUSTER FORGET <host02 key>
# flush all data so that the node can be reset
redis-cli -h host01 -a password1 FLUSHALL
redis-cli -h host02 -a password2 FLUSHALL
redis-cli -h host03 -a password3 FLUSHALL
# reset the node
redis-cli -h host01 -a password1 CLUSTER RESET
redis-cli -h host02 -a password2 CLUSTER RESET
redis-cli -h host03 -a password3 CLUSTER RESET
info keyspace
Its currently incremental and purged at end of month . But i want to have per day between mentioned time for some kpi analysis .
Set up a cron job of:
redis-cli -h host -p port info keyspace | grep db0 | sed 's/.*keys=\([0-9]*\).*/\1/' | xargs redis-cli -h host -p port set metric:keys:$(date "+%y-%m-%d-%H")
This will get you a set of keys in Redis with metrics at specific hour.
~$ redis-cli -h host -p port get metric:keys:18-06-15-12
"25"
This one-liner will get keyspace info, filter info for db0 (change for any other you interested in), extract the number, send back to Redis as a metric. You also can change that to a hash so metric itself won't change your number. But for 1m+ instances a couple of keys won't matter. Or you can store them in another db if you want.
I have setup up Redis master slave configuration having one master (6379 port) and 3 slaves (6380,6381,6382) running in the same machine. Looks like cluster is setup properly as I can see the following output on running info command:
# Replication
role:master
connected_slaves:3
slave0:ip=127.0.0.1,port=6380,state=online,offset=29,lag=1
slave1:ip=127.0.0.1,port=6381,state=online,offset=29,lag=1
slave2:ip=127.0.0.1,port=6382,state=online,offset=29,lag=1
master_repl_offset:43
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:42
But wherever I try to add new key in master, I get the following error:
(error) CLUSTERDOWN Hash slot not served
Using redis-3.0.7 in Mac OS X Yosemite.
I had the same issue, turned out I forgot to run the create cluster:
cd /path/to/utils/create-cluster
./create-cluster create
http://redis.io/topics/cluster-tutorial#creating-a-redis-cluster-using-the-create-cluster-script
To fix slots issue while insertion:
redis-cli --cluster fix localhost:6379
You can use ruby script buddled with redis for creating clusters as mentioned below :
/usr/local/redis-3.2.11/src/redis-trib.rb create --replicas 1 192.168.142.128:7001 192.168.142.128:7002 192.168.142.128:7003 192.168.142.128:7004 192.168.142.128:7005 192.168.142.128:7006
There is a hash slot not allotted to any master. Check the hash slots by looking at the column 9 in the output of following command (column 9 will be empty if no hash slots for that node):
redis-cli -h masterIP -p masterPORT CLUSTER NODES
The hash slots can be allotted by using the following command.
redis-cli -h masterIP -p masterPORT CLUSTER ADDSLOTS SLOTNNUMBER
But this has to be done for every slot number individually without missing. Use a bat script to include it in for loop. something like,
for /L %a in (0,1,5400) Do redis-cli -h 127.0.0.1 -p 7001 cluster addslots %a
Also, this command works before assigning slaves to master. After this ADDSLOTS step and completing the setup, the SET and GET worked fine. Remember to use -c along with redis-cli before SET to enable cluster support.
The issue comes when one or more Redis nodes gets corrupted and can no longer serve its configured hash slots.
You will have to bootstrap the cluster again to make sure the nodes agree on the hash slots to serve.
If the Redis nodes contain data or a key in the database 0, you will have to clear this data before rerunning the bootstrap.