How to delete keys matching a certain pattern in redis using redis-cli. I would like to delete all foo's from the following list.
KEYS *
foo:1
foo:2
bar:1
foo:3
bar:2
foo:4
As mentioned in the comment on the question, there are many other answers to this here already. Definitely read the one linked above if you are thinking about doing this in a production sever.
The one I found most useful for occasional command-line cleanup was:
redis-cli KEYS "*" | xargs redis-cli DEL
from "How to atomically delete keys matching a pattern using Redis".
I wanted to delete thousands of keys by pattern after some searches I found these points:
if you have more than one db on redis you should determine the database using -n [number]
if you have a few keys use del but if there are thousands or millions of keys it's better to use unlink because unlink is non-blocking while del is blocking, for more information visit this page unlink vs del
also keys are like del and is blocking
so I used this code to delete keys by pattern:
redis-cli -n 2 --scan --pattern '[your pattern]' | xargs redis-cli -n 2 unlink
I just published a command line interface utility to npm and github that allows you to delete keys that match a given pattern (even *) from a Redis database.
You can find the utility here:
https://www.npmjs.com/package/redis-utils-cli
If someone want to do same operation in AWS Elasticache redis, then you can connect with SSH to your EC2 server which is supposed to access AWS Redis server then you can use below command.
redis-cli -h <HOST> -p <PORT> --scan --pattern "patter*n" | xargs redis-cli -h <HOST> -p <PORT> unlink
Replace Host and port with AWS redis server host and port.
Also if your redis setup needs password authentication then use,
redis-cli -h <HOST> -p <PORT> -a <PASSWORD> --scan --pattern "patter*n" | xargs redis-cli -h <HOST> -p <PORT> -a <PASSWORD> unlink
Replace Host, port and password with AWS redis server host, port and password.
You can also use above commands for localhost.
Related
I am trying to use redis-cli --pipe to bulk upload some commands to my AWS Elasticache for redis cluster. The commands come from parsing a file via a custom awk command, which helps generate some HSET commands. The awk command is in a custom shell script. When my Elasticache for redis server had cluster-mode disabled, doing something like the following worked like a charm:
sh script_containing_awk.sh $FILE_TO_PARSE | redis-cli -h <Primary_endpoint> -p <port> --tls --cacert <path/to/cert> --pipe
Due to an internal project requirement, the Elasticache for Redis server has been re-created with cluster-mode enabled, and hence I am adding the -c flag to the above command to specify as such.
I see the following results when trying to work with my Elasticache for Redis server with cluster-mode enabled:
I can connect to the cluster via the configuration endpoint no problem!
Single command uploads work (i.e: redis-cli -h <config_endpoint> -p <port> -c --tls --cacert <path/to/certs> SET key value)
It would be extremely convenient to just pipe output from my script to the cli:
sh script_containing_awk.sh $FILE_TO_PARSE | redis-cli -h <config_endpoint> -p <port> -c --tls --cacert <path/to/cert> --pipe
but adding the --pipe flag results in "MOVED" errors.
I have tried modifying the script to include {} (ex: HSET {user1}:hash field1 val1 field2 val2 ... brackets to try to force keys to the same CLUSTER SLOTS, but I still get the "MOVED" errors and I am attempting to bulk upload millions of keys so I don't think they would all fit in the same slot anyway.
Does anyone have experience getting --pipe to work with cluster-mode enabled Redis/Elasticache?
Thanks!
I am sure you understand that the core difference between Cluster Mode Disabled and Cluster Mode Enabled is that there is a split in your total Key slots.
Just to put in context;
CMD - Let's say we have 4 node cluster with 1 Primary and 3 Replicas.
if we have 100 key slots -
All the 100 key slots will be there in all the nodes. 3 of them will serve Read only commands and 1 of the node will serve all the commands.
CME - Let's say we have 4 nodes split in 2 shards - 1 replica and 1 primary each.
We can look at them as logical sub-clusters ie. they will have different sets of key-slots. Ideally a 50-50 split.
Now, the MOVED message is not necessarily an error.
When you connect to the configuration endpoint, by default you are being connected with one of the primary nodes (chosen at random, at first).
when you make a command, the client sends that command and the primary node decides if it has the correct hash-slot to serve that command.
As explained here, if the node does not have the hash-slot that your client is looking for, it will redirect you with a MOVED message.
So, I would assume MOVED messages are somewhat expected with CME clusters.
When I scan the Redis using below command:
redis-cli -h <redis_master_ip> -p 6379 --scan --pattern '*'
it returns keys which belong to this node, but it also returns many keys which belong to another redis node. Therefore if I run below command:
redis-cli -h <redis_master_ip> -p 6379 object freq <some_keys_from_scan>
I get error like Error: MOVED 90 <another_redis_master_ip>:6379
Due to the same reason, I get the same error when running:
redis-cli -h <redis_master_ip> -p 6379 --hotkeys
Note both the <redis_master_ip> and <another_redis_master_ip> are a part of redis cluster.
The document https://redis.io/commands/scan defines scan as: "iterates the set of keys in the currently selected Redis database". My understanding is it should only scan the keys belong to the current node. my redis cluster is 6.0.10.
Does anybody know why executing scan return the keys of another node? I am only interested in getting the keys of this node.
I see another link mentioned the same issue but no solution yet: https://github.com/redis/redis/issues/4810
My redis cluster is borked. One node seems to think it's a 3-node cluster with no replication, another node seems to think it's alone, and the third node agrees with the first. I want to have a 3-node cluster with 2 replicated nodes (no splitting).
I don't want to have to figure out how to un-bork my redis cluster, instead: I want to start over.
The existing question How do I delete everything in Redis? doesn't include clusters, for which everything is way more complicated.
Desired result: Clean installation. No cluster config.
Unfortunately simply reinstalling the whole program appears insufficient. How do I tell it to really --yes-I-am-really-sure forget everything about itself and all the other nodes it knows? In particular:
Remove all non-config-file-cluster-configuration.
The command CLUSTER FORGET gets me partway there, but nodes can't forget themselves? Why not? Anyway, somehow convincing them to do that might be enough.
The difficult part is in realizing these three things:
There's both redis-cli CLUSTER and redis-cli --cluster, and the cluster commands are randomly divided between the two.
When usingredis-cli --cluster for something involving two nodes, the -a password parameter must be provided twice, both before and after the --cluster part. The one provided after is for the target node, the one provided before is for the 'config' node.
Commands using redis CLUSTER only apply to one node, so must be repeated for each node. (They cannot access the other nodes)
Knowing this, we can proceed wiping the necessary things:
redis-cli -h host01 -a password CLUSTER NODES
prints a list of nodes with long identifiers. Proceed with
# forget about the other nodes in the cluster on each node
redis-cli -h host01 -a password1 CLUSTER FORGET <host02 key>
redis-cli -h host01 -a password1 CLUSTER FORGET <host03 key>
redis-cli -h host02 -a password2 CLUSTER FORGET <host01 key>
redis-cli -h host02 -a password2 CLUSTER FORGET <host03 key>
redis-cli -h host03 -a password3 CLUSTER FORGET <host01 key>
redis-cli -h host03 -a password3 CLUSTER FORGET <host02 key>
# flush all data so that the node can be reset
redis-cli -h host01 -a password1 FLUSHALL
redis-cli -h host02 -a password2 FLUSHALL
redis-cli -h host03 -a password3 FLUSHALL
# reset the node
redis-cli -h host01 -a password1 CLUSTER RESET
redis-cli -h host02 -a password2 CLUSTER RESET
redis-cli -h host03 -a password3 CLUSTER RESET
info keyspace
Its currently incremental and purged at end of month . But i want to have per day between mentioned time for some kpi analysis .
Set up a cron job of:
redis-cli -h host -p port info keyspace | grep db0 | sed 's/.*keys=\([0-9]*\).*/\1/' | xargs redis-cli -h host -p port set metric:keys:$(date "+%y-%m-%d-%H")
This will get you a set of keys in Redis with metrics at specific hour.
~$ redis-cli -h host -p port get metric:keys:18-06-15-12
"25"
This one-liner will get keyspace info, filter info for db0 (change for any other you interested in), extract the number, send back to Redis as a metric. You also can change that to a hash so metric itself won't change your number. But for 1m+ instances a couple of keys won't matter. Or you can store them in another db if you want.
Is it possible to copy all keys from one Redis instance to another remote instance using MIGRATE? I've tried COPY, REPLACE and KEYS without any luck. Each time I get a NOKEY response. If I use any of the MIGRATE commands with a single key it works.
Examples:
MIGRATE my.redis 6379 "*" 0 5000 REPLACE // NOKEY
MIGRATE my.redis 6379 "*" 0 5000 COPY // NOKEY
MIGRATE my.redis 6379 "" 0 5000 KEYS * // NOKEY
MIGRATE my.redis 6379 "" 0 5000 KEYS test // OK
This is an improvement on the answer provided by #ezain since I am unable to post comments. The command uses the correct redis syntax for processing batches of keys, but the arguments to xargs result in the command being called once for every key instead of just once with all the keys included (which means it'll take much more time to complete than is necessary). The following will be much faster in all cases:
redis-cli --raw KEYS '*' | xargs redis-cli MIGRATE my.redis 6379 "" 0 5000 KEYS
If the destination is password protected:
redis-cli --raw KEYS '*' | xargs redis-cli MIGRATE my.redis 6379 "" 0 5000 AUTH password-here KEYS
try run in your shell
redis-cli keys '*' | xargs -I '{}' redis-cli migrate my.redis 6379 "" 0 5000 KEYS '{}'
For a big DBs with a lot of keys it's better to use --scan instead of keys, to avoid Redis lock on KEYS command:
redis-cli --scan | xargs redis-cli MIGRATE my.redis 6379 "" 0 5000 KEYS
Not really related to the question, but in case someone will need it: Redis does not support MIGRATE with a password before 3.0. After 3.0, you can add AUTH parameter to check the permission:
MIGRATE 192.168.0.33 6379 "" 0 5000 AUTH mypassword KEYS user:{info}:age
If you are running on non-managed¹ redis instances, the most ideal way would probably to run the target instance as a replica temporarly and then disable (after all data is copied) the replication.
see the REPLICAOF command in redis. how to apply it (all commands on the target instance):
initiate the replication: $ replicaof source_hostname_or_ip source_port
after everything is done: $ replicaof no one
If you can't use this command¹ then you can try this script on the digital ocean blog: https://www.digitalocean.com/community/tutorials/how-to-migrate-redis-data-to-a-digitalocean-managed-database#step-3-%E2%80%94-building-the-migration-script
################
¹ - managed services often restrict the usage of this command see here or here.
I'm not advocating using this, but I tried all of these examples, and many others and did not work. I ended up doing it myself in PHP, so maybe this will help someone else who is stuck.
<?php
$redisSource = new Redis();
$redisSource->connect('1.2.3.4', 6379);
$redisSource->auth('password');
$redisTarget = new Redis();
$redisTarget->connect('127.0.0.1', 6379);
foreach($redisSource->keys('*') as $key) {
$redisTarget->set($key, $redisSource->get($key));
}