I am trying to copy all keys from db=2 to db=3 in same redis which is running at port 6380.
I have followed few other SO questions but getting NOKEY as response. Please find query for the same.Could anybody help what is wrong with below command?
redis-3.2.8/src/redis-cli -p 6380 -n 2 --scan | xargs redis-
3.2.8/src/redis-cli -p 6380 migrate localhost 6380 '' 3 0 copy keys
It's not an error.
In your case, the keys you scanned might have been expired when you try to migrate it. If the key you want to migrate doesn't exist in the source database, MIGRATE command will return NOKEY.
Related
I am trying to use redis-cli --pipe to bulk upload some commands to my AWS Elasticache for redis cluster. The commands come from parsing a file via a custom awk command, which helps generate some HSET commands. The awk command is in a custom shell script. When my Elasticache for redis server had cluster-mode disabled, doing something like the following worked like a charm:
sh script_containing_awk.sh $FILE_TO_PARSE | redis-cli -h <Primary_endpoint> -p <port> --tls --cacert <path/to/cert> --pipe
Due to an internal project requirement, the Elasticache for Redis server has been re-created with cluster-mode enabled, and hence I am adding the -c flag to the above command to specify as such.
I see the following results when trying to work with my Elasticache for Redis server with cluster-mode enabled:
I can connect to the cluster via the configuration endpoint no problem!
Single command uploads work (i.e: redis-cli -h <config_endpoint> -p <port> -c --tls --cacert <path/to/certs> SET key value)
It would be extremely convenient to just pipe output from my script to the cli:
sh script_containing_awk.sh $FILE_TO_PARSE | redis-cli -h <config_endpoint> -p <port> -c --tls --cacert <path/to/cert> --pipe
but adding the --pipe flag results in "MOVED" errors.
I have tried modifying the script to include {} (ex: HSET {user1}:hash field1 val1 field2 val2 ... brackets to try to force keys to the same CLUSTER SLOTS, but I still get the "MOVED" errors and I am attempting to bulk upload millions of keys so I don't think they would all fit in the same slot anyway.
Does anyone have experience getting --pipe to work with cluster-mode enabled Redis/Elasticache?
Thanks!
I am sure you understand that the core difference between Cluster Mode Disabled and Cluster Mode Enabled is that there is a split in your total Key slots.
Just to put in context;
CMD - Let's say we have 4 node cluster with 1 Primary and 3 Replicas.
if we have 100 key slots -
All the 100 key slots will be there in all the nodes. 3 of them will serve Read only commands and 1 of the node will serve all the commands.
CME - Let's say we have 4 nodes split in 2 shards - 1 replica and 1 primary each.
We can look at them as logical sub-clusters ie. they will have different sets of key-slots. Ideally a 50-50 split.
Now, the MOVED message is not necessarily an error.
When you connect to the configuration endpoint, by default you are being connected with one of the primary nodes (chosen at random, at first).
when you make a command, the client sends that command and the primary node decides if it has the correct hash-slot to serve that command.
As explained here, if the node does not have the hash-slot that your client is looking for, it will redirect you with a MOVED message.
So, I would assume MOVED messages are somewhat expected with CME clusters.
When I scan the Redis using below command:
redis-cli -h <redis_master_ip> -p 6379 --scan --pattern '*'
it returns keys which belong to this node, but it also returns many keys which belong to another redis node. Therefore if I run below command:
redis-cli -h <redis_master_ip> -p 6379 object freq <some_keys_from_scan>
I get error like Error: MOVED 90 <another_redis_master_ip>:6379
Due to the same reason, I get the same error when running:
redis-cli -h <redis_master_ip> -p 6379 --hotkeys
Note both the <redis_master_ip> and <another_redis_master_ip> are a part of redis cluster.
The document https://redis.io/commands/scan defines scan as: "iterates the set of keys in the currently selected Redis database". My understanding is it should only scan the keys belong to the current node. my redis cluster is 6.0.10.
Does anybody know why executing scan return the keys of another node? I am only interested in getting the keys of this node.
I see another link mentioned the same issue but no solution yet: https://github.com/redis/redis/issues/4810
Is it possible to copy all keys from one Redis instance to another remote instance using MIGRATE? I've tried COPY, REPLACE and KEYS without any luck. Each time I get a NOKEY response. If I use any of the MIGRATE commands with a single key it works.
Examples:
MIGRATE my.redis 6379 "*" 0 5000 REPLACE // NOKEY
MIGRATE my.redis 6379 "*" 0 5000 COPY // NOKEY
MIGRATE my.redis 6379 "" 0 5000 KEYS * // NOKEY
MIGRATE my.redis 6379 "" 0 5000 KEYS test // OK
This is an improvement on the answer provided by #ezain since I am unable to post comments. The command uses the correct redis syntax for processing batches of keys, but the arguments to xargs result in the command being called once for every key instead of just once with all the keys included (which means it'll take much more time to complete than is necessary). The following will be much faster in all cases:
redis-cli --raw KEYS '*' | xargs redis-cli MIGRATE my.redis 6379 "" 0 5000 KEYS
If the destination is password protected:
redis-cli --raw KEYS '*' | xargs redis-cli MIGRATE my.redis 6379 "" 0 5000 AUTH password-here KEYS
try run in your shell
redis-cli keys '*' | xargs -I '{}' redis-cli migrate my.redis 6379 "" 0 5000 KEYS '{}'
For a big DBs with a lot of keys it's better to use --scan instead of keys, to avoid Redis lock on KEYS command:
redis-cli --scan | xargs redis-cli MIGRATE my.redis 6379 "" 0 5000 KEYS
Not really related to the question, but in case someone will need it: Redis does not support MIGRATE with a password before 3.0. After 3.0, you can add AUTH parameter to check the permission:
MIGRATE 192.168.0.33 6379 "" 0 5000 AUTH mypassword KEYS user:{info}:age
If you are running on non-managed¹ redis instances, the most ideal way would probably to run the target instance as a replica temporarly and then disable (after all data is copied) the replication.
see the REPLICAOF command in redis. how to apply it (all commands on the target instance):
initiate the replication: $ replicaof source_hostname_or_ip source_port
after everything is done: $ replicaof no one
If you can't use this command¹ then you can try this script on the digital ocean blog: https://www.digitalocean.com/community/tutorials/how-to-migrate-redis-data-to-a-digitalocean-managed-database#step-3-%E2%80%94-building-the-migration-script
################
¹ - managed services often restrict the usage of this command see here or here.
I'm not advocating using this, but I tried all of these examples, and many others and did not work. I ended up doing it myself in PHP, so maybe this will help someone else who is stuck.
<?php
$redisSource = new Redis();
$redisSource->connect('1.2.3.4', 6379);
$redisSource->auth('password');
$redisTarget = new Redis();
$redisTarget->connect('127.0.0.1', 6379);
foreach($redisSource->keys('*') as $key) {
$redisTarget->set($key, $redisSource->get($key));
}
How to delete keys matching a certain pattern in redis using redis-cli. I would like to delete all foo's from the following list.
KEYS *
foo:1
foo:2
bar:1
foo:3
bar:2
foo:4
As mentioned in the comment on the question, there are many other answers to this here already. Definitely read the one linked above if you are thinking about doing this in a production sever.
The one I found most useful for occasional command-line cleanup was:
redis-cli KEYS "*" | xargs redis-cli DEL
from "How to atomically delete keys matching a pattern using Redis".
I wanted to delete thousands of keys by pattern after some searches I found these points:
if you have more than one db on redis you should determine the database using -n [number]
if you have a few keys use del but if there are thousands or millions of keys it's better to use unlink because unlink is non-blocking while del is blocking, for more information visit this page unlink vs del
also keys are like del and is blocking
so I used this code to delete keys by pattern:
redis-cli -n 2 --scan --pattern '[your pattern]' | xargs redis-cli -n 2 unlink
I just published a command line interface utility to npm and github that allows you to delete keys that match a given pattern (even *) from a Redis database.
You can find the utility here:
https://www.npmjs.com/package/redis-utils-cli
If someone want to do same operation in AWS Elasticache redis, then you can connect with SSH to your EC2 server which is supposed to access AWS Redis server then you can use below command.
redis-cli -h <HOST> -p <PORT> --scan --pattern "patter*n" | xargs redis-cli -h <HOST> -p <PORT> unlink
Replace Host and port with AWS redis server host and port.
Also if your redis setup needs password authentication then use,
redis-cli -h <HOST> -p <PORT> -a <PASSWORD> --scan --pattern "patter*n" | xargs redis-cli -h <HOST> -p <PORT> -a <PASSWORD> unlink
Replace Host, port and password with AWS redis server host, port and password.
You can also use above commands for localhost.
I am new to YCSB and i want to benchmark Redis using more than one cluster. I have tried with only one Redis on my localhost with the following command.
./bin/ycsb load redis -p redis.host=localhost -p redis.port=6379 -P workloads/workloada -p recordcount=200000 -s > d.dat
I am getting the currect ops/sec and other data.
Now i need to know how can run YCSB for more than one cluster.
Can anybody give an answer (steps to run this).
And it would be helpful if anyone can help to run Couchbase YCSB too.
Thanx..!!
The following steps need to be done to perform YCSB bench marking with redis cluster
1) configure redis cluster with different nodes by referring to the document
http://redis.io/topics/cluster-tutorial
2) open the ycsb terminal,set up redis host ip address, port and specify the required parameters needed
./bin/ycsb load redis -p redis.host="ip address" -p redis.port="port" -P workloads/workloada -p recordcount=200000 -s > d.dat