Is it possible to copy all keys from one Redis instance to another remote instance using MIGRATE? I've tried COPY, REPLACE and KEYS without any luck. Each time I get a NOKEY response. If I use any of the MIGRATE commands with a single key it works.
Examples:
MIGRATE my.redis 6379 "*" 0 5000 REPLACE // NOKEY
MIGRATE my.redis 6379 "*" 0 5000 COPY // NOKEY
MIGRATE my.redis 6379 "" 0 5000 KEYS * // NOKEY
MIGRATE my.redis 6379 "" 0 5000 KEYS test // OK
This is an improvement on the answer provided by #ezain since I am unable to post comments. The command uses the correct redis syntax for processing batches of keys, but the arguments to xargs result in the command being called once for every key instead of just once with all the keys included (which means it'll take much more time to complete than is necessary). The following will be much faster in all cases:
redis-cli --raw KEYS '*' | xargs redis-cli MIGRATE my.redis 6379 "" 0 5000 KEYS
If the destination is password protected:
redis-cli --raw KEYS '*' | xargs redis-cli MIGRATE my.redis 6379 "" 0 5000 AUTH password-here KEYS
try run in your shell
redis-cli keys '*' | xargs -I '{}' redis-cli migrate my.redis 6379 "" 0 5000 KEYS '{}'
For a big DBs with a lot of keys it's better to use --scan instead of keys, to avoid Redis lock on KEYS command:
redis-cli --scan | xargs redis-cli MIGRATE my.redis 6379 "" 0 5000 KEYS
Not really related to the question, but in case someone will need it: Redis does not support MIGRATE with a password before 3.0. After 3.0, you can add AUTH parameter to check the permission:
MIGRATE 192.168.0.33 6379 "" 0 5000 AUTH mypassword KEYS user:{info}:age
If you are running on non-managed¹ redis instances, the most ideal way would probably to run the target instance as a replica temporarly and then disable (after all data is copied) the replication.
see the REPLICAOF command in redis. how to apply it (all commands on the target instance):
initiate the replication: $ replicaof source_hostname_or_ip source_port
after everything is done: $ replicaof no one
If you can't use this command¹ then you can try this script on the digital ocean blog: https://www.digitalocean.com/community/tutorials/how-to-migrate-redis-data-to-a-digitalocean-managed-database#step-3-%E2%80%94-building-the-migration-script
################
¹ - managed services often restrict the usage of this command see here or here.
I'm not advocating using this, but I tried all of these examples, and many others and did not work. I ended up doing it myself in PHP, so maybe this will help someone else who is stuck.
<?php
$redisSource = new Redis();
$redisSource->connect('1.2.3.4', 6379);
$redisSource->auth('password');
$redisTarget = new Redis();
$redisTarget->connect('127.0.0.1', 6379);
foreach($redisSource->keys('*') as $key) {
$redisTarget->set($key, $redisSource->get($key));
}
Related
I am trying to use redis-cli --pipe to bulk upload some commands to my AWS Elasticache for redis cluster. The commands come from parsing a file via a custom awk command, which helps generate some HSET commands. The awk command is in a custom shell script. When my Elasticache for redis server had cluster-mode disabled, doing something like the following worked like a charm:
sh script_containing_awk.sh $FILE_TO_PARSE | redis-cli -h <Primary_endpoint> -p <port> --tls --cacert <path/to/cert> --pipe
Due to an internal project requirement, the Elasticache for Redis server has been re-created with cluster-mode enabled, and hence I am adding the -c flag to the above command to specify as such.
I see the following results when trying to work with my Elasticache for Redis server with cluster-mode enabled:
I can connect to the cluster via the configuration endpoint no problem!
Single command uploads work (i.e: redis-cli -h <config_endpoint> -p <port> -c --tls --cacert <path/to/certs> SET key value)
It would be extremely convenient to just pipe output from my script to the cli:
sh script_containing_awk.sh $FILE_TO_PARSE | redis-cli -h <config_endpoint> -p <port> -c --tls --cacert <path/to/cert> --pipe
but adding the --pipe flag results in "MOVED" errors.
I have tried modifying the script to include {} (ex: HSET {user1}:hash field1 val1 field2 val2 ... brackets to try to force keys to the same CLUSTER SLOTS, but I still get the "MOVED" errors and I am attempting to bulk upload millions of keys so I don't think they would all fit in the same slot anyway.
Does anyone have experience getting --pipe to work with cluster-mode enabled Redis/Elasticache?
Thanks!
I am sure you understand that the core difference between Cluster Mode Disabled and Cluster Mode Enabled is that there is a split in your total Key slots.
Just to put in context;
CMD - Let's say we have 4 node cluster with 1 Primary and 3 Replicas.
if we have 100 key slots -
All the 100 key slots will be there in all the nodes. 3 of them will serve Read only commands and 1 of the node will serve all the commands.
CME - Let's say we have 4 nodes split in 2 shards - 1 replica and 1 primary each.
We can look at them as logical sub-clusters ie. they will have different sets of key-slots. Ideally a 50-50 split.
Now, the MOVED message is not necessarily an error.
When you connect to the configuration endpoint, by default you are being connected with one of the primary nodes (chosen at random, at first).
when you make a command, the client sends that command and the primary node decides if it has the correct hash-slot to serve that command.
As explained here, if the node does not have the hash-slot that your client is looking for, it will redirect you with a MOVED message.
So, I would assume MOVED messages are somewhat expected with CME clusters.
I am struggling to connect redis in sentinel mode using redis-cli.
I've tried:
redis-cli -h my_host -p my_port -a my_password
I randomly picked up one from 3 sentinel nodes to connect.
However, when logged in, it seems different with single mode redis, where I can manipulate dbs with set or hset commmand.
For instance, when I am typing:
select 0 (trying to select db 0)
It returns:
redis_sentinel_node_1_ip:port> select 0
(error) ERR unknown command `select`, with args beginning with: `0`
Could anyone help ?
While Sentinel and Redis use the same communication protocol (and live in the same executable as well), they support a very different set of commands. Afaik, Sentinel does not have the notion of multiple databases, so SELECT wouldn't have any sense there. If you just want to test a random command, you can use PING which is supported by both Redis and Sentinel.
You may want to review the set of commands Sentinel supports on the official docs.
I am trying to copy all keys from db=2 to db=3 in same redis which is running at port 6380.
I have followed few other SO questions but getting NOKEY as response. Please find query for the same.Could anybody help what is wrong with below command?
redis-3.2.8/src/redis-cli -p 6380 -n 2 --scan | xargs redis-
3.2.8/src/redis-cli -p 6380 migrate localhost 6380 '' 3 0 copy keys
It's not an error.
In your case, the keys you scanned might have been expired when you try to migrate it. If the key you want to migrate doesn't exist in the source database, MIGRATE command will return NOKEY.
I am trying to test my cache was implemented with redis clustering (cluster by server not client).
I have to flush redis every time I run a unit test.
when I try to run flushdb command I got this error:
Cannot use 'FLUSHDB' with redis-cluster.
it seems that I can run flushdb command in cluster mode only when I set the slot but I do not know how to do it. (I have overridden redis wrapper of laravel so laravel is not the case If you learn me how to use predis I can adopt it with laravel)
For deleting by pattern:
redis-cli --raw keys "$PATTERN" | xargs redis-cli del
for example:
redis-cli KEYS "prefix:*" | xargs redis-cli DEL
For deleting all keys from one db:
redis-cli flushdb
For deleting all keys from all dbs:
redis-cli flushall
For cluster mode you need to use this bash script:
https://gist.github.com/yaud/85e0382d26c189bdf84f0297cd54f479
to remove all nodes from master nodes (slave nodes will be synced)
How to delete keys matching a certain pattern in redis using redis-cli. I would like to delete all foo's from the following list.
KEYS *
foo:1
foo:2
bar:1
foo:3
bar:2
foo:4
As mentioned in the comment on the question, there are many other answers to this here already. Definitely read the one linked above if you are thinking about doing this in a production sever.
The one I found most useful for occasional command-line cleanup was:
redis-cli KEYS "*" | xargs redis-cli DEL
from "How to atomically delete keys matching a pattern using Redis".
I wanted to delete thousands of keys by pattern after some searches I found these points:
if you have more than one db on redis you should determine the database using -n [number]
if you have a few keys use del but if there are thousands or millions of keys it's better to use unlink because unlink is non-blocking while del is blocking, for more information visit this page unlink vs del
also keys are like del and is blocking
so I used this code to delete keys by pattern:
redis-cli -n 2 --scan --pattern '[your pattern]' | xargs redis-cli -n 2 unlink
I just published a command line interface utility to npm and github that allows you to delete keys that match a given pattern (even *) from a Redis database.
You can find the utility here:
https://www.npmjs.com/package/redis-utils-cli
If someone want to do same operation in AWS Elasticache redis, then you can connect with SSH to your EC2 server which is supposed to access AWS Redis server then you can use below command.
redis-cli -h <HOST> -p <PORT> --scan --pattern "patter*n" | xargs redis-cli -h <HOST> -p <PORT> unlink
Replace Host and port with AWS redis server host and port.
Also if your redis setup needs password authentication then use,
redis-cli -h <HOST> -p <PORT> -a <PASSWORD> --scan --pattern "patter*n" | xargs redis-cli -h <HOST> -p <PORT> -a <PASSWORD> unlink
Replace Host, port and password with AWS redis server host, port and password.
You can also use above commands for localhost.