I am using Redis cluster version redis-5.0.5. I want to see all the keys present in my Redis cluster. I know for standalone we use KEYS * to get all the keys.
what is the way to see all keys in Redis cluster?
$ redis-cli -h hostname -p 90001 -c
hostname:90001> KEYS *
(empty list or set)
// I have data on my cluster
Basically, you'd need to run KEYS * (not in production, please!) on every one of the nodes. The cli can do this with the '--cluster call' command, like so:
redis-cli --cluster call hostname:90001 KEYS "*"
Requirement:
redis-cli
awk
grep
xargs
May be can try this assuming redis server reside in localhost with the default port 6379:
redis-cli cluster nodes | awk '{print $2" "$3}' | grep master | awk -F # '{print $1}' | awk -F : '{print " -h "$1" -p "$2" --scan"}' | xargs -L 1 redis-cli -c
Longer version base question above (90001 port number seriously?) and also you can change the pattern (* no filter) for filtering certain key pattern:
redis-cli -h hostname -p 90001 cluster nodes | awk '{print $2" "$3}' | grep master | awk -F # '{print $1}' | awk -F : '{print " -h "$1" -p "$2" --scan --pattern *"}' | xargs -L 1 redis-cli -c
It connects to any one of the redis node to get cluster info and then execute the keys scanning command on each of the master node.
The SCAN command may be what you're looking for, but it's O(N) so the more keys you have, the slower it's going to be. Also, check out this answer by Marc Gravell for another approach using sets: Get values by key pattern in StackExchange.Redis
How does one deletes keys with certain prefix from Redis 5+?
I've tried following, yet didn't work for me(
root#1acb94e11aa2:/data# redis-cli --version
redis-cli 5.0.4
root#1acb94e11aa2:/data# redis-cli -n 9 KEYS ISO:* | wc -l
935
root#1acb94e11aa2:/data# redis-cli -n 9 KEYS ISO:* | xargs -0 redis-cli -n 9 DEL
(integer) 0
root#1acb94e11aa2:/data# redis-cli -n 9 KEYS ISO:* | wc -l
935
root#1acb94e11aa2:/data# redis-cli -n 9 --scan --pattern ISO:* | xargs -0 redis-cli -n 9 unlink
(integer) 0
root#1acb94e11aa2:/data#
Please advise.
As long as your key names do not include spaces, you should be able to run this:
$ redis-cli -n 9 --scan --pattern "ISO:*" | xargs -n 1 redis-cli -n 9 UNLINK
EDIT: if they do include spaces, you can do:
$ redis-cli -n 9 --scan --pattern "ISO:*" | xargs -n 1 -d "\n" redis-cli -n 9 UNLINK
FIX:
root#1acb94e11aa2:/data# redis-cli -n 9 KEYS ISO:* | xargs -d "\n" redis-cli -n 9 del
(integer) 262
root#1acb94e11aa2:/data#
root#1acb94e11aa2:/data# redis-cli -n 9 KEYS ISO:*
(empty list or set)
root#1acb94e11aa2:/data#
redis-cli -s /data/redis/redis.sock --scan --pattern "*abcd|6128*" | xargs -L 100 redis-cli -s /data/redis/redis.sock DEL
above command is not deleting adta from redis and giving following output
(integer) 0
While the keys command works perfectly
redis-cli -s /data/redis/redis.sock KEYS 'abcd|6291*' | xargs redis-cli -s /data/redis/redis.sock DEL;
Is there is something wrong i am doing
Try xargs with -L 1 instead. Worked for me.
redis-cli -s /data/redis/redis.sock --scan --pattern "*abcd|6128*" | xargs -L 1 redis-cli -s /data/redis/redis.sock DEL
BTW, KEYS should be avoided in production environments as it is a blocking command.
scan only applies to some keys (by default on 10 keys per iteration). It returns an offset to continuesly run scan until you reach offset 0. Then you have sampled all keys. More details are in the documentation: http://redis.io/commands/scan
Keys on the other hand samples all keys in the db in one pass. It is also blocking due to redises single threaded architecture which might be bad for performance of other clients.
Instade of move I want to copy all my keys from a particular db to another.
Is it possible in redis if yes than how ?
If you can't use MIGRATE COPY because of your redis version (2.6) you might want to copy each key separately which takes longer but doesn't require you to login to the machines themselves and allows you to move data from one database to another.
Here's how I copy all keys from one database to another (but without preserving ttls)
#set connection data accordingly
source_host=localhost
source_port=6379
source_db=0
target_host=localhost
target_port=6379
target_db=1
#copy all keys without preserving ttl!
redis-cli -h $source_host -p $source_port -n $source_db keys \* | while read key; do
echo "Copying $key"
redis-cli --raw -h $source_host -p $source_port -n $source_db DUMP "$key" \
| head -c -1 \
| redis-cli -x -h $target_host -p $target_port -n $target_db RESTORE "$key" 0
done
Keys are not going to be overwritten, in order to do that, delete those keys before copying or simply flush the whole target database before starting.
Copies all keys from database number 0 to database number 1 on localhost.
redis-cli --scan | xargs redis-cli migrate localhost 6379 '' 1 0 copy keys
If you use the same server/port you will get a timeout error but the keys seem to copy successfully anyway. GitHub Redis issue #1903
redis-cli -a $source_password -p $source_port -h $source_ip keys /*| while read key;
do echo "Copying $key";
redis-cli --raw -a $source_password -h $source_ip -p $source_port -n $dbname DUMP "$key"| head -c -1| redis-cli -x -a $destination_password -h $destination_IP -p $destination_port RESTORE "$key" 0;
Latest solution:
Use the RIOT open-source command line tool provided by Redislabs to copy the data.
Reference: https://developer.redis.com/riot/riot-redis/cookbook.html#_performing_migration
GitHub project link: https://github.com/redis-developer/riot
How to install: https://developer.redis.com/riot/riot-redis/
# Source Redis db
SH=test1-redis.com
SP=6379
# Target Redis db
TH=test1-redis.com
TP=6379
# Copy from db0 to db1 (standalone Redis db, Or cluster mode disabled)
#
riot-redis -h $SH -p $SP --db 0 replicate -h $TH -p $TP --db 1 --batch 10000 \
--scan-count 10000 \
--threads 4 \
--reader-threads 4 \
--reader-batch 500 \
--reader-queue 2000 \
--reader-pool 4
RIOT is quicker, supports multithreading, and works well with cross-environment Redis data copy ( AWS Elasticache, Redis OSS, and Redislabs ).
Not directly. I would suggest to use the always convenient redis-rdb-tools package (from Sripathi Krishnan) to extract the data from a normal rdb dump, and reinject it to another instance.
See https://github.com/sripathikrishnan/redis-rdb-tools
As far as I understand you need to copy keys from a particular DB (e.g 5 ) to a particular DB say 10. If that is the case you can use redis database dumper (https://github.com/r043v/rdd). Although as per documentation it has a switch (-d) to select a database for operation but didn't work for me, so what I did
1.) Edit the rdd.c file and look for int main(int argc,char argv) function
2.) Change the DB to as per your requirement
3.) compile the src by **make
4.) Dump all keys using ./rdd -o "save.rdd"
5.) Edit the rdd.c file again and change the DB
6.) Make again
7.) Import by using ./rdd "save.rdd" -o insert -s "IP" -p"Port"
I know this is old, but for those of you coming here form Google:
I just published a command line interface utility to npm and github that allows you to copy keys that match a given pattern (even *) from one Redis database to another.
You can find the utility here:
https://www.npmjs.com/package/redis-utils-cli
Try using dump to first dump all the keys and then restore the same
If migrating keys inside of the same redis engine, then you might use internal command MOVE for that (pipelining for more speed):
#!/bin/bash
#set connection data accordingly
source_host=localhost
source_port=6379
source_db=4
target_db=0
total=$(redis-cli -n 4 keys \* | sed 's/^/MOVE /g' | sed 's/$/ '$target_db'/g' | wc -c)
#copy all keys without preserving ttl!
time redis-cli -h $source_host -p $source_port -n $source_db keys \* | \
sed 's/^/MOVE /g' | sed 's/$/ 0/g' | \
pv -s $total | \
redis-cli -h $source_host -p $source_port -n $source_db >/dev/null
I installed rabbitmqadmin and was able to list all the exchanges and queues. How can I use rabbitmqadmin or rabbitmqctl to delete all the queues.
First, list your queues:
rabbitmqadmin list queues name
Then from the list, you'll need to manually delete them one by one:
rabbitmqadmin delete queue name='queuename'
Because of the output format, doesn't appear you can grep the response from list queues. Alternatively, if you're just looking for a way to clear everything (read: reset all settings, returning the installation to a default state), use:
rabbitmqctl stop_app
rabbitmqctl reset # Be sure you really want to do this!
rabbitmqctl start_app
Actually super easy with management plugin and policies:
Goto Management Console (localhost:15672)
Goto Admin tab
Goto Policies tab(on the right side)
Add Policy
Fill Fields
Virtual Host: Select
Name: Expire All Policies(Delete Later)
Pattern: .*
Apply to: Queues
Definition: expires with value 1 (change type from String to Number)
Save
Checkout Queues tab again
All Queues must be deleted
And don't forget to remove policy!!!!!!.
With rabbitmqadmin you can remove them with this one-liner:
rabbitmqadmin -f tsv -q list queues name | while read queue; do rabbitmqadmin -q delete queue name=${queue}; done
In Rabbit version 3.7.10 you can run below command with root permission:
rabbitmqctl list_queues | awk '{ print $1 }' | xargs -L1 rabbitmqctl delete_queue
Try this:
rabbitmqadmin list queues name | awk '{print $2}' | xargs -I qn rabbitmqadmin delete queue name=qn
If you don't have rabbitmqadmin installed, try to purge queues with rabbitmqctl:
rabbitmqctl list_queues | awk '{ print $1 }' | xargs -L1 rabbitmqctl purge_queue
If you're trying to delete queues because they're unused and you don't want to reset, one option is to set the queue TTL very low via a policy, wait for the queues to be auto-deleted once the TTL is passed and then remove the policy (https://www.rabbitmq.com/ttl.html).
rabbitmqctl.bat set_policy delq ".*" '{"expires": 1}' --apply-to queues
To remove the policy
rabbitmqctl clear_policy delq
Note that this only works for unused queues
Original info here: http://rabbitmq.1065348.n5.nabble.com/Deleting-all-queues-in-rabbitmq-td30933.html
I made a deleteRabbitMqQs.sh, which accepts arguments to search the list of queues for, selecting only ones matching the pattern you want. If you offer no arguments, it will delete them all! It shows you the list of queues its about to delete, letting you quit before doing anything destructive.
for word in "$#"
do
args=true
newQueues=$(rabbitmqctl list_queues name | grep "$word")
queues="$queues
$newQueues"
done
if [ $# -eq 0 ]; then
queues=$(rabbitmqctl list_queues name | grep -v "\.\.\.")
fi
queues=$(echo "$queues" | sed '/^[[:space:]]*$/d')
if [ "x$queues" == "x" ]; then
echo "No queues to delete, giving up."
exit 0
fi
read -p "Deleting the following queues:
${queues}
[CTRL+C quit | ENTER proceed]
"
while read -r line; do
rabbitmqadmin delete queue name="$line"
done <<< "$queues"
If you want different matching against the arguments you pass in, you can alter the grep in line four. When deleting all queues, it won't delete ones with three consecutive spaces in them, because I figured that eventuality would be rarer than people who have rabbitmqctl printing its output out in different languages.
Enjoy!
Here is a way to do it with PowerShell. the URL may need to be updated
$cred = Get-Credential
iwr -ContentType 'application/json' -Method Get -Credential $cred 'http://localhost:15672/api/queues' | % {
ConvertFrom-Json $_.Content } | % { $_ } | ? { $_.messages -gt 0} | % {
iwr -method DELETE -Credential $cred -uri $("http://localhost:15672/api/queues/{0}/{1}" -f [System.Web.HttpUtility]::UrlEncode($_.vhost), $_.name)
}
You can use rabbitmqctl eval as below:
rabbitmqctl eval 'IfUnused = false, IfEmpty = true, MatchRegex =
<<"^prefix-">>, [rabbit_amqqueue:delete(Q, IfUnused, IfEmpty) || Q <-
rabbit_amqqueue:list(), re:run(element(4, element(2, Q)), MatchRegex)
=/= nomatch ].'
The above will delete all empty queues in all vhosts that have a name
beginning with "prefix-".
You can edit the variables IfUnused, IfEmpty,
and MatchRegex as per your requirement.
Removing all queues using rabbitmqctl one liner
rabbitmqctl list_queues | awk '{ print $1 }' | sed 's/Listing//' | xargs -L1 rabbitmqctl purge_queue
You need not reset rabbitmq server to delete non-durable queues. Simply stop the server and start again and it will remove all the non-durable queues available.
In case you only want to purge the queues which are not empty (a lot faster):
rabbitmqctl list_queues | awk '$2!=0 { print $1 }' | sed 's/Listing//' | xargs -L1 rabbitmqctl purge_queue
For me, it takes 2-3 seconds to purge a queue (both empty and non-empty ones), so iterating through 50 queues is such a pain while I just need to purge 10 of them (40/50 are empty).
I tried rabbitmqctl and reset commands but they are very slow.
This is the fastest way I found (replace your username and password):
#!/bin/bash
# Stop on error
set -eo pipefail
USER='guest'
PASSWORD='guest'
curl -sSL -u $USER:$PASSWORD http://localhost:15672/api/queues/%2f/ | jq '.[].name' | sed 's/"//g' | xargs -L 1 -I# curl -XDELETE -sSL -u $USER:$PASSWORD http://localhost:15672/api/queues/%2f/#
# To also delete exchanges uncomment next line
# curl -sSL -u $USER:$PASSWORD http://localhost:15672/api/exchanges/%2f/ | jq '.[].name' | sed 's/"//g' | xargs -L 1 -I# curl -XDELETE -sSL -u $USER:$PASSWORD http://localhost:15672/api/exchanges/%2f/#
Note: This only works with the default vhost /
I tried the above pieces of code but I did not do any streaming.
sudo rabbitmqctl list_queues | awk '{print $1}' > queues.txt; for line in $(cat queues.txt); do sudo rabbitmqctl delete_queue "$line"; done.
I generate a file that contains all the queue names and loops through it line by line to the delete them. For the loops, while read ... did not do it for me. It was always stopping at the first queue name.
Here is a faster version (using parallel install sudo apt-get install parallel) expanding on the excellent answer by #admenva
parallel -j 50 rabbitmqadmin -H YOUR_HOST_OR_LOCALHOST -q delete queue name={} ::: $(rabbitmqadmin -H YOUR_HOST_OR_LOCALHOST -f tsv -q list queues name)
This commands deletes all your queues
python rabbitmqadmin.py \
-H YOURHOST -u guest -p guest -f bash list queues | \
xargs -n1 | \
xargs -I{} \
python rabbitmqadmin.py -H YOURHOST -u guest -p guest delete queue name={}
This script is super simple because it uses -f bash, which outputs the queues as a list.
Then we use xargs -n1 to split that up into multiple variables
Then we use xargs -I{} that will run the command following, and replace {} in the command.
To list queues,
./rabbitmqadmin -f tsv -q list queues
To delete a queue,
./rabbitmqadmin delete queue name=name_of_queue
For whose have a problem with installing rabbitmqadmin, You should firstly install python.
UNIX-like operating system users need to copy rabbitmqadmin to a directory in PATH, e.g. /usr/local/bin.
Windows users will need to ensure Python is on their PATH, and invoke rabbitmqadmin as python.exe rabbitmqadmin.
Then
Browse to http://{hostname}:15672/cli/rabbitmqadmin to download.
Go to the containing folder then run cmd with administrator privilege
To list Queues
python rabbitmqadmin list queues.
To delete Queue
python rabbitmqadmin delete queue name=Name_of_queue
To Delete all Queues
1- Declare Policy
python rabbitmqadmin declare policy name='expire_all_policies' pattern=.* definition={\"expires\":1} apply-to=queues
2- Remove the policy
python rabbitmqadmin delete policy name='expire_all_policies'
Following command worked for me:
sudo rabbitmqctl list_queues | awk '{print $1}' | xargs -I qn sudo rabbitmqctl delete_queue qn
There's a way to remove all queues and exchanges without scripts and full reset. You can just delete and re-create a virtual host from admin interface. This will work even for vhost /.
The only thing you'll need to restore is permissions for the newly created vhost.
Okay, important qualifier for this answer:
The question does ask to use either rabbitmqctl OR rabbitmqadmin to solve this, my answer needed to use both. Also, note that this was tested on MacOS 10.12.6 and the versions of the rabbitmqctl and rabbitmqadmin that are installed when installing rabbitmq with Homebrew and which is identified with brew list --versions as rabbitmq 3.7.0
rabbitmqctl list_queues -p <VIRTUAL_HOSTNAME> name | sed 1,2d | xargs -I qname rabbitmqadmin --vhost <VIRTUAL_HOSTNAME> delete queue name=qname
Another option is to delete the vhost associated with the queues. This will delete everything associated with the vhost, so be warned, but it is easy and fast.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
This is a method I use. It is easy, clear and effective.
This is the document:
Vhost=the_vhost_name
User=user_name
Password=the_passworld
for i in `rabbitmqctl list_queues -p $Vhost | awk '{ print $1 }'`
do
echo "queu_name: $i"
curl -u $User:$Passworld -H "content-type:application/json" -XDELETE http://localhost:15672/api/queues/$Vhost/$i
done
Try this:
rabbitmqctl list_queues -q name > q.txt
IFS=$'\n' read -d '' -r -a queues < q.txt
count=${#queues[#]}
i=1; while (($i < $count)); do echo ${queues[$i]};rabbitmqctl delete_queue ${queues[$i]};i=$((i+1)); done
As per https://stackoverflow.com/a/52002145/3278855
To automate that, it's possible to use this curl:
curl -X PUT --data '{"pattern":".*","apply-to":"all","definition":{"expires":1},"priority":0}' -u guest:guest 'http://localhost:15672/api/policies/%2f/clear' && \
curl -X DELETE -u guest:guest 'http://localhost:15672/api/policies/%2f/clear'
Please note that %2f is default vhost name (/) and guest:guest is login:password
rabbitmqadmin list queues|awk 'NR>3{print $4}'|head -n-1|xargs -I qname rabbitmqadmin delete queue name=qname