Delete all the queues from RabbitMQ? - rabbitmq

I installed rabbitmqadmin and was able to list all the exchanges and queues. How can I use rabbitmqadmin or rabbitmqctl to delete all the queues.

First, list your queues:
rabbitmqadmin list queues name
Then from the list, you'll need to manually delete them one by one:
rabbitmqadmin delete queue name='queuename'
Because of the output format, doesn't appear you can grep the response from list queues. Alternatively, if you're just looking for a way to clear everything (read: reset all settings, returning the installation to a default state), use:
rabbitmqctl stop_app
rabbitmqctl reset # Be sure you really want to do this!
rabbitmqctl start_app

Actually super easy with management plugin and policies:
Goto Management Console (localhost:15672)
Goto Admin tab
Goto Policies tab(on the right side)
Add Policy
Fill Fields
Virtual Host: Select
Name: Expire All Policies(Delete Later)
Pattern: .*
Apply to: Queues
Definition: expires with value 1 (change type from String to Number)
Save
Checkout Queues tab again
All Queues must be deleted
And don't forget to remove policy!!!!!!.

With rabbitmqadmin you can remove them with this one-liner:
rabbitmqadmin -f tsv -q list queues name | while read queue; do rabbitmqadmin -q delete queue name=${queue}; done

In Rabbit version 3.7.10 you can run below command with root permission:
rabbitmqctl list_queues | awk '{ print $1 }' | xargs -L1 rabbitmqctl delete_queue

Try this:
rabbitmqadmin list queues name | awk '{print $2}' | xargs -I qn rabbitmqadmin delete queue name=qn

If you don't have rabbitmqadmin installed, try to purge queues with rabbitmqctl:
rabbitmqctl list_queues | awk '{ print $1 }' | xargs -L1 rabbitmqctl purge_queue

If you're trying to delete queues because they're unused and you don't want to reset, one option is to set the queue TTL very low via a policy, wait for the queues to be auto-deleted once the TTL is passed and then remove the policy (https://www.rabbitmq.com/ttl.html).
rabbitmqctl.bat set_policy delq ".*" '{"expires": 1}' --apply-to queues
To remove the policy
rabbitmqctl clear_policy delq
Note that this only works for unused queues
Original info here: http://rabbitmq.1065348.n5.nabble.com/Deleting-all-queues-in-rabbitmq-td30933.html

I made a deleteRabbitMqQs.sh, which accepts arguments to search the list of queues for, selecting only ones matching the pattern you want. If you offer no arguments, it will delete them all! It shows you the list of queues its about to delete, letting you quit before doing anything destructive.
for word in "$#"
do
args=true
newQueues=$(rabbitmqctl list_queues name | grep "$word")
queues="$queues
$newQueues"
done
if [ $# -eq 0 ]; then
queues=$(rabbitmqctl list_queues name | grep -v "\.\.\.")
fi
queues=$(echo "$queues" | sed '/^[[:space:]]*$/d')
if [ "x$queues" == "x" ]; then
echo "No queues to delete, giving up."
exit 0
fi
read -p "Deleting the following queues:
${queues}
[CTRL+C quit | ENTER proceed]
"
while read -r line; do
rabbitmqadmin delete queue name="$line"
done <<< "$queues"
If you want different matching against the arguments you pass in, you can alter the grep in line four. When deleting all queues, it won't delete ones with three consecutive spaces in them, because I figured that eventuality would be rarer than people who have rabbitmqctl printing its output out in different languages.
Enjoy!

Here is a way to do it with PowerShell. the URL may need to be updated
$cred = Get-Credential
iwr -ContentType 'application/json' -Method Get -Credential $cred 'http://localhost:15672/api/queues' | % {
ConvertFrom-Json $_.Content } | % { $_ } | ? { $_.messages -gt 0} | % {
iwr -method DELETE -Credential $cred -uri $("http://localhost:15672/api/queues/{0}/{1}" -f [System.Web.HttpUtility]::UrlEncode($_.vhost), $_.name)
}

You can use rabbitmqctl eval as below:
rabbitmqctl eval 'IfUnused = false, IfEmpty = true, MatchRegex =
<<"^prefix-">>, [rabbit_amqqueue:delete(Q, IfUnused, IfEmpty) || Q <-
rabbit_amqqueue:list(), re:run(element(4, element(2, Q)), MatchRegex)
=/= nomatch ].'
The above will delete all empty queues in all vhosts that have a name
beginning with "prefix-".
You can edit the variables IfUnused, IfEmpty,
and MatchRegex as per your requirement.

Removing all queues using rabbitmqctl one liner
rabbitmqctl list_queues | awk '{ print $1 }' | sed 's/Listing//' | xargs -L1 rabbitmqctl purge_queue

You need not reset rabbitmq server to delete non-durable queues. Simply stop the server and start again and it will remove all the non-durable queues available.

In case you only want to purge the queues which are not empty (a lot faster):
rabbitmqctl list_queues | awk '$2!=0 { print $1 }' | sed 's/Listing//' | xargs -L1 rabbitmqctl purge_queue
For me, it takes 2-3 seconds to purge a queue (both empty and non-empty ones), so iterating through 50 queues is such a pain while I just need to purge 10 of them (40/50 are empty).

I tried rabbitmqctl and reset commands but they are very slow.
This is the fastest way I found (replace your username and password):
#!/bin/bash
# Stop on error
set -eo pipefail
USER='guest'
PASSWORD='guest'
curl -sSL -u $USER:$PASSWORD http://localhost:15672/api/queues/%2f/ | jq '.[].name' | sed 's/"//g' | xargs -L 1 -I# curl -XDELETE -sSL -u $USER:$PASSWORD http://localhost:15672/api/queues/%2f/#
# To also delete exchanges uncomment next line
# curl -sSL -u $USER:$PASSWORD http://localhost:15672/api/exchanges/%2f/ | jq '.[].name' | sed 's/"//g' | xargs -L 1 -I# curl -XDELETE -sSL -u $USER:$PASSWORD http://localhost:15672/api/exchanges/%2f/#
Note: This only works with the default vhost /

I tried the above pieces of code but I did not do any streaming.
sudo rabbitmqctl list_queues | awk '{print $1}' > queues.txt; for line in $(cat queues.txt); do sudo rabbitmqctl delete_queue "$line"; done.
I generate a file that contains all the queue names and loops through it line by line to the delete them. For the loops, while read ... did not do it for me. It was always stopping at the first queue name.

Here is a faster version (using parallel install sudo apt-get install parallel) expanding on the excellent answer by #admenva
parallel -j 50 rabbitmqadmin -H YOUR_HOST_OR_LOCALHOST -q delete queue name={} ::: $(rabbitmqadmin -H YOUR_HOST_OR_LOCALHOST -f tsv -q list queues name)

This commands deletes all your queues
python rabbitmqadmin.py \
-H YOURHOST -u guest -p guest -f bash list queues | \
xargs -n1 | \
xargs -I{} \
python rabbitmqadmin.py -H YOURHOST -u guest -p guest delete queue name={}
This script is super simple because it uses -f bash, which outputs the queues as a list.
Then we use xargs -n1 to split that up into multiple variables
Then we use xargs -I{} that will run the command following, and replace {} in the command.

To list queues,
./rabbitmqadmin -f tsv -q list queues
To delete a queue,
./rabbitmqadmin delete queue name=name_of_queue

For whose have a problem with installing rabbitmqadmin, You should firstly install python.
UNIX-like operating system users need to copy rabbitmqadmin to a directory in PATH, e.g. /usr/local/bin.
Windows users will need to ensure Python is on their PATH, and invoke rabbitmqadmin as python.exe rabbitmqadmin.
Then
Browse to http://{hostname}:15672/cli/rabbitmqadmin to download.
Go to the containing folder then run cmd with administrator privilege
To list Queues
python rabbitmqadmin list queues.
To delete Queue
python rabbitmqadmin delete queue name=Name_of_queue
To Delete all Queues
1- Declare Policy
python rabbitmqadmin declare policy name='expire_all_policies' pattern=.* definition={\"expires\":1} apply-to=queues
2- Remove the policy
python rabbitmqadmin delete policy name='expire_all_policies'

Following command worked for me:
sudo rabbitmqctl list_queues | awk '{print $1}' | xargs -I qn sudo rabbitmqctl delete_queue qn

There's a way to remove all queues and exchanges without scripts and full reset. You can just delete and re-create a virtual host from admin interface. This will work even for vhost /.
The only thing you'll need to restore is permissions for the newly created vhost.

Okay, important qualifier for this answer:
The question does ask to use either rabbitmqctl OR rabbitmqadmin to solve this, my answer needed to use both. Also, note that this was tested on MacOS 10.12.6 and the versions of the rabbitmqctl and rabbitmqadmin that are installed when installing rabbitmq with Homebrew and which is identified with brew list --versions as rabbitmq 3.7.0
rabbitmqctl list_queues -p <VIRTUAL_HOSTNAME> name | sed 1,2d | xargs -I qname rabbitmqadmin --vhost <VIRTUAL_HOSTNAME> delete queue name=qname

Another option is to delete the vhost associated with the queues. This will delete everything associated with the vhost, so be warned, but it is easy and fast.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

This is a method I use. It is easy, clear and effective.
This is the document:
Vhost=the_vhost_name
User=user_name
Password=the_passworld
for i in `rabbitmqctl list_queues -p $Vhost | awk '{ print $1 }'`
do
echo "queu_name: $i"
curl -u $User:$Passworld -H "content-type:application/json" -XDELETE http://localhost:15672/api/queues/$Vhost/$i
done

Try this:
rabbitmqctl list_queues -q name > q.txt
IFS=$'\n' read -d '' -r -a queues < q.txt
count=${#queues[#]}
i=1; while (($i < $count)); do echo ${queues[$i]};rabbitmqctl delete_queue ${queues[$i]};i=$((i+1)); done

As per https://stackoverflow.com/a/52002145/3278855
To automate that, it's possible to use this curl:
curl -X PUT --data '{"pattern":".*","apply-to":"all","definition":{"expires":1},"priority":0}' -u guest:guest 'http://localhost:15672/api/policies/%2f/clear' && \
curl -X DELETE -u guest:guest 'http://localhost:15672/api/policies/%2f/clear'
Please note that %2f is default vhost name (/) and guest:guest is login:password

rabbitmqadmin list queues|awk 'NR>3{print $4}'|head -n-1|xargs -I qname rabbitmqadmin delete queue name=qname

Related

How to remove non-running nodes from RabbitMQ cluster

I need to delete all nodes that are not running in a RabbitMQ cluster via the command line.
I have tried rabbitmqctl forget_cluster_node, but I'm not sure how to get the list of non-running nodes.
I see all the nodes and running_nodes in the output of rabbitmqctl cluster_status. Can someone help me parse it and let me know if there is any other solution to delete the nodes from a cluster easily?
Figured out by myself
# Remove nodes that are not running from the cluster
nodes=($(egrep -o '[a-z0-9#-]+' <<< $(sudo rabbitmqctl cluster_status --formatter json | jq .nodes.disc)))
running_nodes=($(egrep -o '[a-z0-9#-]+' <<< $(sudo rabbitmqctl cluster_status --formatter json | jq .running_nodes)))
for node in ${nodes[#]}
do
match_count=0
for rnode in ${running_nodes[#]}
do
if [ "${node}" == "${rnode}" ]
then
match_count=1
break
fi
done
if [ $match_count == 1 ]
then
continue
else
sudo rabbitmqctl forget_cluster_node $node
fi
done
In my case the proposed script didn't worked, possibly due RabbitMQ version output differs on the version I'm using(3.9.13). Anyhow, this is what I ended up using:
#!/bin/bash
offline_nodes=$(rabbitmqctl --quiet --formatter json cluster_status \
| jq -r '.disk_nodes-.running_nodes | .[]')
for node in ${offline_nodes[#]}; do
rabbitmqctl forget_cluster_node "$node"
done

How to delete rabbitmq shovels through commandline

I created shovels using ,
rabbitmqctl set_parameter shovel "MyShovel" ......
Now i want to delete it,
Can you please tell me on how to achieve it using commandline.
I tried this and it works more or less instantly:
rabbitmqctl clear_parameter shovel MyShovel
You can also list and delete shovels from the command line using rabbitmqadmin
Listing Shovels command
rabbitmqadmin --host=HOST --port=443 --ssl --vhost=/ --username=USERNAME --password=PASSWORD -f tsv -q list parameters name component
Delete a Shovel command
rabbitmqadmin --host=HOST --port=443 --ssl --vhost=/ --username=USERNAME --password=PASSWORD -f tsv -q delete parameter component=shovel name=nameOfShovel

Use qdel to delete all my jobs at once, not one at a time

This is a rather simple question but I haven't been able to find an answer.
I have a large number of jobs running in a cluster (>20) and I'd like to delete them all and start over.
According to this site I should be able to just do:
qdel -u netid
to get rid of them all, but in my case that returns:
qdel: invalid option -- 'u'
usage: qdel [{ -a | -c | -p | -t | -W delay | -m message}] [<JOBID>[<JOBID>]|'all'|'ALL']...
-a -c, -m, -p, -t, and -W are mutually exclusive
which obviously indicates that the command does not work.
Just to check, I did:
qstat -u <username>
and I do get a list of all my jobs, but:
qdel -u <username>
also fails.
Found the answer buried in an old supercluster.org thread:
qselect -u <username> | xargs qdel
Worked flawlessly.
Building on what Gabriel answered:
qselect -u <username> | xargs qdel
qselect -u <username> -s <state> | xargs qdel
<state> would be R for running jobs only.
qselect will allow you to select job based on other criterias, like ressources asked (-l), destination queue (-q) ...
qdel -u <username>
will only work with SGE
sometimes a simple grep/cut can help too:
qstat | grep $USER | cut -d. -f1 | xargs qdel
This way we can also grep on a particular keyword for the jobs and delete them.
HTH
Try
$ qdel {id1..id2}
So for example:
$ qdel {1148613..1148650}
For UGE:
qstat -u | gawk '{print $1}' | xargs qdel
# Delete all jobs owned by the current user.
#
# Command breakdown:
# ------------------
#
# qselect
# -u selects all jobs that belong to the current user
# -s EHQRTW selects all job states except for Complete
#
# xargs
# --no-run-if-empty Do not run qdel if the result set is empty
# to avoid triggering a usage error.
#
# qdel
# -a delete jobs asynchronously
#
# The backslashes are a trick to avoid matching any shell aliases.
\qselect -u $(whoami) -s EHQRTW | \xargs --no-run-if-empty \qdel -a
Another possibility is to do qdel all. It deletes all jobs from everyone. When you don't have access for other people's job, it deletes only your jobs.
It is not the most beautiful solution, but it is surely the shortest!
qstat | cut -d. -f1 | sed "s; \(.*\) 0;qdel \1;" | bash
sed's power.
Just use the following command:
qdel all
It will cancel all jobs running on cluster.

Copy all keys from one db to another in redis

Instade of move I want to copy all my keys from a particular db to another.
Is it possible in redis if yes than how ?
If you can't use MIGRATE COPY because of your redis version (2.6) you might want to copy each key separately which takes longer but doesn't require you to login to the machines themselves and allows you to move data from one database to another.
Here's how I copy all keys from one database to another (but without preserving ttls)
#set connection data accordingly
source_host=localhost
source_port=6379
source_db=0
target_host=localhost
target_port=6379
target_db=1
#copy all keys without preserving ttl!
redis-cli -h $source_host -p $source_port -n $source_db keys \* | while read key; do
echo "Copying $key"
redis-cli --raw -h $source_host -p $source_port -n $source_db DUMP "$key" \
| head -c -1 \
| redis-cli -x -h $target_host -p $target_port -n $target_db RESTORE "$key" 0
done
Keys are not going to be overwritten, in order to do that, delete those keys before copying or simply flush the whole target database before starting.
Copies all keys from database number 0 to database number 1 on localhost.
redis-cli --scan | xargs redis-cli migrate localhost 6379 '' 1 0 copy keys
If you use the same server/port you will get a timeout error but the keys seem to copy successfully anyway. GitHub Redis issue #1903
redis-cli -a $source_password -p $source_port -h $source_ip keys /*| while read key;
do echo "Copying $key";
redis-cli --raw -a $source_password -h $source_ip -p $source_port -n $dbname DUMP "$key"| head -c -1| redis-cli -x -a $destination_password -h $destination_IP -p $destination_port RESTORE "$key" 0;
Latest solution:
Use the RIOT open-source command line tool provided by Redislabs to copy the data.
Reference: https://developer.redis.com/riot/riot-redis/cookbook.html#_performing_migration
GitHub project link: https://github.com/redis-developer/riot
How to install: https://developer.redis.com/riot/riot-redis/
# Source Redis db
SH=test1-redis.com
SP=6379
# Target Redis db
TH=test1-redis.com
TP=6379
# Copy from db0 to db1 (standalone Redis db, Or cluster mode disabled)
#
riot-redis -h $SH -p $SP --db 0 replicate -h $TH -p $TP --db 1 --batch 10000 \
--scan-count 10000 \
--threads 4 \
--reader-threads 4 \
--reader-batch 500 \
--reader-queue 2000 \
--reader-pool 4
RIOT is quicker, supports multithreading, and works well with cross-environment Redis data copy ( AWS Elasticache, Redis OSS, and Redislabs ).
Not directly. I would suggest to use the always convenient redis-rdb-tools package (from Sripathi Krishnan) to extract the data from a normal rdb dump, and reinject it to another instance.
See https://github.com/sripathikrishnan/redis-rdb-tools
As far as I understand you need to copy keys from a particular DB (e.g 5 ) to a particular DB say 10. If that is the case you can use redis database dumper (https://github.com/r043v/rdd). Although as per documentation it has a switch (-d) to select a database for operation but didn't work for me, so what I did
1.) Edit the rdd.c file and look for int main(int argc,char argv) function
2.) Change the DB to as per your requirement
3.) compile the src by **make
4.) Dump all keys using ./rdd -o "save.rdd"
5.) Edit the rdd.c file again and change the DB
6.) Make again
7.) Import by using ./rdd "save.rdd" -o insert -s "IP" -p"Port"
I know this is old, but for those of you coming here form Google:
I just published a command line interface utility to npm and github that allows you to copy keys that match a given pattern (even *) from one Redis database to another.
You can find the utility here:
https://www.npmjs.com/package/redis-utils-cli
Try using dump to first dump all the keys and then restore the same
If migrating keys inside of the same redis engine, then you might use internal command MOVE for that (pipelining for more speed):
#!/bin/bash
#set connection data accordingly
source_host=localhost
source_port=6379
source_db=4
target_db=0
total=$(redis-cli -n 4 keys \* | sed 's/^/MOVE /g' | sed 's/$/ '$target_db'/g' | wc -c)
#copy all keys without preserving ttl!
time redis-cli -h $source_host -p $source_port -n $source_db keys \* | \
sed 's/^/MOVE /g' | sed 's/$/ 0/g' | \
pv -s $total | \
redis-cli -h $source_host -p $source_port -n $source_db >/dev/null

docker rabbitmq hostname issue

I am build an image using Dockerfile, and I would like to add users to RabbitMQ right after installation. The problem is that during build hostname of the docker container is different from when I run the resultant image. RabbitMQ loses that user; because of changed hostname it uses another DB.
I connot change /etc/hosts and /etc/hostname files from inside a container, and looks that RabbitMQ is not picking my changes to RABBITMQ_NODENAME and HOSTNAME variables.
The only thing that I found working is running this before starting RabbitMQ broker:
echo "NODENAME=rabbit#localhost" >> /etc/rabbitmq/rabbitmq.conf.d/ewos.conf
But then I will have to run docker image with changed hostname all the time.
docker run -h="localhost" image
Any ideas on what can be done? Maybe the solution is to add users to RabbitMQ not on build but on image run?
Just here is example how to configure from Dockerfile properly:
ENV HOSTNAME localhost
RUN /etc/init.d/rabbitmq-server start ; rabbitmqctl add_vhost /test; /etc/init.d/rabbitmq-server stop
This is remember your config.
Yes, I would suggest to add users when the container runs for the first time.
Instead of starting RabbitMQ directly, you can run a wrapper script that will take care of all the setup, and then start RabbitMQ. If the last step of the wrapper script is a process start, remember that you can use exec so that the new process replaces the script itself.
This is how I did it.
Dockerfile
FROM debian:jessie
MAINTAINER Francesco Casula <fra.casula#gmail.com>
VOLUME ["/var/www"]
WORKDIR /var/www
ENV HOSTNAME my-docker
ENV RABBITMQ_NODENAME rabbit#my-docker
COPY scripts /root/scripts
RUN /bin/bash /root/scripts/os-setup.bash && \
/bin/bash /root/scripts/install-rabbitmq.bash
CMD /etc/init.d/rabbitmq-server start && \
/bin/bash
os-setup.bash
#!/bin/bash
echo "127.0.0.1 localhost" > /etc/hosts
echo "127.0.1.1 my-docker" >> /etc/hosts
echo "my-docker" > /etc/hostname
install-rabbitmq.bash
#!/bin/bash
echo "NODENAME=rabbit#my-docker" > /etc/rabbitmq/rabbitmq-env.conf
echo 'deb http://www.rabbitmq.com/debian/ testing main' | tee /etc/apt/sources.list.d/rabbitmq.list
wget -O- https://www.rabbitmq.com/rabbitmq-release-signing-key.asc | apt-key add -
apt-get update
cd ~
wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.5/rabbitmq-server_3.6.5-1_all.deb
dpkg -i rabbitmq-server_3.6.5-1_all.deb
apt-get install -f -y
/etc/init.d/rabbitmq-server start
sleep 3
rabbitmq-plugins enable amqp_client mochiweb rabbitmq_management rabbitmq_management_agent \
rabbitmq_management_visualiser rabbitmq_web_dispatch webmachine
rabbitmqctl delete_user guest
rabbitmqctl add_user bunny password
rabbitmqctl set_user_tags bunny administrator
rabbitmqctl delete_vhost /
rabbitmqctl add_vhost symfony_prod
rabbitmqctl set_permissions -p symfony_prod bunny ".*" ".*" ".*"
rabbitmqctl add_vhost symfony_dev
rabbitmqctl set_permissions -p symfony_dev bunny ".*" ".*" ".*"
rabbitmqctl add_vhost symfony_test
rabbitmqctl set_permissions -p symfony_test bunny ".*" ".*" ".*"
/etc/init.d/rabbitmq-server restart
IS_RABBIT_INSTALLED=`rabbitmqctl status | grep RabbitMQ | grep "3\.6\.5" | wc -l`
if [ "$IS_RABBIT_INSTALLED" = "0" ]; then
exit 1
fi
IS_RABBIT_CONFIGURED=`rabbitmqctl list_users | grep bunny | grep "administrator" | wc -l`
if [ "$IS_RABBIT_CONFIGURED" = "0" ]; then
exit 1
fi
Don't forget to run the container by specifying the right host with the -h flag:
docker run -h my-docker -it --name=my-docker -v $(pwd)/htdocs:/var/www my-docker
The only thing that helped me was to change default value in rabbitmq-env.conf of MNESIA_BASE property to MNESIA_BASE=/data and I added this command RUN mkdir /data in Dockerfile before starting server and add users.