I was taking a look at the queues with old messages and while trying to purge a queue I accidentally deleted deleted from RabbitMQ anyone know away to recover it?
Thanks for the answers and help. I resolved the problem by recreating the queue and binding it to the exchange.
Related
I am new to Active MQ but sometimes the queues are not being processed and keep piling up, Is it a good practice to purge?, Isnt there any other solution that may prevent me from keeping all my messages for reprocessing apart from purging? I really dont want to loose the queues, Is this possible?
The correct way to deal with this is to set an expiration on messages such that after a given time the broker can discard them. Letting messages just pile into queues without regard to their lifetime will lead you into all sorts of problems most notably storage.
You need to develop a strategy for how long the messages should live so that the broker can start getting rid of them once they are no longer of use. If you don't do that then purging the queue is you only option.
I am using activemq 5.5.0 version in my project.
I am facing one problem due to some reason my inflight count keep on increasing ,
Rate of increment is not high.
After some days it will become equal to prefetch size then my queue stops responding.
Could anyone can help me how to make inflight count 0 without deleting the queue ?
You can make the in-flight count go to 0 by purging the queue from the web console (there's a Purge link on each queue on the Queues page); that will delete all messages in the queue without deleting the queue itself.
That will satisfy the question as you've asked it, though I suspect it's not really what you want and what you meant was "without deleting any messages from the queue". But if I'm wrong and this is what you wanted, then great; if not, answer my questions in the comment I added and we'll go from there.
When my system's data changes I publish every single change to at least 4 different consumers (around 3000 messages a second) so I want to use a message broker.
Most of the consumers are responsible to update their database tables with the change.
(The DBs are different - couch, mysql, etc therefor solutions such as using their own replication mechanism or using db triggers is not possible)
questions
Does anyone have an experience with data replication between DBs using a message broker?
is it a good practice?
What do I do in case of failures?
Let's say, using RabbitMQ, the client removed 10,000 messages from the queue, acked, and threw an exception each time before handling them. Now they are lost. Is there a way to go back in the queue?
(re-queueing them will mess their order ).
Is using rabbitMQ a good practice? Isn't the ability to go back in the queue as in Kafka important to fail scenarios?
Thanks.
I don't have experience with DB replication using message brokers, but maybe this can help put you in the right track:
2. What do I do in case of failures?
Let's say, using RabbitMQ, the client removed 10,000 messages from the
queue, acked, and threw an exception each time before handling them.
Now they are lost. Is there a way to go back in the queue?
You can use dead lettering to avoid losing messages. I'd suggest to not ack until you are sure the consumers have processed them successfully, unless it is a long-running task. In case of failure, basic.reject instead of basic.ack to send them to a dead-letter queue. You have a medium throughput, so gotta be careful with that.
However, the order is not guaranteed. You'll need to implement a manual mechanism to recover them in the order they were published, maybe by using message headers with some sort of timestamp or id mechanism, to re-process them in the correct order.
I am using celery on rabbitmq. I have been sending thousands of messages to the queue and they are being processed successfully and everything is working just fine. However, the number of messages in several rabbitmq queues are growing quite large (hundreds of thousands of items in the queue). The queues are named celeryev.[...] (see screenshot below). Is this appropriate behavior? What is the purpose of these queues and shouldn't they be regularly purged? Is there a way to purge them more regularly, I think they are taking up quite a bit of disk space.
You can use the CELERY_EVENT_QUEUE_TTL celery option (only working with amqp), that will set the message expiry time, after which it will be deleted from the queue.
For anyone else who is running into problems with a celeryev queue becoming very large and threatening the disk space on your rabbitmq server, beware the accepted answer! Here's my suggestion. Just issue this command on your rabbitmq instance:
rabbitmqctl set_policy limit_celeryev_queues "^celeryev\." '{"max-length":1000000}' --apply-to queues
This will limit any queue beginning with "celeryev" to 1 Million entries. I did some experimenting with a stuck flower instance causing a runaway celeryev queue, and setting CELERY_EVENT_QUEUE_TTL / CELERY_EVENT_QUEUE_EXPIRES did not help control the queue size.
In my testing, I started a flower process, then SIGSTOP'ed it, and watched its celeryev queue start running away. Neither of these two settings helped at all. I confirmed SIGCONT'ing the flower process would bring the queue back to 0 rapidly. I am not certain why these two knobs didn't help, but it may have something to do with how RabbitMQ implements these two settings.
First, the Per-Message TTL corresponding to CELERY_EVENT_QUEUE_TTL only establishes an expiration time on each queue entry -- AIUI it will not automatically delete the message out of the queue to save space upon expiration. Second, the Queue TTL corresponding to CELERY_EVENT_QUEUE_EXPIRES says that it "... guarantees that the queue will be deleted, if unused for at least the expiration period". However, I believe that their definition of "unused" may be too strict to kick in for e.g. an overburdened, stuck, or killed flower process.
EDIT: Unfortunately, one problem with this suggestion is that the set_policy ... apply-to queues will only impact existing queues, and flower can and will create new queues which may overflow.
Celery use celeryev prefixed queues (and exchange) for monitoring, you can configure it as you want or disable at all (celery control disable_events).
You just have to set a config to your Celery.
If you want to avoid Celery from creating celeryev.* queues:
CELERY_SEND_EVENTS = False # Will not create celeryev.* queues
If you need these queues for monitoring purpose (CeleryFlower for instance), you may regularly purge them:
CELERY_EVENT_QUEUE_EXPIRES = 60 # Will delete all celeryev. queues without consumers after 1 minute.
The solution came from here: https://www.cloudamqp.com/docs/celery.html
You can limit the queue size in RabbitMQ with x-max-length queue declaration argument
http://www.rabbitmq.com/maxlength.html
In the ActiveMQ KahaDB documentation, it mentions that you can archive KahaDB data files so they can be replayed if needed later. Yet, through some searching and looking through their documentation and the draft copy of ActiveMQ in Action, I can't find any example or clues how to actually do the replay of those files.
I'm hoping someone out there can point me in the direction on what needs to be done in order to actually perform a replay.
KahaDB only replays messages/events when a broker is started to return the broker to the state prior to the broker being stopped (recovering persistent messages, etc.)
It does not retain historical messages to be replayed on demand. Once a message is dequeued successfully, then its removed from the KahaDB data files.
If you have such a requirement to copy messages for auditing/reuse, then I suggest look into something like mirrored queues or using the camel wire-tap pattern.