I am sending messages to an Active MQ queue through Mule.
I want that, that only the latest message remains in the queue and not any previous one.
How can this be achieved?
Thank you very much.
set the queue size to be 1, the default eviction policy, which is oldestMessageEvictionStrategy, should kick out the previous message and put in the newest one. You can also set other policies for priorities, such as OldestMessageWithLowestPriorityEvictionStrategy or mess around with the TTL settings on your messages.
You may also get some warnings about full queue though, so be prepared to handle that.
Related
This morning I found that my rabbitmq instances does not have several queues that are usually there.
What I noticed is a pattern, that the remaining queues are the ones that had consumers attached to those.
The queues that are gone are mostly retry and DLQs. How does one investigate what happened? What do I look for and where?
Update:
This is my queue details :
The expires is your problem there, if the queue has not been used for some time, it will get dropped.
Also I would get rid of the message-ttl unless you want your messages to be dropped after certain amount of time.
More info here: http://www.rabbitmq.com/ttl.html
These are rabbitmq configuration settings you must change, here some more info https://www.rabbitmq.com/parameters.html
This is due to a feature of TTL or Time to Live for a queue .
As shown in this example below
The value against expires correspond to 28 days. So If a queue hasn't had any consumer for 28 days it gets deleted.
It could also be possible that the auto-delete property is set to true in which case the queue will automatically get deleted the moment the last consumer gets disconnected.
You should be able to get both these parameters in the rabbitmq console where you check the queue properties .
I've been tasked with investigating why the db-*.log files are not clearing.
From what I have found through vast searching, everything points to the messages being on the queue still. I've looked at hawtio at the queues on all the configured topics and the queue size is zero.
From my understanding the Enqueue size and Dequeue size in theory should be the same, but they're not. Seems my Dequeue size is 0.
I've looked at the topics and there's no operation to purge them.
I'd like to be able to clear out all messages so that the kahadb logs will disappear.
I think you point on one weakness of the ActiveMQ itself: it cannot guarantee the consumers are really strict when consuming the messages.
We have similar problems with our ActiveMQ (5.10.7) because it seems the KahaDB make likes a "disk fragmentation" and we noticed this could be from at least two issues with consumers:
Case 1: Slow consumer
We have in our system a consumer which cannot consume many messages at once. if only one unconsumed message stays in a KahaDB page, it will keep all the whole page (with all others messages which are already consumed, and acknowledged).
For preventing the KahaDB Storage to reach 100% (which will slows the producers) we transfer the messages in another ActiveMQ instance temporary queue like this:
from("activemqPROD:queue:BIG_QUEUE_UNCONSUMED")
.to("activemqTEMP:queue:TEMP_BIG_QUEUE");
then pushing them back:
from("activemqTEMP:queue:TEMP_BIG_QUEUE")
.to("activemqPROD:queue:BIG_QUEUE_UNCONSUMED");
The alternative is to store them on file system then reload them, but you loose the JMS (and custom) headers. With the temporary queue solution you keep all headers.
Case 2: Consumer who never gives acknowledgement
Sometimes even we make the previous operation, even all unconsumed queues are empty, the storage stays higher than 0%.
By looking into the KahaDB file we can see there are still pages present even no more messages in all QUEUES.
For the TOPICS, we stopped using durable subscriptions, then the storage should also stays at 0%.
The potential cause (this is a supposition, but with a strong confidence) is that some of the consumed messages were never acknowledged properly.
The reason we think this is the cause, it is because in the logs, we can still see messages
"not removing data file: 12345 as contained ack(s) refer to referenced file: [12344, 12345]"
This can happens for example when the consumer is disconnecting abruptly (they consumed some messages but disconnect before sending the ack)
In our case the messages never expires, then this could also be a potential issue for this case. However it is not clear if setting an expiration can destroy "non-acked" messages.
Because we do not want to loose any event, there is no expiration time for these specific queues.
According to your question, it looks you are in the second case, then our solution is:
Be sure no more producer / consumer are connecting to the ActiveMQ
Be sure all queues and durable topics are empty
Delete all files in the KahaDB storage (from file system)
Restart ActiveMQ (fresh)
Unfortunately we did not find a better way to manage with these cases, if someone else have a better alternative we would be happy to know it.
This article can also give you some solution (like setting an expiry policy for the ActiveMQ.DLQ queue).
add this log config to log4j.properties. Then you can see exactly what is holding kahadb files in kahadb.log.
log4j.appender.kahadb=org.apache.log4j.RollingFileAppender
log4j.appender.kahadb.file=${activemq.base}/data/kahadb.log
log4j.appender.kahadb.maxFileSize=1024KB
log4j.appender.kahadb.maxBackupIndex=5
log4j.appender.kahadb.append=true
log4j.appender.kahadb.layout=org.apache.log4j.PatternLayout
log4j.appender.kahadb.layout.ConversionPattern=%d [%-15.15t] %-5p %-30.30c{1} - %m%n
log4j.logger.org.apache.activemq.store.kahadb.MessageDatabase=TRACE, kahadb
As alternative: once you've found out which Queue is causing the log to exist, you could map it to its own KahaDB like described here http://activemq.apache.org/kahadb.html
I'm currently using ActiveMQ for my queueing system, and I'm wanting to make the transition to RabbitMQ. One feature I've been using that belongs to ActiveMQ is a redelivery policy, as sometimes our consumer rejects a message because it cannot handle it at this time, but may want to try again later, so it requeues it.
Right now in AMQP, when I reject a message, it's instantly pulled off the queue again immediately and tried again.
Is there a way, in RabbitMQ, to specify a redelivery policy for a queue, consumer, or message?
I also had problems with that behaviour. According to documentation (as far as I remember, maybe in newer version something changed) after requeue it is not stated where a message will be placed (it was described as undetermined). In my testcases (with version 2.8.2) some of messages were put to the end of a queue and one message (precisely first from clients prefetch) land on beggining (and being consumed immediately). In our application this caused livelock.
You could walkaround this by publishing copy of message to a queue and acking already delivered one in one transaction (but I recommend to carefully read section about transactions in docs) or use deadlettering to deal with temporaly unprocessable messages.
After I purged the message of a queue (not a topic), the pending message number of this queue in Active MQ Admin console changed to negative.
I assumed the pending message number of any queue ought to be positive. Is it issue of purging? Or is it a bug of activemq?
It seems to be a bug in ActiveMQ. I've personally witnessed this bug on numerous occasions when purging a queue. It does not seem to affect actual message delivery however.
There are a number of open issues in the ActiveMQ issue tracker related to this problem. You can vote them up if you want:
https://issues.apache.org/jira/browse/AMQ-3472
https://issues.apache.org/jira/browse/AMQ-3111
This problem is solved in ActiveMQ 5.8 :
https://issues.apache.org/jira/browse/AMQ-4598
Purging doesnt stop the mq from processing the queued jobs. Restart the ActiveMQ is only getting rid of the existing jobs. The down side to this is, this will make all the numbers reset to default.
I am using RabbitMQ server.
For publishing messages, I set the immediate field to true and tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that the number of messages in the queue was zero.
Then, I changed the immediate flag to false and again tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that a total of 100,000 messages were in queues (till now, no consumer was present).
After that, I started a consumer and it consumed all the 100,000 messages.
Can anybody please help me in understanding about the immediate bit field and this behavior too? Also, I could not understand the concept of the mandatory bit field.
The immediate and mandatory fields are part of the AMQP specification, and are also covered in the RabbitMQ FAQ to clarify how its implementers interpreted their meaning:
Mandatory
This flag tells the server how to
react if a message cannot be routed to
a queue. Specifically, if mandatory is
set and after running the bindings the
message was placed on zero queues then
the message is returned to the sender
(with a basic.return). If mandatory
had not been set under the same
circumstances the server would
silently drop the message.
Or in my words, "Put this message on at least one queue. If you can't, send it back to me."
Immediate
For a message published with immediate
set, if a matching queue has ready
consumers then one of them will have
the message routed to it. If the lucky
consumer crashes before ack'ing
receipt the message will be requeued
and/or delivered to other consumers on
that queue (if there's no crash the
messaged is ack'ed and it's all done
as per normal). If, however, a
matching queue has zero ready
consumers the message will not be
enqueued for subsequent redelivery on
from that queue. Only if all of the
matching queues have no ready
consumers that the message is returned
to the sender (via basic.return).
Or in my words, "If there is at least one consumer connected to my queue that can take delivery of a message right this moment, deliver this message to them immediately. If there are no consumers connected then there's no point in having my message consumed later and they'll never see it. They snooze, they lose."
http://www.rabbitmq.com/blog/2012/11/19/breaking-things-with-rabbitmq-3-0/
Removal of "immediate" flag
What changed? We removed support for the
rarely-used "immediate" flag on AMQP's basic.publish.
Why on earth did you do that? Support for "immediate" made many parts
of the codebase more complex, particularly around mirrored queues. It
also stood in the way of our being able to deliver substantial
performance improvements in mirrored queues.
What do I need to do? If you just want to be able to publish messages
that will be dropped if they are not consumed immediately, you can
publish to a queue with a TTL of 0.
If you also need your publisher to be able to determine that this has
happened, you can also use the DLX feature to route such messages to
another queue, from which the publisher can consume them.
Just copied the announcement here for a quick reference.