I am new to rabbitmq and I want to configure it to just store the last value added to a queue. How can I configure this in order to store just the last value
If this isn't posible what queue I can use?
Thanks
RabbitMQ is a queues system, you cannot store only the "last" value. It stores all the messages as FIFO
I'd suggest to start with the basic AMQP concepts:
https://www.rabbitmq.com/getstarted.html
Although what you try to do doesn't seem to fit a standard use of messaging systems like RabbitMQ, I guess you could get the desired behavior by configuring the queue length to 1
The default behaviour for RabbitMQ when a maximum queue length or size is set and the maximum is reached is to drop or dead-letter messages from the front of the queue (i.e. the oldest messages in the queue).
So defining a queue with length 1 means only the latest message is kept.
Related
This morning I found that my rabbitmq instances does not have several queues that are usually there.
What I noticed is a pattern, that the remaining queues are the ones that had consumers attached to those.
The queues that are gone are mostly retry and DLQs. How does one investigate what happened? What do I look for and where?
Update:
This is my queue details :
The expires is your problem there, if the queue has not been used for some time, it will get dropped.
Also I would get rid of the message-ttl unless you want your messages to be dropped after certain amount of time.
More info here: http://www.rabbitmq.com/ttl.html
These are rabbitmq configuration settings you must change, here some more info https://www.rabbitmq.com/parameters.html
This is due to a feature of TTL or Time to Live for a queue .
As shown in this example below
The value against expires correspond to 28 days. So If a queue hasn't had any consumer for 28 days it gets deleted.
It could also be possible that the auto-delete property is set to true in which case the queue will automatically get deleted the moment the last consumer gets disconnected.
You should be able to get both these parameters in the rabbitmq console where you check the queue properties .
I want to implement a priority work queue, in which the priority of a group of messages can change once they are in the queue. Since it is a work queue with variable processing time, the messages are not assigned using round-robin algorithm, but are pulled from the queue when a resource is free (using per-consumer limit).
I came up with 2 ideas for implementation:
Use priority queue from RabbitMQ, and when a request for priority change comes, read messages with this priority from the queue and re-send them with different priority. (I am not sure this is a good approach, given the O(n) complexity.)
Use several queues with distinct names for each group of messages, and use a separate queue to communicate the current priority list (ordered list of queue names) to workers. (Using this approach, I am not sure how to make the list of priorities "persistent", so that newly joined worker knows what is the current priority list.)
How would you implement it? Is RabbitMQ viable option for this use case?
your idea "priority of a message can change once they are in the queue" IMO is not possible with rabbitmq because rabbitmq only allows you to get messages from the head of a queue.
for example:
you have N queues each used for a different priority
each queue has 100+ messages
your idea requires you to reach into the middle of a queue to get a specific message but this is not possible with rabbitmq so the thought experiment stops here because you can only get messages at the head of a queue
your idea IMO would require using something else besides rabbitmq.
a quick and dirty idea that would work with rabbitmq now and is similar to your idea:
create one rabbitmq queue with N priorities
submit a message with priority x
if you need to change the priority to higher priority like priority y then you could send the same message again but with a new higher priority y
this would ensure the new message is processed faster
the side effect is that you may process the same request twice
you could fix the side effect in your design by having a some sort of database for synchronization to keep track of what jobs are completed and then this could avoid processing the job twice
there are many other details that would need to be addressed like keeping the original message around somehow outside of rabbitmq, concurrency, etc, etc,
After reading documentation about what is Rabbit and what does, I have some common questions:
Case is: Producer sends one message to some consumers (subscribers).
I need a short explanation for all points of list below, what to use, and what to dig further.
How to clear queue and stop sending message to consumers after
specific time/date?
Can I include to confirmed message user's data like JSON?
Where is stored this data? In the same queue?
How to filter confirmed messages and then clear queue?
How to clear queue after specific time/date?
What happens if not one consumer no confirms message, how long they are stored?
Does consumer subscribe on queue or can subscribe on exchange too?
Using model one to many for sending message, how to set who have to
get message first/last or at the same time, here described that, but not clear is it on client or server side?
If no consumers, how to re-push message to another queue and close
current?
Each consumer has own queue?
Thank you in advance and any comment to this question!
If you can elaborate some of your questions and include what is your use case, I can edit the answer.
1 - As long as consumer is alive rabbitmq sends incoming messages to consumer. You can give TTL to messages/queues if you want them to expire after some time.
https://www.rabbitmq.com/ttl.html
2 - What you mean?
3 - Rabbitmq stores the data in mnesia database.
https://www.rabbitmq.com/persistence-conf.html
https://www.rabbitmq.com/relocate.html
4 - What you mean by filterig messages and clear queue? Successfully consumed messages removed from the queue immediatly.
5 - You can give ttl to queue or declare queue as auto delete
https://www.rabbitmq.com/ttl.html
https://www.rabbitmq.com/queues.html
6 - If consumers don't send ack to rabbit, messages stays unack as long as memory becomes full or rabbit becomes unavailable
7 - Both. A consumer can create its own queue and bind it to an exchange or it can consume from existing queue. It depends on the use case.
8 - It is hard to answer this without knowing details of what you mean by one-to-many. Direct exchange or fanout or whatelse, how many queues etc.
But, in rabbitmq, messages ordered by publish order by default.
According to link you shared, rabbitmq sends messages first to higher priority consumers until consumer prefetch count (unack messages on consumer) becomes its limits.
9 - You need to handle this case in the code. Or you can use management ui with Shovel plugin.
https://www.rabbitmq.com/management.html
https://www.rabbitmq.com/shovel.html
10 - Again, it depends on the design and use case.
I have a amqp connector setup that listens on a single queue for JSON messages and is working fine. The business has dropped a use case that my application now needs to listen on multiple queues in order of priority. For example having three queues:
HighQ
NormalQ
LowQ
I want the mule connector to first read from HighQ until empty, then NormalQ until empty and LowQ until empty. Restarting from HighQ after every message.
I feel like this should be standard but my google foo is failing me.
Any pointers in the right direction?
In the usecase you specified I think it would be better to go with a single queue, but posting messages with 3 priority levels.
THis way the messages are always read in the order of their priority with the highest prioritymessage are always read first.
So you can make the message producers to post the messages onto the queue with 3 priority levels(say 9 for high , 4 for normal, 0 for low).
You inbound JMS endpoint will read all the messages with priority of 9 first. Then it will read all the messages with priotiy of 4 and then the messages with priority of 0.
Sample JMS Outbound posting messages with priority.
<jms:outbound-endpoint queue="StudioOUT" connector-ref="MyAppJMS" doc:name="JMS">
<set-property propertyName="Priority" value="9"/>
</jms:outbound-endpoint>
I hope this should address your scenario.
More on priority of JMS.
http://www.christianposta.com/blog/?p=289
Dealing with message priority is really something that your broker should handle. Dealing with this yourself can be tricky and cumbersome.
Processing queues sequentially in order to simulate priority seems like a bad idea. Lets say you've processed all messages from the high priority queue and start processing the normal priority queue.
While processing the normal priority queue new messages are coming in on the high priority queue. These high priority messages will be sitting there until both the normal and low priority queues are entirely processed.
You could probably improve your mechanism to handle situations like this a bit better, but it will be hard to make it bullet proof. You really don't want to deal with stuff like this yourself.
The JMS api has the concept of 'message priority' built in, but that's of little use to U if you aren't using a JMS broker.
If you're using rabbit mq then you should have a look at this stackoverflow post: rabbitmq-and-message-priority.
As Rabbit MQ queues are basically FIFO queues there's no easy way to use "real" prioritized message (such as in JMS).
There is however a plugin that claims to provide the functionality that you are looking for: rabbitmq-priority-queue.
According to the documentation the next version of RabbitMQ (3.5.0) will support prioritized queues out of the box.
If using the plugin isn't an option and if the priority of the messages is really important then I would not use the pattern you described using multiple queues. The pattern also doesn't scale very well if more priority levels are needed. I would opt to receive all the messages on a single channel (given that each message has a property that represents the priority level) and forward them to a (non amqp) new channel that handles the resequencing for you. An open-source product that could help you with this is Apache ActiveMQ but there are also other options available.
I have a rabbitmq cluster used as a working queue. There are 5 kinds of consumers who want to consume exactly the same data.
What I know for now is using fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
My question is, does rabbitmq support to push the same data to multi consumers? Just like a message need to be acked for a specified times to be deleted.
I got the following answer from rabbitmq email group. In short, the answer is no... and what I did above is the correct way.
http://rabbitmq.1065348.n5.nabble.com/Does-rabbitmq-support-to-push-the-same-data-to-multi-consumers-td36169.html#a36170
... fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
You can consume with 5 consumers from one queue if you do not want to duplicate messages.
does rabbitmq support to push the same data to multiple consumers
In AMQP protocol terms you publish message to exchange and then broker (RabbitMQ) decide what to do with messages - assume it figured out the queue message intended for (one or more) and then put that message on top of that queue (queues in RabbitMQ are classic FIFO queues which is somehow break AMQP implementation in RabbitMQ). Only after that message may be delivered to consumer (or die due to queue length limit or per-queue or per-message ttl, if any).
message need to be acked for a specified times to be deleted
There are no way to change message body or attributes after message being published (actually, Dead Letter Exchanges extension and some other may change routing key, for example and add,remove and change some headers, but this is very specific case). So if you want to track ack's number you have to re-publish consumed message with changed body or header (depends on where do you plan to store ack's counter, but headers fits pretty nice for this.
Also note, that there are redeliverd message attribute which denotes whether message was already was consumed, but then redelivered. This flag doesn't count redelivers number so it usage is quite limited.