I need to enqueue messages based on time. Every message has a "start date" and the message with the earliest start date needs to be at the front of the queue. Is there a way to do this with RabbitMQ?
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
The documentation is clear that priority must be between 0 and 255 (https://www.rabbitmq.com/priority.html). You could use multiple queues but keep in mind that priority queues have a performance cost.
Related
Under the hood, how is a FIFO queue turned into a priority queue in a distributed fashion? Are they actually swapping the underlying datastructure, or is it a "hacked" fix
The underlying data structures are multiple queues, each assigned a priority. Each queue is an Erlang VM process. This is why having more than 10 or so priorities isn't recommended as performance suffers. If your load is light enough, this may be acceptable.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
When a message is published with a priority header, the message with the higher priority value gets placed on the head of the queue. This is done by actually swapping the messages in the queue. This is all done when the message is waiting to be consumed in the queue. In order to allow RabbitMQ to actually prioritise the messages, set the basic.qos of the consumer as low as possible. So if a consumer connects to an empty queue whose basic.qos is not set and to which messages are subsequently published, the messages may not spend any time at all waiting in the queue. In this case, the priority queue will not get any opportunity to prioritise them.
Reference: https://www.rabbitmq.com/priority.html
I need a solution where I can set a minimum delay between the messages that are polled out of the Queue. I do not want to delay every message by a fixed amount of miliseconds.
Lets say the Queue get 3 messages in the first second. But I want to pull every 5 seconds. So my Client does not get overloaded with to many request.
Is there a way to solve this with rabbitmq or do i have to change to some other framework?
Any time you ask for assistance about RabbitMQ (or any software), you must provide information about what versions of software you are using, and what client libraries. That way the people who are assisting you can do so effectively.
Your client should consume from the queue using the basic.consume method. Set the channel's "prefetch" value to the maximum number of unacknowledged messages you wish for that consumer to receive at once (you can set it to 1 if you only want one message at a time). Then, do your work and only acknowledge the messages after the desired amount of time has elapsed.
Be certain that this does not result in messages accumulating in queues. You will monitor your RabbitMQ installation, right?
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I want to implement a priority work queue, in which the priority of a group of messages can change once they are in the queue. Since it is a work queue with variable processing time, the messages are not assigned using round-robin algorithm, but are pulled from the queue when a resource is free (using per-consumer limit).
I came up with 2 ideas for implementation:
Use priority queue from RabbitMQ, and when a request for priority change comes, read messages with this priority from the queue and re-send them with different priority. (I am not sure this is a good approach, given the O(n) complexity.)
Use several queues with distinct names for each group of messages, and use a separate queue to communicate the current priority list (ordered list of queue names) to workers. (Using this approach, I am not sure how to make the list of priorities "persistent", so that newly joined worker knows what is the current priority list.)
How would you implement it? Is RabbitMQ viable option for this use case?
your idea "priority of a message can change once they are in the queue" IMO is not possible with rabbitmq because rabbitmq only allows you to get messages from the head of a queue.
for example:
you have N queues each used for a different priority
each queue has 100+ messages
your idea requires you to reach into the middle of a queue to get a specific message but this is not possible with rabbitmq so the thought experiment stops here because you can only get messages at the head of a queue
your idea IMO would require using something else besides rabbitmq.
a quick and dirty idea that would work with rabbitmq now and is similar to your idea:
create one rabbitmq queue with N priorities
submit a message with priority x
if you need to change the priority to higher priority like priority y then you could send the same message again but with a new higher priority y
this would ensure the new message is processed faster
the side effect is that you may process the same request twice
you could fix the side effect in your design by having a some sort of database for synchronization to keep track of what jobs are completed and then this could avoid processing the job twice
there are many other details that would need to be addressed like keeping the original message around somehow outside of rabbitmq, concurrency, etc, etc,
I need a queue with limited message inside considering not only message in queue but also the unacked ones. Is there a way to configure this server side? If yes is it possible using kobu as library?
Thank you
The RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
The documentation clearly states that queue length uses the count of ready messages: https://www.rabbitmq.com/maxlength.html
I am new to rabbitmq and I want to configure it to just store the last value added to a queue. How can I configure this in order to store just the last value
If this isn't posible what queue I can use?
Thanks
RabbitMQ is a queues system, you cannot store only the "last" value. It stores all the messages as FIFO
I'd suggest to start with the basic AMQP concepts:
https://www.rabbitmq.com/getstarted.html
Although what you try to do doesn't seem to fit a standard use of messaging systems like RabbitMQ, I guess you could get the desired behavior by configuring the queue length to 1
The default behaviour for RabbitMQ when a maximum queue length or size is set and the maximum is reached is to drop or dead-letter messages from the front of the queue (i.e. the oldest messages in the queue).
So defining a queue with length 1 means only the latest message is kept.