I am new to rabbit and trying to understand if it can answer our use case.
Is there an option to consume batch of messages and control on the way the batch messages are consolidated?
For example, I want to consume all messages with some property but this property depends on other conditions and changes.
Let's say I have many tasks to acquire data and those tasks have start and end date, for decreasing the amount of actions in the consumer I want to consume message that their time span overlap and run one action for all tasks in the consumer.
Thanks
Related
RabbitMQ supports message priority: https://www.rabbitmq.com/priority.html
MassTransit allows user to set this up when configuring endpoints and when sending/publishing a message.
Question: Would it be possible to set a message priority when using a Routing Slip in MassTransit?
My Problem: We have a screen that can schedule items or process them right away. If scheduled, items can be processed in batches. If hundreds of items are processed at the same time, saving a record on the screen can take minutes because the message would go to the end of the queue, which can lead to a bad user experience.
So, if it's not possible to set the priority, what is the alternative here?
Thanks!
Your easiest option? Setup your activity services so that they host two endpoints, one for execute (anything, including batch) and one for execute-interactive, that you use when it is an interactive request. When you build the routing slip, use the appropriate queues for the activity execution, and you're off and running. Batch won't interfere because it's on a separate set of endpoints.
Your other option is a lot harder, and would involve creating send middleware that looks for RoutingSlip and checks some value and sets the priority.
I have tasks with an ids, and each task has some amount of jobs to do:
every job for each user;
for each task, number of users and jobs are different.
And I want to put all the jobs to one queue with name task{id}, so I can control when the task is done (by empty queue task{id}), and automatically delete it with rabbitMQ help, and control a number of consumers working on one task{id}.
And I want my consumers works all the time like daemons and choose the queues with jobs automatically to work with.
The main question here is how to get name of a task inside a consumers to bind them to it?
Or maybe there can be some another trick with rabbitMQ to do so, without knowing name of a queue?
You need to use the Event Exchange Plugin , this plugin allows you to consumes internal events and re-publishes them to a topic exchange, thus exposing the events to clients (applications).
You can bind to queue.created Event which will give you the name of the queue in the message header and you can then use it to bind your consumer to that specific queue
I want to implement a priority work queue, in which the priority of a group of messages can change once they are in the queue. Since it is a work queue with variable processing time, the messages are not assigned using round-robin algorithm, but are pulled from the queue when a resource is free (using per-consumer limit).
I came up with 2 ideas for implementation:
Use priority queue from RabbitMQ, and when a request for priority change comes, read messages with this priority from the queue and re-send them with different priority. (I am not sure this is a good approach, given the O(n) complexity.)
Use several queues with distinct names for each group of messages, and use a separate queue to communicate the current priority list (ordered list of queue names) to workers. (Using this approach, I am not sure how to make the list of priorities "persistent", so that newly joined worker knows what is the current priority list.)
How would you implement it? Is RabbitMQ viable option for this use case?
your idea "priority of a message can change once they are in the queue" IMO is not possible with rabbitmq because rabbitmq only allows you to get messages from the head of a queue.
for example:
you have N queues each used for a different priority
each queue has 100+ messages
your idea requires you to reach into the middle of a queue to get a specific message but this is not possible with rabbitmq so the thought experiment stops here because you can only get messages at the head of a queue
your idea IMO would require using something else besides rabbitmq.
a quick and dirty idea that would work with rabbitmq now and is similar to your idea:
create one rabbitmq queue with N priorities
submit a message with priority x
if you need to change the priority to higher priority like priority y then you could send the same message again but with a new higher priority y
this would ensure the new message is processed faster
the side effect is that you may process the same request twice
you could fix the side effect in your design by having a some sort of database for synchronization to keep track of what jobs are completed and then this could avoid processing the job twice
there are many other details that would need to be addressed like keeping the original message around somehow outside of rabbitmq, concurrency, etc, etc,
I'm positive I'm missing a nuance of MassTranist and/or RabbitMQ, but how long do durable (permanent?) messages stay on queues?
The situation I'm thinking of is one in which all consumers of a certain type of event are unavailable - obviously when they come back up, you want them to be able to take the appropriate actions based on the events they "missed" while they were offline.
However, what about the case when a new consumer starts reading off of the same queue after days/months/years? Is that consumer now going to be pulling in all events since the beginning of time? I'm almost certain that's not the case, but how is durability balanced with timeliness?
As I know MassTransit doesn't control message lifetime. RabbitMQ doing the same, thus message will stay in queue forever. The only exception from this is request/response model in which you can set up timeout period in which you want accept response.
In common way if you need to control lifetime you can store creating time in the message and check it in consumers.
I'm new to RabbitMQ and I'm wondering how to implement the following: producer creates tasks for multiple sites, there's a bunch of consumers that should process these tasks one by one, but only talking to 1 site with concurrency of 1, without starting a new task for this site before the previous one ended. This way slow site would be processed slowly, and the fast ones - fast (as opposed by slow sites taking up all the worker capacity).
Ideally a site would be processed only by one worker at a time, being replaced by another worker if it dies. This seems like a task for exclusive queues, but apparently there's no easy way to list and subscribe to new queues. What is the proper way to achieve such results with RabbitMQ?
I think you may have things the wrong way round. For workers you have 1 or more producers sending to 1 exchange. The exchange has 1 queue (you can send directly to the queue, but all that is really doing is going via a default exchange, I prefer to be explicit). All consumers connect to the single queue and read off tasks in turn. You should set the queue to require messages to be ACKed before removing them. That way if a process dies it should be returned to the queue and picked up by the next consumer/worker.