I was looking into rabbitmq to build simple reminder application to send emails. I liked idea of priority and thought I can use the timestamp as one but when check the docs it says only can be used number between 1-255.
Is there any other queue I can use to achieve receiving messages based on smallest number of priority but will not have limit as rabbitmq?
Related
I need a solution where I can set a minimum delay between the messages that are polled out of the Queue. I do not want to delay every message by a fixed amount of miliseconds.
Lets say the Queue get 3 messages in the first second. But I want to pull every 5 seconds. So my Client does not get overloaded with to many request.
Is there a way to solve this with rabbitmq or do i have to change to some other framework?
Any time you ask for assistance about RabbitMQ (or any software), you must provide information about what versions of software you are using, and what client libraries. That way the people who are assisting you can do so effectively.
Your client should consume from the queue using the basic.consume method. Set the channel's "prefetch" value to the maximum number of unacknowledged messages you wish for that consumer to receive at once (you can set it to 1 if you only want one message at a time). Then, do your work and only acknowledge the messages after the desired amount of time has elapsed.
Be certain that this does not result in messages accumulating in queues. You will monitor your RabbitMQ installation, right?
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I want to implement a priority work queue, in which the priority of a group of messages can change once they are in the queue. Since it is a work queue with variable processing time, the messages are not assigned using round-robin algorithm, but are pulled from the queue when a resource is free (using per-consumer limit).
I came up with 2 ideas for implementation:
Use priority queue from RabbitMQ, and when a request for priority change comes, read messages with this priority from the queue and re-send them with different priority. (I am not sure this is a good approach, given the O(n) complexity.)
Use several queues with distinct names for each group of messages, and use a separate queue to communicate the current priority list (ordered list of queue names) to workers. (Using this approach, I am not sure how to make the list of priorities "persistent", so that newly joined worker knows what is the current priority list.)
How would you implement it? Is RabbitMQ viable option for this use case?
your idea "priority of a message can change once they are in the queue" IMO is not possible with rabbitmq because rabbitmq only allows you to get messages from the head of a queue.
for example:
you have N queues each used for a different priority
each queue has 100+ messages
your idea requires you to reach into the middle of a queue to get a specific message but this is not possible with rabbitmq so the thought experiment stops here because you can only get messages at the head of a queue
your idea IMO would require using something else besides rabbitmq.
a quick and dirty idea that would work with rabbitmq now and is similar to your idea:
create one rabbitmq queue with N priorities
submit a message with priority x
if you need to change the priority to higher priority like priority y then you could send the same message again but with a new higher priority y
this would ensure the new message is processed faster
the side effect is that you may process the same request twice
you could fix the side effect in your design by having a some sort of database for synchronization to keep track of what jobs are completed and then this could avoid processing the job twice
there are many other details that would need to be addressed like keeping the original message around somehow outside of rabbitmq, concurrency, etc, etc,
I'm evaluating using RabbitMQ as a message queue broker/framework to replace an internally built message queue (using C# if that matters).
I will have a service with N threads, each thread being a consumer for a specific queue. There may be more than one thread that shares the same queue. I believe I would use the prefetch property of 1 for each consumer so that a consumer thread receives 1 message at a time.
Lets use an example where I have 4 consumer threads and all look at a queue named "reports". These reports can be run for different "customers". I want to avoid allowing a customer to monopolize the queue so lets say I don't want any single customer to use more than 2 consumers at any given time (if there are other queued messages waiting). If there are no other queued messages waiting for other customers then I'd like to allow all consumers to be eligible.
With my limited RabbitMQ understanding so far, I believe that I could either use a topic pattern to indicate the customer or I could use a custom header.
I'd like to know if there's a design pattern to support defining a limit per unique header/customer value.
I'm not asking for anyone to write the code for me, I just want to know if anyone can tell me "no that won't work" in advance before I waste a bunch of time getting ramped up.
If my question doesn't make sense please let me know and I'll update it with more information. Thanks in advance.
In out product we need to count occasionally the number of messages in a queue that hold a specific property value.
The straight forward way is to use a queue browser with a selector. But:
Is there a way to get this statistics without browsing all relevant messages? We don't need the data, only the count.
If we do go for the queue browser approach, will it also browse messages that were consumed by some consumer, but not acknowledged yet?
There is not guaranteed way to count messages in a queue based on a selector. There are a number of reasons for this one of which is that the JMS Spec does not require that a QueueBrowser even return every message on a Queue. ActiveMQ won't return more than the number of messages that it can fit into memory so if you have a deep queue you won't have an accurate count. None of the management tools will do this for you either since it's not really ActiveMQ's job to be a database.
Here's some articles on messaging anti-patterns that you should read so you are better prepared for adventures in messaging.
Article 1 and Article 2
We are currently evaluating RabbitMQ. Trying to determine how best to implement some of our processes as Messaging apps instead of traditional DB store and grab. Here is the scenario. We have a department of users who perform similar tasks. As they submit work to the server applications we would like the server app to send messages back into a notification window saying what was done - to all the users, not just the one submitting the work. This is all easy to do.
The question is we would like these message to live for say 4 hours in the Queue. If a new user logs in or say a supervisor they would get all the messages from the last 4 hours delivered to their notification window. This gives them a quick way to review what has recently happened and what is going on without having to ask others, "have you talked to John?", "Did you email him is itinerary?", etc.
So, how do we publish messages that have a lifetime of x hours from the time they were published AND any new consumers that connect will get all of these messages delivered in chronological order? And preferably the messages just disappear after they have expired from the queue.
Thanks
There is Per-Queue Message TTL and Per-Message TTL in RabbitMQ. If I am right you can utilize them for your task.
In addition to the above answer, it would be better to have the application/client publish messages to two queues. Consumer would consume from one of the queues while the other queue can be configured using per queue-message TTL or per message TTL to retain the messages.
Queuing messages you do to get a message from one point to the other reliable. So the sender can work independently from the receiver. What you propose is working with a temporary persistent store.
A sql database would fit perfectly, but also a mongodb would work nicely. You drop a document in mongo, give it a ttl and let the database handle the expiration.
http://docs.mongodb.org/master/tutorial/expire-data/