I have read that If a consumer takes a message and does not acknowledge that it has been successfully processed within a given time window, then another consumer is given the same message to guarantee "at-least-once" delivery and I have seen this happening in our production as well.
But I would like to know is there any flag through which we can configure the time interval and only when that time period crosses re-attempt the delivery to an another consumer.
I did a lot of search but couldn't find any such flag, so if anyone who has used this property earlier, please do let me know about it.
Thanks
then another consumer is given the same message to guarantee "at-least-once" delivery
That is not true; the message remains in an un-ack'd state until the consumer acks or nacks it, or the connection to that consumer is lost; at which time the message becomes availabe to be sent to another consumer.
Related
I need a solution where I can set a minimum delay between the messages that are polled out of the Queue. I do not want to delay every message by a fixed amount of miliseconds.
Lets say the Queue get 3 messages in the first second. But I want to pull every 5 seconds. So my Client does not get overloaded with to many request.
Is there a way to solve this with rabbitmq or do i have to change to some other framework?
Any time you ask for assistance about RabbitMQ (or any software), you must provide information about what versions of software you are using, and what client libraries. That way the people who are assisting you can do so effectively.
Your client should consume from the queue using the basic.consume method. Set the channel's "prefetch" value to the maximum number of unacknowledged messages you wish for that consumer to receive at once (you can set it to 1 if you only want one message at a time). Then, do your work and only acknowledge the messages after the desired amount of time has elapsed.
Be certain that this does not result in messages accumulating in queues. You will monitor your RabbitMQ installation, right?
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Shovels
consumes messages from the queue,
re-publishes each message to the destination broker (using, by default, the original exchange name and routing_key when
applicable).
I could not find any documentation what's the expected behavior for message TTLs when shovels are involved:
Does the time used for calculating TTL start when message is received
at the source broker or at the destination broker? Or is it just valid for the first publish, that is at the source broker?
What happens if the expiration time elapses before the message reaches the destination broker?
So, I think you answered the question in the documentation you pasted in. All shovel does is move messages from one queue to another, re-publishing them in the process. It's going to preserve all original message properties, which theoretically includes the TTL property.
That being said, I don't believe this is something you need to worry about.
Message TTL starts when the queue receives the message. When the message is re-published, the clock resets on the new queue.
Messages being transported by shovel will ideally spend no more than a few milliseconds in the initial queue, if they even end up there at all (a message queue with a consumer attached doesn't actually enqueue any messages under most conditions). So, the time spent in the first queue should be so small that it doesn't matter.
Message lifetime should have a fair amount of tolerance for network transport, etc., so the activities of shovel are on par with the normal noise.
If you find yourself in the situation where a large number of messages are accumulating in the queue before they can be shovel'd, then you might need to handle expiration in your application. There are other benefits and caveats to doing this, but you get a little finer-grained control overall.
I'm trying to use RabbitMQ in a more unconventional way (though at this point i can pick any other message queue implementation if needed)
I have one queue (I can have more if needed) that where customers are fetching N messages asynchronous. After they do their work I send the results from the client to the db.
I have two problems: first I don't want that they will work on the same message, second I want to grantee that I wont lose messages in case that my customer will close the browser or just stop working.
I looked at the documentation and saw the TTL which was perfect for me if I could alter that message that got timeout isn't going to be deleted but to move to another queue. can't find a way to alter this.
Moreover I looked at the confirmation option which in the first glance looked what I wanted,that mechanism is working like this: when the consumer gets a message he send confirmation to queue, I thought I can delay this confirm and send it when the work is done on the client side.
my problem was that I can't program the queue that if any message didn't get confirm then return it to the queue (or to another).
I also find how to do a scheduled message but it didn't help either because I don't want that the message will be inserted to the queue in five min,I want that when a customer will receive a message it will be locked in the queue for 5 min until confirm to delete is set otherwise return it to the queue.
Can I do temporary queue that enables my mechanism?
If someone can help with one of the problems or suggest another architecture or option to do it in another MQ it would be great.
Resources:
confirmation:
http://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/
post about locks but his problem was a batcher component:
Locks and batch fetch messages with RabbitMq
TTL:
https://www.rabbitmq.com/ttl.html
Schedule a message:
https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/
my problem was that I can't program the queue that if any message
didnt get confirm then return it to the queue (or to another).
RabbitMQ does this anyhow, so all you have to do is switch off the auto-ack flag, you figured this out
I thought I can delay this confirm and send it when the work is done
on the client side.
so just send the ACK once you've finished with processing the message.
All the unacknowledged messages remain in the queue and are re-delivered to next consumer (or the same one when it's up again, depending on your setup)
I'm positive I'm missing a nuance of MassTranist and/or RabbitMQ, but how long do durable (permanent?) messages stay on queues?
The situation I'm thinking of is one in which all consumers of a certain type of event are unavailable - obviously when they come back up, you want them to be able to take the appropriate actions based on the events they "missed" while they were offline.
However, what about the case when a new consumer starts reading off of the same queue after days/months/years? Is that consumer now going to be pulling in all events since the beginning of time? I'm almost certain that's not the case, but how is durability balanced with timeliness?
As I know MassTransit doesn't control message lifetime. RabbitMQ doing the same, thus message will stay in queue forever. The only exception from this is request/response model in which you can set up timeout period in which you want accept response.
In common way if you need to control lifetime you can store creating time in the message and check it in consumers.
In the console pane rabbitmq one day I had accumulated 8000 posts, but I am embarrassed that their status is idle at the counter ready and total equal to 1. What status should be completed at the job, idle? In what format is registered x-pires? It seems to me that I had something wrong =(
While it's difficult to fully understand what you are asking, it seems that you simply don't have anything pulling messages off of the queue in question.
In general, RabbitMQ will hold on to a message in a queue until a listener pulls it off and successfully ACKs, indicating that the message was successfully processed. You can configure queues to behave differently by setting a Time-To-Live (TTL) on messages or having different queue durabilities (eg. destroyed when there are no more listeners), but the default is to play it safe.