I have exchange and queue. Producer doesn't need consumption confirmation, but messages in some cases can be un-processable by consumer in current moment, because of lack of other data. Because of this, I want to return those messages to the end of queue. How to do this? Or is it done automatically when I reject message?
Flow:
Message1 gets consumed and creates some record in database.
Message2 gets consumed and checks if there is record in database, and if yes, then it updates the record. If there is no record in database, message should be returned to the end of queue.
So there is message ordering problem and in general situation I get messages in order, because most of components deliver their messages correctly. I want to solve potential situation, when Producer of Message1 wasn't able to put message to exchange immediately because of heavy load or other reason. In this situation, Message2 will be consumed first but there will be no sufficient informations in database to process it. I want this message to be returned back to queue, but be sure that this Message2 will go to the tail of queue. If it will go to head, I will get infinite loop if I use only one queue.
Side question is, if it's possible to track how many times consumers tried to process message but returned it. If there is possibility to put message to the tail of queue like I described before, but for some reason Producer of Message1 died, and there will be no Message1, I want to make Message2 dead after some number of retries or some time.
RabbitMQ always puts rejected messages at the head of the queue. To put them at the tail, you will have to publish them yourself (e.g. using RabbitTemplate). You can add a header with a count of retries.
Related
Consider the following scenario:
A message is put on a queue. The queue is declared to be durable.
A single consumer gets the message from the queue but do not acknowledge it. A basic consume was performed on the message by the consumer. At this moment, I can see in rabbitmq management website that the ready_message count goes down by 1 and messages_unacknowledged count goes up by 1.
After that, a basic reject with requeue was performed on the message's delivery tag/info. message_ready and message_unacknowledged count still same here on the queue.
Now, I performed a basic recover on the same consumer on the same channel. However, even after executing the basic recover and adding some sleep did not make the the message_unacknowledged count go down by 1 and message_ready count go up by 1.
Finally, when I close the current channel, I see the message_unacknowledged count go down by 1 and message_ready count go up by 1.
My question is, Is it not possible to do a recover the unacknowledged messages on the same consumer unless you close your current channel in RabbitMQ?
I'm trying to use RabbitMQ in a more unconventional way (though at this point i can pick any other message queue implementation if needed)
I have one queue (I can have more if needed) that where customers are fetching N messages asynchronous. After they do their work I send the results from the client to the db.
I have two problems: first I don't want that they will work on the same message, second I want to grantee that I wont lose messages in case that my customer will close the browser or just stop working.
I looked at the documentation and saw the TTL which was perfect for me if I could alter that message that got timeout isn't going to be deleted but to move to another queue. can't find a way to alter this.
Moreover I looked at the confirmation option which in the first glance looked what I wanted,that mechanism is working like this: when the consumer gets a message he send confirmation to queue, I thought I can delay this confirm and send it when the work is done on the client side.
my problem was that I can't program the queue that if any message didn't get confirm then return it to the queue (or to another).
I also find how to do a scheduled message but it didn't help either because I don't want that the message will be inserted to the queue in five min,I want that when a customer will receive a message it will be locked in the queue for 5 min until confirm to delete is set otherwise return it to the queue.
Can I do temporary queue that enables my mechanism?
If someone can help with one of the problems or suggest another architecture or option to do it in another MQ it would be great.
Resources:
confirmation:
http://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/
post about locks but his problem was a batcher component:
Locks and batch fetch messages with RabbitMq
TTL:
https://www.rabbitmq.com/ttl.html
Schedule a message:
https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/
my problem was that I can't program the queue that if any message
didnt get confirm then return it to the queue (or to another).
RabbitMQ does this anyhow, so all you have to do is switch off the auto-ack flag, you figured this out
I thought I can delay this confirm and send it when the work is done
on the client side.
so just send the ACK once you've finished with processing the message.
All the unacknowledged messages remain in the queue and are re-delivered to next consumer (or the same one when it's up again, depending on your setup)
I want to read the payload, or messageId of unacknowledged messages in a RabbitMQ queue. Is this possible?
The reason I want to do so is I trying to use RabbitMQ dead letter feature to build a cycle to for auto-generating message periodically.
Briefly, create two queues - work queue and delay queue.
Set TTL of the message in delay queue as the time frequency of need to periodically. Can have different messages with different TTL for different job purpose;
put a message into the delay queue. When the message expires, it gets republished into the work queue. The message can sit in the work queue as long as needed until a consumer is up to consume it.
One consumer picks up the message, and process it. If processing succeeds, the consumer needs acknowledge the work queue, and then write the message back to the delay queue; If processing fails (e.g., the thread crashes), no acknowledgement. Then the message would re-appear in the worker queue automatically. Then another consumer can take up the job. When the message sent back to the delay queue gets expired again, it gets republished, then re-consumed by a consumer ...... A cycle constructed, workload distributed.
I want to make sure there is no missing or duplicate messages in the cycle since I do not want missing job or double doing the job at the same time. However, there is tiny tiny chance duplicate messages can happen. Below show the consumer first write back the message to delay queue, and acknowledge the work queue. If the thread crashes right between below two lines, the message would be in the delay queue, and Rabbit republish the message again into work queue. The end up with duplicate messages in the cycle.
channel.basicPublish(DELAY_EXCHANGE, "", null, message.getBytes());
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
To prevent above, I want to add a dog watch logic after above two line:
Check the total number of messages in the cycle (total messages in both queues) to see whether it is equal my expected number (I expected the number less 10);
If the number does not matches, I want to figure out which one is missing or which one is duplicate, then deal with it. I do not care about the sequence of those messages, or the frequency has been disturbed since this is a really really edge case to consider. I can easily retrieve those messages which are ready and requeue them. But the problem is how to deal with those unacknowledged messages?
Thank you very much in advance!
Roy
It's not possible to read unacknowledged messages from other context the original messages was consumed and held as un-aked.
I am using RabbitMQ server.
For publishing messages, I set the immediate field to true and tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that the number of messages in the queue was zero.
Then, I changed the immediate flag to false and again tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that a total of 100,000 messages were in queues (till now, no consumer was present).
After that, I started a consumer and it consumed all the 100,000 messages.
Can anybody please help me in understanding about the immediate bit field and this behavior too? Also, I could not understand the concept of the mandatory bit field.
The immediate and mandatory fields are part of the AMQP specification, and are also covered in the RabbitMQ FAQ to clarify how its implementers interpreted their meaning:
Mandatory
This flag tells the server how to
react if a message cannot be routed to
a queue. Specifically, if mandatory is
set and after running the bindings the
message was placed on zero queues then
the message is returned to the sender
(with a basic.return). If mandatory
had not been set under the same
circumstances the server would
silently drop the message.
Or in my words, "Put this message on at least one queue. If you can't, send it back to me."
Immediate
For a message published with immediate
set, if a matching queue has ready
consumers then one of them will have
the message routed to it. If the lucky
consumer crashes before ack'ing
receipt the message will be requeued
and/or delivered to other consumers on
that queue (if there's no crash the
messaged is ack'ed and it's all done
as per normal). If, however, a
matching queue has zero ready
consumers the message will not be
enqueued for subsequent redelivery on
from that queue. Only if all of the
matching queues have no ready
consumers that the message is returned
to the sender (via basic.return).
Or in my words, "If there is at least one consumer connected to my queue that can take delivery of a message right this moment, deliver this message to them immediately. If there are no consumers connected then there's no point in having my message consumed later and they'll never see it. They snooze, they lose."
http://www.rabbitmq.com/blog/2012/11/19/breaking-things-with-rabbitmq-3-0/
Removal of "immediate" flag
What changed? We removed support for the
rarely-used "immediate" flag on AMQP's basic.publish.
Why on earth did you do that? Support for "immediate" made many parts
of the codebase more complex, particularly around mirrored queues. It
also stood in the way of our being able to deliver substantial
performance improvements in mirrored queues.
What do I need to do? If you just want to be able to publish messages
that will be dropped if they are not consumed immediately, you can
publish to a queue with a TTL of 0.
If you also need your publisher to be able to determine that this has
happened, you can also use the DLX feature to route such messages to
another queue, from which the publisher can consume them.
Just copied the announcement here for a quick reference.
We are having MVC application which reads data from MSMQ.
We are trying to find out a way to read message from queue and remove it from queue only if user has done a successful operation on the queue.
The message should remain in the queue until user completes the processing, the message should not be available to anyone else until the user who is processing the message object has finished the operation.
Is there a property for a Message object to be set as Peeked which will not allow reading of this message again until ether it is put back into the queue or removed from the queue?
We are not sure if using MSMQ is a good idea in this case?
It sounds like you need to use your queue(s) in transactional mode. Then, your client can receive a message, process it, and then commit the transaction, at which point the message will be finally dequeued. While the transaction is active, however, other clients will not see the message -- it will be held in reserve until the transaction completes or is aborted.
This MSDN article has a decent overview of usage patterns for reliable messaging with MSMQ:
http://msdn.microsoft.com/en-us/library/ms978430.aspx
The Queue is the right idea. Your approach of "leave it in the queue, locked, but still kind-of-available" is wrong.
You may need multiple queues.
Process A enqueues something in Queue 1
Process B dequeues from Queue 1 and starts work.
If B is successful, that's it.
Otherwise, it gets queued somewhere else (perhaps the same queue, or perhaps Queue 2) for follow-up work.
If it went back into Queue 1, B will find it again, eventually. If it went to another Queue, then another process does cleanup, logging, error fixup or whatever, possibly putting something back in Queue 1.
A Queue isn't a database -- there's nothing stateful (no "don't look at me, I'm being processed").
A Queue is transient storage. Someone writes, someone else reads, and that's it.
If you want reliability, read this: http://msdn.microsoft.com/en-us/library/ms978430.aspx
And this: http://blogs.msdn.com/shycohen/archive/2006/02/20/535717.aspx
And this: http://www.request-response.com/blog/PermaLink,guid,03fb0e40-b446-42b5-ad90-3be9b0260cb5.aspx
Reliability is a feature of the queue, not your application. You can do a "recoverable read". It's a transaction that's part of the queue API.