I'm not sure how to resiliently handle RabbitMQ messages in the event of an intermittent outage.
I subscribe in a windows service, read the message, then store it my database. If I can't process the record because of the data I publish it to a dead letter queue for a human to address and reprocess.
I am not sure what to do if I have some intermittent technical issue that will fix itself (database reboot, network outage, drive space, etc). I don't want hundreds of messages showing up on dead letter that just needed to wait for a for a glitch but now would be waiting on a human.
Currently, I re-queue the event and retry it once, but it retries so fast the issue is not usually resolved. I thought of retrying forever but I don't want a real issue to get stuck in an infinite loop.
Is a broad topic but from the server side you could persist your messages and make your queues durable, this means that in the eventuality the server gets restarted they won't be lost, check more here How to persist messages during RabbitMQ broker restart?
For the consumer (client) it will depend on how you configure your client, from the docs:
In the event of network failure (or a node crashing), messages can be duplicated, and consumers must be prepared to handle them. If possible, the simplest way to handle this is to ensure that your consumers handle messages in an idempotent way rather than explicitly deal with deduplication.
If a message is delivered to a consumer and then requeued (because it was not acknowledged before the consumer connection dropped, for example) then RabbitMQ will set the redelivered flag on it when it is delivered again (whether to the same consumer or a different one). This is a hint that a consumer may have seen this message before (although that's not guaranteed, the message may have made it out of the broker but not into a consumer before the connection dropped). Conversely if the redelivered flag is not set then it is guaranteed that the message has not been seen before. Therefore if a consumer finds it more expensive to deduplicate messages or process them in an idempotent manner, it can do this only for messages with the redelivered flag set.
Check more here: https://www.rabbitmq.com/reliability.html#consumer
Related
If I use publisher confirms, I can be (reasonably) sure that a message sent to an exchange on the RabbitMQ server, and which received ACK from the RabbitMQ server is not lost even if the RabbitMQ server crashes (power outage for example).
However what happens when a message arrives at a dead letter exchange after a manual rejection in the consumer? (channel.basicReject, I use Spring AMQP.)
Can I still be sure that in the case in which the original message is dequeued from the queue to which the consumer is listening, and the RabbitMQ server subsequently crashes, I will eventually find the message, after the RabbitMQ server is restarted, in the queues which are bound to the dead letter exchange (if normally the message would have arrived there)?
If the answer is negative, is there a way to ensure that this is the case?
As #GaryRussell suggested, I posted a similar question on rabbitmq-users Google group.
Here is the answer I got from Daniil Fedotov
"Hi,
There is no delivery guarantees in place. Dead lettering does not check if the message was enqueued or saved to disk.
Dead-lettering does not use publisher confirms or any other confirm mechanisms.
It's not that easy to implement reliable dead-lettering from one queue to another and there are plans to address this issue eventually, but it may take a while.
If you want to safely reject messages from the consumer without a risk of losing them - you can publish them from the consumer application manually to the dead-letter queue, wait for the confirmation and then reject."
From my understanding RabbitMQ producers require acknowledgment when sending messages to the broker which provides a delivery-guarantee. Kafka producers does not require acknowledgement from the broker. Does that mean there’s no delivery-guarantee with Kafka? If not, how does Kafka provide delivery guarantee without acknowledgement?
Is my understanding correct? Please correct any misunderstandings that I have as I’m still learning about these systems.
Kafka is actually flexible about this.
The number of acknowledgements for producers is configurable. The configuration is called RequiredAcks. In fact, required acks is set on ProduceRequest level, but I've never seen implementations where a single producer instance allows producing messages with different required acks settings.
RequiredAcks is an integer value which means "how many acknowledgements the broker should wait before responding to a produce request".
Having RequiredAcks set to 0 (VERY not recommended for production) means "fire and forget", e.g. broker will respond immediately without waiting until data is written to log. This is the case where you could lose messages without even knowing about that.
Having RequiredAcks set to 1 means "wait until data is written to local log", where local log is log of the broker that received the request. Once your data is written to local log, broker responds.
Having RequiredAcks set to -1 means "wait until the data is written to local log AND replicated by all ISRs".
Each ProduceRequest also has a Timeout field, which means "maximum time to wait for necessary amount of acknowledgements".
So Kafka supports acknowledging requests but allows turning acknowledgements off.
In 0.9.0.0 and above, the producer#send has a return value Future you can get the offset of the message in the broker's partition. Meantime, you can implement Callback, if there is no exception, it's mean that the message has been sent to the correct broker.
I have done RabbitMQ queue ccing to find out whether there are messages published to queues. How do I find if all of them are delivered to consumers.
While this question is not altogether clear, let me address the issue of how (or whether) to know if a particular message (let's call it message x) was delivered to a consumer.
First, some theory.
Message queuing is commonly used across networks - and networks can be unreliable. Further, the machines operating the message system may be unreliable.
Message queues are usually designated for the processing of a particular type of message. The processing of the message itself may be unreliable.
As a result of the foregoing, messages have the possibility to be processed/consumed zero or more times (i.e. a message can be dropped, processed once, or processed more than once).
Now, RabbitMQ contains some features that attempt to mitigate the possible failure modes (primary using acknowledgments), but no mitigation technique can be 100% reliable. Therefore, while the reliability is higher, it cannot be guaranteed - and your application needs to be able to cope with the occasional possibility of failure.
There is an inherent assumption in the question that the original publisher of message x cares about the consumer of message x. This indicates that a two-way exchange (e.g. RPC) is needed - one from publisher to consumer for message x, then from consumer back to original publisher (message y). The original publisher maintains state while the consumer processes message x, and the receipt of the message y response closes out the state machine.
If the intent is to simply publish a stream of messages, the publisher should neither be aware of the consumers nor care whether or not the messages are consumed. However, from an application monitoring standpoint, you presumably would care. You (as the systems administrator) could do a few things to see if messages are being consumed:
Monitor the RabbitMQ management console to see publish/consume rates, as well as queue length
Set up logging and tracing in your application (perhaps dumping logs off to elasticsearch) - then set up a log analyzer to detect abnormal log conditions
Set up performance monitoring on the consuming computers - if there is a problem, you will likely see abnormal statistics on variables like processor time and memory use
Send an occasional test message, which can be specially configured to put a marker in the logs, and look for that marker.
I am using RabbitMQ server.
For publishing messages, I set the immediate field to true and tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that the number of messages in the queue was zero.
Then, I changed the immediate flag to false and again tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that a total of 100,000 messages were in queues (till now, no consumer was present).
After that, I started a consumer and it consumed all the 100,000 messages.
Can anybody please help me in understanding about the immediate bit field and this behavior too? Also, I could not understand the concept of the mandatory bit field.
The immediate and mandatory fields are part of the AMQP specification, and are also covered in the RabbitMQ FAQ to clarify how its implementers interpreted their meaning:
Mandatory
This flag tells the server how to
react if a message cannot be routed to
a queue. Specifically, if mandatory is
set and after running the bindings the
message was placed on zero queues then
the message is returned to the sender
(with a basic.return). If mandatory
had not been set under the same
circumstances the server would
silently drop the message.
Or in my words, "Put this message on at least one queue. If you can't, send it back to me."
Immediate
For a message published with immediate
set, if a matching queue has ready
consumers then one of them will have
the message routed to it. If the lucky
consumer crashes before ack'ing
receipt the message will be requeued
and/or delivered to other consumers on
that queue (if there's no crash the
messaged is ack'ed and it's all done
as per normal). If, however, a
matching queue has zero ready
consumers the message will not be
enqueued for subsequent redelivery on
from that queue. Only if all of the
matching queues have no ready
consumers that the message is returned
to the sender (via basic.return).
Or in my words, "If there is at least one consumer connected to my queue that can take delivery of a message right this moment, deliver this message to them immediately. If there are no consumers connected then there's no point in having my message consumed later and they'll never see it. They snooze, they lose."
http://www.rabbitmq.com/blog/2012/11/19/breaking-things-with-rabbitmq-3-0/
Removal of "immediate" flag
What changed? We removed support for the
rarely-used "immediate" flag on AMQP's basic.publish.
Why on earth did you do that? Support for "immediate" made many parts
of the codebase more complex, particularly around mirrored queues. It
also stood in the way of our being able to deliver substantial
performance improvements in mirrored queues.
What do I need to do? If you just want to be able to publish messages
that will be dropped if they are not consumed immediately, you can
publish to a queue with a TTL of 0.
If you also need your publisher to be able to determine that this has
happened, you can also use the DLX feature to route such messages to
another queue, from which the publisher can consume them.
Just copied the announcement here for a quick reference.
Does anyone know if the pop operation on a RabbitMQ queue is atomic?
I have several processes reading from the same queue (the queue is marked as durable, running on version 2.0.0) and I am seeing some quite odd behaviour.
If your multiple processes are consuming messages from the same queue then they should never consume the same message.
Here are the caveats, though:
If a message has been delivered by the broker to one of your consumers and it rejects the message (or terminates before getting a chance to acknowledge it) then the broker will put it back on the same queue and it would be delivered to one of your remaining active consumers.
If your consumers are pulling from distinct queues -- each with a matching binding -- then the broker will put copies of the message on each queue and each consumer will get a copy of the same message.