We are working in a microservice architecture and we are using RabbitMQ as a message broker. We want to avoid the scenarios where the following happens:
An entity begins its creation but it takes a while for it to finish.
The system decides that the creation time has taken too long and that the entity should be deleted due to a timeout, so it sends out a message to delete the entity which is currently still being created
Delete message gets consumed and the system checks whether the entity exists and does not find it due to the entity still being in the process of being created.
Delete entity message consumer returns an error due to not finding the entity.
How can we ensure that the delete message is consumed after the create message is finished in such a way that we do not block the consumption of other messages?
How can we ensure that the delete message is consumed after the create message is finished in such a way that we do not block the consumption of other messages?
Let's say your entity creation timeout is N. The worker(s) responsible for creating entities should know about this timeout, and should be able to cancel entity creation should N be reached. This isn't strictly necessary but it sounds like your entity creation may be resource intensive so cancellation should be a feature you have.
If your workers know to cancel entity creation when timeout N is reached, then perhaps you don't even need the deletion message?
If you keep the delete message, the workers processing that could do the following:
First, ensure your queue has a Dead Letter Exchange configured
Consume the message, and try to delete the entity
If deletion succeeds, great, ack the message with RabbitMQ and you're done
If deletion fails, nack (reject) the message with RabbitMQ and do set requeue to be false. This will cause the message to be routed to the dead-letter exchange
A worker should consume from a queue bound to this dead-letter exchange. You could have a queue dedicated to re-trying entity deletions. When a worker consumes a message from this queue, it can re-try the deletion. If it fails, you can reject it again (after a delay, of course) and, if this queue has the same dead-letter settings, the same process will happen
Finally, ensure that your deletion workers respect the count property and only try a certain number of times to delete an entity. If a limit is exceeded, this should create an exception in your system
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
I'm not sure how to resiliently handle RabbitMQ messages in the event of an intermittent outage.
I subscribe in a windows service, read the message, then store it my database. If I can't process the record because of the data I publish it to a dead letter queue for a human to address and reprocess.
I am not sure what to do if I have some intermittent technical issue that will fix itself (database reboot, network outage, drive space, etc). I don't want hundreds of messages showing up on dead letter that just needed to wait for a for a glitch but now would be waiting on a human.
Currently, I re-queue the event and retry it once, but it retries so fast the issue is not usually resolved. I thought of retrying forever but I don't want a real issue to get stuck in an infinite loop.
Is a broad topic but from the server side you could persist your messages and make your queues durable, this means that in the eventuality the server gets restarted they won't be lost, check more here How to persist messages during RabbitMQ broker restart?
For the consumer (client) it will depend on how you configure your client, from the docs:
In the event of network failure (or a node crashing), messages can be duplicated, and consumers must be prepared to handle them. If possible, the simplest way to handle this is to ensure that your consumers handle messages in an idempotent way rather than explicitly deal with deduplication.
If a message is delivered to a consumer and then requeued (because it was not acknowledged before the consumer connection dropped, for example) then RabbitMQ will set the redelivered flag on it when it is delivered again (whether to the same consumer or a different one). This is a hint that a consumer may have seen this message before (although that's not guaranteed, the message may have made it out of the broker but not into a consumer before the connection dropped). Conversely if the redelivered flag is not set then it is guaranteed that the message has not been seen before. Therefore if a consumer finds it more expensive to deduplicate messages or process them in an idempotent manner, it can do this only for messages with the redelivered flag set.
Check more here: https://www.rabbitmq.com/reliability.html#consumer
I am using java client of
https://www.rabbitmq.com/tutorials/tutorial-six-java.html
. My setup is RPC. My server is creating queue and client is also creating same queue and sending the message. After receiving message server is performing some operation and sending result back to client.
Now if server created the queue and connect with it while queue get's deleted for some reason. The server is not throwing any exception and when the client is creating the same queue and putting messages server is not getting those messages either as it's not connected.
How do server knows that the queue get deleted?
Thanks so much
It sounds like the following situation is happening:
Queue A is created.
Consumer 1 subscribes to Queue A
Queue A is deleted while Consumer 1 is still active
Queue A is re-created (call it A')
Now, you're wondering why Consumer 1 is not getting any messages? You would have to re-subscribe your consumer. I don't usually delete queues, because there is no need to do so under any reasonable scenario (instead, use the queue.expires property to handle auto-deletion of queues).
According to the AMQP 0-9-1 Specification,
When a queue is deleted any pending messages are sent to a dead-letter
queue if this is defined in the server configuration, and all
consumers on the queue are cancelled.
So, based on the description of the behavior, this is a bug with the consumer. It should throw an exception or otherwise exit the consuming loop in this case. In any case, you'll have to re-subscribe to A' before you'll get any more messages.
I have one direct exchange. There is also one queue, bound to this exchange.
I have two consumers for that queue. The consumers are manually ack'ing the messages once they've done the corresponding processing.
The messages are logically ordered/sorted, and should be processed in that order. Is it possible to enforce that all messages are received and processed sequentially accross consumer A and consumer B? In other words, prevent A and B from processing messages at the same time.
Note: the consumers are not sharing the same connection and/or channel. This means I cannot use <channel>.basicQoS(1);.
Rationale of this question: both consumers are identicall. If one goes down, the other queue starts processing messages and everything keeps working without any required intervention.
One approach to handling failover in a case where you want redundant consumers but need to process messages in a specific order is to use the exclusive consumer option when setting up the bind to the queue, and to have two consumers who keep trying to bind even when they can't get the exclusive lock.
The process is something like this:
Consumer A starts first and binds to the queue as an exclusive consumer. Consumer A begins processing messages from the queue.
Consumer B starts next and attempts to bind to the queue as an exclusive consumer, but is rejected because the queue already has an exclusive consumer.
On a recurring basis, consumer B attempts to get an exclusive bind on the queue but is rejected.
Process hosting consumer A crashes.
Consumer B attempts to bind to the queue as an exclusive consumer, and succeeds this time. Consumer B starts processing messages from the queue.
Consumer A is brought back online, and attempts an exclusive bind, but is rejected now.
Consumer B continues to process messages in FIFO order.
While this approach doesn't provide load sharing, it does provide redundancy.
Even though this is already answered. May be this can help others.
RabbitMQ has a feature known as Single Active Consumer, which matches your case.
We can have N consumers attached to a Queue but only 1 (one) of them will be actively consuming messages from the Queue. Fail-over happens only when active consumer fails.
Kindly take a look at the link https://www.rabbitmq.com/consumers.html#single-active-consumer
Thank you
Usually the point of a MQ system is to distribute workload. Of course, there are some situations where processing of message N depends on result of processing the message N-1, or even the N-1 message itself.
If A and B can't process messages at the same time, then why not just have A or just B? As I see it, you are not saving anything with having 2 consumers in a way that one can work only when the other one is not...
In your case, it would be best to have one consumer but to actually do the parallelisation (not a word really) on the processing part.
Just to add that RMQ is distributing messages evenly to all consumers (in round-robin fashion) regardless on any criteria. Of course this is when prefetch is set to 1, which by default it is. More info on that here, look for "fair dispatch".
I am using RabbitMQ server.
For publishing messages, I set the immediate field to true and tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that the number of messages in the queue was zero.
Then, I changed the immediate flag to false and again tried sending 50,000 messages. Using rabbitmqctl list_queues, I saw that a total of 100,000 messages were in queues (till now, no consumer was present).
After that, I started a consumer and it consumed all the 100,000 messages.
Can anybody please help me in understanding about the immediate bit field and this behavior too? Also, I could not understand the concept of the mandatory bit field.
The immediate and mandatory fields are part of the AMQP specification, and are also covered in the RabbitMQ FAQ to clarify how its implementers interpreted their meaning:
Mandatory
This flag tells the server how to
react if a message cannot be routed to
a queue. Specifically, if mandatory is
set and after running the bindings the
message was placed on zero queues then
the message is returned to the sender
(with a basic.return). If mandatory
had not been set under the same
circumstances the server would
silently drop the message.
Or in my words, "Put this message on at least one queue. If you can't, send it back to me."
Immediate
For a message published with immediate
set, if a matching queue has ready
consumers then one of them will have
the message routed to it. If the lucky
consumer crashes before ack'ing
receipt the message will be requeued
and/or delivered to other consumers on
that queue (if there's no crash the
messaged is ack'ed and it's all done
as per normal). If, however, a
matching queue has zero ready
consumers the message will not be
enqueued for subsequent redelivery on
from that queue. Only if all of the
matching queues have no ready
consumers that the message is returned
to the sender (via basic.return).
Or in my words, "If there is at least one consumer connected to my queue that can take delivery of a message right this moment, deliver this message to them immediately. If there are no consumers connected then there's no point in having my message consumed later and they'll never see it. They snooze, they lose."
http://www.rabbitmq.com/blog/2012/11/19/breaking-things-with-rabbitmq-3-0/
Removal of "immediate" flag
What changed? We removed support for the
rarely-used "immediate" flag on AMQP's basic.publish.
Why on earth did you do that? Support for "immediate" made many parts
of the codebase more complex, particularly around mirrored queues. It
also stood in the way of our being able to deliver substantial
performance improvements in mirrored queues.
What do I need to do? If you just want to be able to publish messages
that will be dropped if they are not consumed immediately, you can
publish to a queue with a TTL of 0.
If you also need your publisher to be able to determine that this has
happened, you can also use the DLX feature to route such messages to
another queue, from which the publisher can consume them.
Just copied the announcement here for a quick reference.
We are having MVC application which reads data from MSMQ.
We are trying to find out a way to read message from queue and remove it from queue only if user has done a successful operation on the queue.
The message should remain in the queue until user completes the processing, the message should not be available to anyone else until the user who is processing the message object has finished the operation.
Is there a property for a Message object to be set as Peeked which will not allow reading of this message again until ether it is put back into the queue or removed from the queue?
We are not sure if using MSMQ is a good idea in this case?
It sounds like you need to use your queue(s) in transactional mode. Then, your client can receive a message, process it, and then commit the transaction, at which point the message will be finally dequeued. While the transaction is active, however, other clients will not see the message -- it will be held in reserve until the transaction completes or is aborted.
This MSDN article has a decent overview of usage patterns for reliable messaging with MSMQ:
http://msdn.microsoft.com/en-us/library/ms978430.aspx
The Queue is the right idea. Your approach of "leave it in the queue, locked, but still kind-of-available" is wrong.
You may need multiple queues.
Process A enqueues something in Queue 1
Process B dequeues from Queue 1 and starts work.
If B is successful, that's it.
Otherwise, it gets queued somewhere else (perhaps the same queue, or perhaps Queue 2) for follow-up work.
If it went back into Queue 1, B will find it again, eventually. If it went to another Queue, then another process does cleanup, logging, error fixup or whatever, possibly putting something back in Queue 1.
A Queue isn't a database -- there's nothing stateful (no "don't look at me, I'm being processed").
A Queue is transient storage. Someone writes, someone else reads, and that's it.
If you want reliability, read this: http://msdn.microsoft.com/en-us/library/ms978430.aspx
And this: http://blogs.msdn.com/shycohen/archive/2006/02/20/535717.aspx
And this: http://www.request-response.com/blog/PermaLink,guid,03fb0e40-b446-42b5-ad90-3be9b0260cb5.aspx
Reliability is a feature of the queue, not your application. You can do a "recoverable read". It's a transaction that's part of the queue API.