I have a service AAA that posts 10 to 50 thousand messages a minute to a RabbitMQ exchange. A .NET Core service BBB subscribes to a queue (to which all messages are routed) and for each message calls another HTTP service CCC over Internet. The problem is CCC is very unreliable, a few times a day it goes completely off for a minute or two and at least once a week it dies for an hour.
I don't have control over AAA or CCC. How can I use RabbitMQ routing features to reliably deliver all messages trought?
For an unreliable third-party service CCC which goes offline for minutes or hours, a circuit-breaker can be useful. Configure the circuit-breaker to break when it detects CCC is offline.
You can monitor the circuit-breaker state to detect when CCC is offline and/or log changes of circuit-state for later analysis.
Polly's circuit-breaker allows you to hook in any custom code on transitions of circuit state, so you could also:
when the circuit breaks, unsubscribe from the RabbitMQ queue.
when the circuit half-opens, resubscribe to the RabbitMQ queue at narrow parallelism (say, with a pre-fetch count of only 1 or 2 ... only enough messages for the circuit-breaker to retry the circuit).
when the circuit closes (healthy again), resubscribe to the RabbitMQ queue at full throughput.
This pattern would prevent you getting 100000s of messages flowing over to the RabbitMQ error/dead-letter/your custom retry queue, as soon as the circuit-breaker detects CCC is offline.
You would still need to consider what happens to the messages that do fail (before the circuit breaks or while retesting it), as described in another answer. Direct them to an error/retry queue. Or if the unsubscribe-when-CCC-is-down pattern works well enough with your real world parameters, you may be able to let messages which fail simply return to the original queue.
If CCC experiences also any transient faults (faulting for only a few seconds), consider introducing a WaitAndRetry policy.
With incoming message rates potentially at 1000s per second, you likely want to consider how you are limiting parallelism of message processing within BBB and/or the timeout you set on calls to CCC. Without this you may risk memory bulges in the consumer as more and more messages arrive while other requests are hanging on a response from CCC before it times out; a high timeout on CCC clearly exacerbates this. Consumer parallelism can be limited by using manual ack and applying a pre-fetch count.
Assuming that your subscription method in AAA is synchronous, throwing an exception will cause Rabbit to put the message into a dead letter queue. From there you could move them back or manually inspect them. Alternatively, you could re-publish it to the same exchange, but you might have an issue with messages that fail over and over filling up the queue. To avoid that, publish it with a header that would prevent it from entering the original queue again. Create another persistent queue that uses your new header as a filter. Presto, you have a re-try queue. You can subscribe to it for retries and/or alerts, use Shovel to move messages back to the original queue, whatever.
Related
I'm not sure how to resiliently handle RabbitMQ messages in the event of an intermittent outage.
I subscribe in a windows service, read the message, then store it my database. If I can't process the record because of the data I publish it to a dead letter queue for a human to address and reprocess.
I am not sure what to do if I have some intermittent technical issue that will fix itself (database reboot, network outage, drive space, etc). I don't want hundreds of messages showing up on dead letter that just needed to wait for a for a glitch but now would be waiting on a human.
Currently, I re-queue the event and retry it once, but it retries so fast the issue is not usually resolved. I thought of retrying forever but I don't want a real issue to get stuck in an infinite loop.
Is a broad topic but from the server side you could persist your messages and make your queues durable, this means that in the eventuality the server gets restarted they won't be lost, check more here How to persist messages during RabbitMQ broker restart?
For the consumer (client) it will depend on how you configure your client, from the docs:
In the event of network failure (or a node crashing), messages can be duplicated, and consumers must be prepared to handle them. If possible, the simplest way to handle this is to ensure that your consumers handle messages in an idempotent way rather than explicitly deal with deduplication.
If a message is delivered to a consumer and then requeued (because it was not acknowledged before the consumer connection dropped, for example) then RabbitMQ will set the redelivered flag on it when it is delivered again (whether to the same consumer or a different one). This is a hint that a consumer may have seen this message before (although that's not guaranteed, the message may have made it out of the broker but not into a consumer before the connection dropped). Conversely if the redelivered flag is not set then it is guaranteed that the message has not been seen before. Therefore if a consumer finds it more expensive to deduplicate messages or process them in an idempotent manner, it can do this only for messages with the redelivered flag set.
Check more here: https://www.rabbitmq.com/reliability.html#consumer
I have the following use case that I'm trying to setup in rabbit MQ:
Normally process A should handle all messages sent to queue A.
However if process A goes down (is no longer consuming from queue A) Then process B should handle the messages until process A comes back up.
At first it looks like consumer priorities might be the solution. https://www.rabbitmq.com/consumer-priority.html. However that will send messages to process B when process A is just blocked working on other messages. I only want them sent to process B when process A is down.
A 2nd option might be dead lettering. https://www.rabbitmq.com/dlx.html. If process A is not reading from queue A the messages will eventually time out and then move to an exchange that forwards them to a queue that process B reads. However that options requires waiting for the message to timeout which is not ideal. Also the message could timeout even while process A is still working which is not ideal.
Any ideas how rabbit MQ could be configured for the use case described above? Thanks
According to your answers to my questions, I would probably use a priority on consumer so that process A handles a maximum of messages, along with a high prefetch count (if possible, and you must ensure your process can handle such a high number).
Then, process B would handle the messages that process A cannot handle due to the high load, or all the messages when process A is not available. It is probably acceptable that in the case of high load some messages are handled with a higher delay. Do not forget to set a low prefetch count for process B.
Hope this helps.
I have a Java application which publishes events to RabbitMQ. It has one very important characteristic: message order must be preserved at all times. The consumer can handle duplicates, but it cannot handle when message 2 is enqueued before message 1, so to say.
I have been reading a lot about RabbitMQ lately, and I feel there is only solution to do this: set the channel in confirm mode (https://www.rabbitmq.com/confirms.html - basically, it forces the broker to acknowledge the publication) and publish one by one. With one by one I mean that the message 2 is only published after RabbitMQ confirmed (via an asynchronous ACK response) that message 1 is actually well received and persisted.
I tried this in a conceptual implementation, and while this works fine, it's uber slow, without exaggerating. Which makes sense: after all, we are now limiting our message rate to 1 message at a time.
So this leads me to my question: are there other, more performant, ways to ensure that message ordering is always preserved (either in RabbitMQ or via different approaches)?
Although my concern is RabbitMQ, I believe this question might be applied to any kind of asynchronous message queue service.
RabbitMQ's clients enqueue in the same order that you sent. It's when subscribers go down, you get network splits or the subscriber NACKs messages that they can get re-ordered; and even then RMQ tries to keep them in the same approximate order by re-queueing at the same position, or as close to the same position.
You can do it like you suggest; take one message at a time, because if you take a message, but crash before you've ACKed it from the broker, it will pop up when your service comes back up, at the same position.
This assumes you only have a single service instance at any given time, consuming from the queue. Which in turn is a distributed systems problem on its own, if you have a scheduler like Kubernetes or Mesos, spawning your service instances.
Another solution would be to ensure ordering of processing in the receiving service, by "resequencing" the messages based on their logical timestamps/sequence numbers.
I've written a much more thorough guide as annotated code here https://github.com/haf/rmq-publisher-confirms-hopac/blob/master/src/Server/Shared/RabbitMQ.fs — with batching you can resequence. Furthermore, if your idempotence builds the consecutive sequence numbers into its logic, you can start taking batches and each event will be idempotent, despite being re-consumed.
I am using MSMQ 4 with WCF. We have a Microsoft Dynamics plugin putting a message on an queue. A service picks up the message and makes an HTTP request to another web server. The web server responds by putting another message on a different queue. A second service picks up the messages and sends the response back to Dynamics...
We have our retry queue set up to retry 3 times and then wait for 5 minutes before retrying again. The Dynamics system some times takes so long (due to other plugins) that we can round-trip before the database transaction commits. The user's aren't seeing the update come through for another 5 minutes.
I am curious if there is a way to configure the retry mechanism to retry incrementally. So, the first time it fails, it only waits a few seconds. If it fails a second time, it waits twice that. And the time between retries just keeps growing.
The problem with just reducing the time between retries is that a bad message could easily fill up a log file.
It turns out there is no built-in way of doing this. One slightly involved option is to create multiple queues, each with its own retry/poison sub-queues, each with a growing retry delay. You can reuse the same handler for each queue - the only thing that changes is the configuration. You also need a handler that can read the poison sub-queues (service) and move the message to the next queue in the chain (client).
So, you set receiveErrorHandling to Move. The maxRetryCycles and receiveRetryCount are just 1. Each queue will use a growing retryCycleDelay. Each queue you create will have a poison sub-queue created for it automatically. You simply read from each poison sub-queue and use a client to move it to the next queue.
I am sure someone could write some code that would automatically create N queues with a growing retryCycleDelay and hook it up all programmatically. Since it is the same handler/client for every queue, it wouldn't be a big deal.
Lets say I have a queue with a bunch of messages in it. I have 2 consumers connected to that queue, both set with a prefetch = 1. The work that these consumers do takes some time, and I don't want to acknowledge the message until the work is done (in case the consumer crashes or something - I want the message to automatically reenter the queue in exceptional cases).
But I also want these consumers to work in parallel, and that doesn't appear to be happening. In other words, as long as there are 2+ messages in the queue, I'd expect both consumers to be busy.
What appears to be happening instead is that consumer 1 receives a message, but consumer 2 will wait until consumer 1 has acknowledged the message. Then consumer 2 receives a message and consumer 1 waits, etc.
Is there an option I'm missing? Or should this be working, I just have a bug in my code somewhere? Or is this not possible?
You should be able to pull messages off the queue while previous messages are still being processed by other consumers. The RabbitMQ tutorial specifically points to parallelism as a strength of round-robin dispatching (http://www.rabbitmq.com/tutorials/tutorial-two-python.html). Are your two consumers running as threads in the same process? I wonder if you've just made a mistake in the implementation.