In one of our applications the back pressure did not work and there was a huge pileup in a queue on RabbitMQ. This caused the RMQ node to choke.
Is there a way to apply flow control (manually) on that queue in such cases? That would have slowed down the producer and given us headroom.
In your case the consumers are not fast enough to handle the messages.
Basically you had a load-spike.
So, it does not mean that you need to stop the publishers.
You could:
Increase the number of the consumers
Use the Lazy queues
you didn't see the flow control because RabbitMQ could handle the messages.
Related
So I'm testing RabbitMQ in one node. Plain and simple,
One producer sends messages to the queue,
Multiple consumers take tasks from that queue.
Currently consumers execute thousands of messages per second, they are too fast so I need them to slow down. Managing consumer-side throttling is not possible due to network unreliable nature.
Collectively consumers must not take more than 10 messages per second altogether from that queue.
Is there a way to configure RabbitMQ so as the queue dispatches a maximum of 10 messages per second?
If I remember correctly, once Rabbit MQ has delivered a message to the queue, it's up to consumers to consume a message. There are various consumers in different languages, you haven't mentioned anything specific, so I'm giving a generic answer.
In my understanding, you shouldn't try to impose any restrictions on Rabbit MQ itself, instead, consider implementing connection pool of message consumers that will be able to handle not more than X messages simultaneously on the client side. Alternatively, you can provide some kind of semaphore at the handler itself, but not on the Rabbit MQ server itself.
If I have two queues from which I want to consume messages, and I use a single SimpleMessageQueueListenerContainer for it, in which order would the listeners be invoked/messages consumed when both queues have messages?
I will try to be more specific of the problem I am working on:
I have a consumer application which needs to consume messages from 2 queues – say regular-jobs-queue and infrequent-jobs-queue. If there are any messages in ‘infrequent-jobs-queue’ I want to consume those before consuming messages from ‘regular-jobs-queue’. I might not be able to combine these and put all messages into a single rabbitmq level priority queue and assign higher priority to infrequent-job message because of some upcoming use-cases like purging regular-jobs without affecting infrequent-jobs and others.
I am aware that RabbitMQ has support for consumer priority but I am not very sure if it will be applicable here. I want all instances of my consumer application to first consume messages of infrequent-jobs-queue if any and not prioritize amongst these consumers.
Or should I like have 2 containers, with dedicated consumer thread(s) per queue and have an internal priority-queue data structure into which I can put messages as and when consumed from rabbitmq queue.
Any help would be really appreciated. Thanks.
~Rashida
You can't do what you want; messages will be delivered with equal priority.
Moving them to an internal in-memory queue will risk message loss.
You might want to consider using one of the RabbitTemplate.receive() or receiveAndConvert() methods instead of a message-driven container.
That way you have complete control.
I am trying to understand how rabbitmq per-connection flow-control works with multiple consumers. In particular what would happen if one consumer were to hang? Would flow control be invoked and how would it affect the rest of the consumers? Would the behaviour depend upon whether the queues were durable or autodeleting?
Thanks.
Rabbit MQ uses "Credit Flow Control".
Essentially, whenever a message is received on a channel a credit is deducted. Credit starts at a default level, e.g. 200, and when it dips below 0, connections are blocked. After a certain number of messages are consumed and ACKed, the credit is bumped up a certain amount.
You can read more about it here:
http://videlalvaro.github.io/2013/09/rabbitmq-internals-credit-flow-for-erlang-processes.html
Per-connection flow control describes what happens when a publisher (or group of publishers) is sending messages to queues faster than the queues are being processed. This is a safety feature as RabbitMQ becomes unstable at some point when the queue fills without bound. From the documentation, this is automatic:
RabbitMQ will block connections which are publishing too quickly for queues to keep up. No configuration is required.
Unfortunately, the documentation is not terribly specific on when/how this flow control is implemented, other than "several times per second." So, if one consumer gets stuck, as long as the other consumer(s) can keep up, flow control should not be triggered.
I am starting with ActiveMQ and I have a usecase. i have n producers sending messages into a Queue Q1. I want to stop the delivery of messages (i.e. i do not want consumers to consume those messages). I want to store the messages for sometime without those being consumed.
I was looking at ways this can be achieved. These two things came into my mind based on what i browsed through.
Using Mirrored queues, so that I can wiretap the messages and save into a virtual queue.
Possibly stop consumers from doing a PULL on the queue.
Another dirty way of doing this is by making consumers not send ack messages once its consumed a message from the queue.
We are currently not happy with either of these.
Any other way you can suggest.
Thanks in advance.
If you always want message delivery to be delayed you can use the scheduler feature of ActiveMQ to delay delivery until a set time or a fixed delay etc.
Other strategies might also work but it really up to you to design something that fits your use case. You can try to use Apache Camel to define a route that implements the logic of your use case to either dispatch a message to a Queue or send it to the scheduler for delayed processing. It all really depends on your use case and requirements.
Pretty new to RabbitMQ and we're still in the investigation stage to see if it's a good fit for our use cases--
We've readily come to the conclusion that our desired topology would have us deploying a few topic based exchanges, and then filtering from there to specific queues. For example, let's say we have a user and an upload exchange, where the user queue might receive messages where the topic is "new-registration" or "friend-request" and the upload exchange might receive messages like "video-upload" or "picture-upload".
Creating the queues, getting them routed to the appropriate queue, and then building listeners to handle the messages for the various queues has been quite straight forward.
What's unclear to me however is if it's possible to do a fanout on a topic exchange?
I.e. I have named queues that are bound to my topic exchange, but I'd like to be able to just throw tons of instances of my listeners at those queues to prevent single points of failure. But to the best of my knowledge, RabbitMQ treats these listeners in a straight forward round robin fashion--e.g. every Nth message always go to the same Nth listener rather than dispatching messages to the first available consumer. This is generally acceptable to us but given the load we anticipate, we'd like to avoid the possibility of hot spots developing amongst our consumer farm.
So, is there some way, either in the queue or exchange configuration or in the consumer code, where we can point our listeners to a topic queue but have the listeners treated in a fanout fashion?
Yes, by having the listeners bind using different queue names, they will be treated in a fanout fashion.
Fanout is 1:N though, i.e. each task can be delivered to multiple listeners like pub-sub. Note that this isn't restricted to a fanout exchange, but also applies if you bind multiple queues to a direct or topic exchange with the same binding key. (Installing the management plugin and looking at the exchanges there may be useful to visualize the bindings in effect.)
Your current setup is a task queue. Each task/message is delivered to exactly one worker/listener. Throw more listeners at the same queue name, and they will process the tasks round-robin as you say. With "fanout" (separate queues for a topic) you will process a task multiple times.
Depending on your platform there may be existing work queue solutions that meet your requirements, such as Resque or DelayedJob for Ruby, Celery for Python or perhaps Octobot or Akka for the JVM.
I don't know for a fact, but I strongly suspect that RabbitMQ will skip consumers with unacknowledged messages, so it should never bottleneck on a single stuck consumer. The comments on their FAQ seem to suggest that RabbitMQ will make an effort to keep things chugging along even in the presence of troublesome consumers.
This is a late answer, but in case others come across this question...
It sounds like what you want is fair dispatch rather than a fan out model (which would publish a given message to every queue).
Fair dispatch will give a message to the next available worker rather than using a simple round-robin approach. This should avoid the "hotspots" you are concerned about, without delivering the same message to multiple consumers.
If this is what you are looking for, then see the "Fair Dispatch" section on this page in the Rabbit docs. A prefetch count of 1 is the key here.