We have a setup where we have many consumers to a Queue.
The problem it seems like only a subset of those consumers are actually doing work.
Example
One Queue has 120 Consumers and has about 1000 messages?
It seems to only process 20 messages at a time though.
Any ideas?
It sounds like you're running into the prefetch count limit. I believe the default is 20.
From https://rabbitmq.docs.pivotal.io/36/rabbit-web-docs/consumer-prefetch.html
channel.basicQos(10, false); // Per consumer limit
channel.basicQos(15, true); // Per channel limit
Just remember that there are design complexities to working with large numbers of concurrent operations. (It can be done, but be careful that you're maintaining data integrity.)
Related
I need to know what is the difference between prefetch count vs no ack in rabbitmq ?
Also
What is the difference between following statements :-
if i set prefetch count say 10 does 10 consumer threads are created ?
Or --
if i register 10 cosumers will it create 10 threads ?
Which of the above is more efficient
To answer this specifically for spring-amqp.
prefetchCount=10 means the broker allows up to 10 unacked message for each consumer; it does not affect the number of threads.
Use concurrentConsumers to create multiple consumers - which will have one thread each.
auto ack means the broker doesn't require acks (so you can lose messages). Spring AMQP also blocks deliveries (to prefetch count) if the listener can't keep up.
Pre-fetch count: How many messages the consumer should read from queue and kept internally rather than picking one message at a time.
No-Ack: Do not acknowledge back that the consumer is done consuming the message.
Those both are used to fine tune your set-up
To address your second part of the question:
If you set prefetch count to 10, 10 consumers won't be created, but your single consumer will fetch 10 messages at a time.
And if you create 10 consumers it will most likely create 10 threads (or processes). It all depends on how you configure it. Most likely you will be wanting to use a thread pool though
I know this question is old, but part of it was never specifically answered, so for anyone who comes here later looking for answers:
If you don't want new messages sent to you as soon as you acknowledge the previous ones, but instead want a message to be sent to you only when you explicitly request one, then you don't want to set up a "consumer" (in RabbitMQ terminology) at all; specifically, you'll want to use AMQP's basic.get operation (which just fetches a single message without creating a consumer) rather the more common basic.consume operation (which registers a consumer that will be sent messages as they become available).
Different libraries and frameworks will have different ways of accomplishing this; for example, in Ruby, using the Bunny client, you can call message = queue.get instead of queue.subscribe do .... In Spring, you'd do something like GetResponse response = channel.basicGet("some.queue", false);.
We have a Java application that gets messages from rabbitmq using Spring AMQP.
For some of the queues, the number of consumers are not increasing resulting in slower messages delivery rate.
e.g. even though the max consumers is set to 50, number of consumers remained 6 for most of the time for the load of 9000 messages.
However, this is not the case with other queues. i..e consumers count reached till 35 for other queues.
We are using SimpleMessageListenerContainer's setMaxConcurrentConsumers API for setting max consumers.
Can someone please help me to understand this?
Configuration:
number of concurrent consumers: 4
number of max concurrent consumers: 50
When asking questions like this, you must always show configuration. Edit your question with complete details.
It depends on your configuration. By default, a new consumer is only added once every 10 seconds, and only if an existing consumer receives 10 messages without any gaps.
If that still doesn't answer your question, turn on DEBUG logging. If you can't figure it out from that, post the log (covering at least startConsumerMinInterval milliseconds) someplace like pastebin or dropbox.
I need to implement sequential message processing on multiple consumers, but only one message per time on the queue. I have a lot of queues, but all of them are sequential and I need multiple consumers support for load balancing and redundancy. Anybody can tell whether it is real or not to limit number of unacknowledged message to 1 per queue?
Anybody can tell whether it is real or not to limit number of unacknowledged message to 1 per queue?
this isn't possible with multiple consumers. you can limit the number of unacknowledged messages using prefetch limit for a single channel, but not across multiple channels / consumers. it is tied to the channel of the consumer, not the queue.
the only way you can achieve this is with a single consumer and a single queue, using prefetch.
even then, you have no guarantee that the messages will arrive in the queue in the correct order.
(this is a fundamental difficulty with distributed systems of any kind, not a rabbitmq limitation)
look at the Message Sequencer and Resequencer patterns to try and put the messages back in order.
but even then, you're going to run into difficulty.
you'll also want to read up on idempotency so you don't re-process a message that has already been processed.
You should be able to configure your consumer to consume only X message(s) at time and same for your channel. Take a look at QOS or Consumer Prefetch
https://www.rabbitmq.com/consumer-prefetch.html
Here is an example, where multi-consumers will acknoledge only one message and channel allow only one message to be acknoledged (whatever how much consumers a plugged on it)
Channel channel = ...;
Consumer consumer1 = ...;
Consumer consumer2 = ...;
channel.basicQos(1, false); // Per consumer limit
channel.basicQos(1, true); // Per channel limit
channel.basicConsume("my-queue1", false, consumer1);
channel.basicConsume("my-queue2", false, consumer2);
Here, a consumer can acknoledge only one message each time, and the channel can only have one unacknoledged message. You didn't mention which language you use so you'll problably have to adapt this example.
Where did the question:
We are using RabbitMQ as task queue. One of specific tasks - sending notices to Vkontakte social network. They api has limit to request per seconds and this limit based on your application size. Just 3 calls for app with less then 100k people and so on. So we need to artificially limit out request to they service. Now this logic is application based. It simple while you can use just one worker per such queue, just set something like sleep(300ms) and be calm. But when your should use N workers this synchronization becomes not trivial.
How to limit throughput with RabbitMQ?
Based on story above. If it were possible set prefetch size not only message based but time based to this logic can be much simple. For example, "qos to 1 message per fetch not faster then 1 time in seconds" or so on.
Is there something like this?
May be other strategy about this?
This is not possible out of the box with RabbitMQ.
You're right, with distributed consumers this throttling becomes a difficult exercise. I would suggest having a look at ZooKeeper which would allow you to synchronize all consumers and throttle processing of messages by leveraging it's Znodes / Watches
for throttled yet scalable solution.
All,
I have a problem with the performance with RabbitMQ when consuming messages when there is a large amount of messages to be consumed e.g. 280,000. It seems to go up and down from a performance perspective. The graph illustrated in the diagram taken from the management console demonstrates this where a consumer averages around 40 messages per second but then jumps up to around 120 messages per second:
The pattern will repeat itself again where it will go back to 40 and up to 120 again etc
Also, if I run the same test 1 hour later, the same up and down effect occurs but the range can vastly vary e.g. from 140 to 400 messages per second.
Note: The consumer does nothing with the messages
Note: Single consumer and ConsumerMessagePrefetchCount = 500
In relation to performance I have the following questions:
Is this up and down behaviour normal and expected or should the consumption speed of messages be steady?
Are the numbers that I am quoting expected or should they be much better/worse?
Any help appreciated
Billy
This behavior is quite normal, the queues are designed to be always close to zero messages. 280,000 is an high number, it means that the producer is faster than the consumer(s) so you have to increase the consumers number.
If you have a spike load, 280,000 could be not high number because you have a time to consume the messages.
There are lots techniques to increase the performances, for example:
Increase the consumer threads, (How many threads do you use to
consume the messages?)
Send messages with noAck
PrefetchCount is very important, an high value couldn’t be a right
solution.
The consumers should be steady, but also the producers should be steady, in load spike situation you need more time or more resources.
A few questions:
What rate do you have ?
Do you consume the messages from the same queue?
Do you need the ACK?
Why do you have 280.000 messages to the queue? Is it just a test or
a real situation ?
I hope it can be useful
As said Alexis Richardson (RabbitMQ) :
The easiest way to increase performance is to change what you are measuring ;-)