We have a Java application that gets messages from rabbitmq using Spring AMQP.
For some of the queues, the number of consumers are not increasing resulting in slower messages delivery rate.
e.g. even though the max consumers is set to 50, number of consumers remained 6 for most of the time for the load of 9000 messages.
However, this is not the case with other queues. i..e consumers count reached till 35 for other queues.
We are using SimpleMessageListenerContainer's setMaxConcurrentConsumers API for setting max consumers.
Can someone please help me to understand this?
Configuration:
number of concurrent consumers: 4
number of max concurrent consumers: 50
When asking questions like this, you must always show configuration. Edit your question with complete details.
It depends on your configuration. By default, a new consumer is only added once every 10 seconds, and only if an existing consumer receives 10 messages without any gaps.
If that still doesn't answer your question, turn on DEBUG logging. If you can't figure it out from that, post the log (covering at least startConsumerMinInterval milliseconds) someplace like pastebin or dropbox.
Related
I have around 300 different consumers / 300 message types / 300 queues with the most wildest functionality behind it.
From the extreme side:
Is the best choice to make 1 windows service (easier to deploy) with 300 consumers listening.
Or 300 windows services (easier to split between devs) each independent 1 consumer but impossible to maintain by support
?
update: from 1 to 300 queues
RabbitMQ can support hundreds of queues simultaneously, and each queue should be responsible for one specific type of message e.g. a response status or an online order information or a stack trace information for further processing by some other unit of work, these three are not same and if you are keeping them all in one then please segregate them into different queues.
If you will keep all the data in one queue it will also effect your application performance as each queue works in a sequential order and since you have 300 consumers that wait for 300 types of messages, almost all of them could be in waiting state and it is also a reason for complex decision making algorithm, if you are using one to figure out the correct consumer.
What could also go wrong with a single queue is that it is now a bottleneck which could obstruct the functioning of the whole application, if that queue fails, because every consumer listens to it. By having different queues the rest of the system can still process if one particular queue faces an issue.
Instead of going for 1 consumer per service you can check if there's anything common and if the services can take up more consumers than one after increasing the number of queues from 1 to many.
We have a setup where we have many consumers to a Queue.
The problem it seems like only a subset of those consumers are actually doing work.
Example
One Queue has 120 Consumers and has about 1000 messages?
It seems to only process 20 messages at a time though.
Any ideas?
It sounds like you're running into the prefetch count limit. I believe the default is 20.
From https://rabbitmq.docs.pivotal.io/36/rabbit-web-docs/consumer-prefetch.html
channel.basicQos(10, false); // Per consumer limit
channel.basicQos(15, true); // Per channel limit
Just remember that there are design complexities to working with large numbers of concurrent operations. (It can be done, but be careful that you're maintaining data integrity.)
I need a solution where I can set a minimum delay between the messages that are polled out of the Queue. I do not want to delay every message by a fixed amount of miliseconds.
Lets say the Queue get 3 messages in the first second. But I want to pull every 5 seconds. So my Client does not get overloaded with to many request.
Is there a way to solve this with rabbitmq or do i have to change to some other framework?
Any time you ask for assistance about RabbitMQ (or any software), you must provide information about what versions of software you are using, and what client libraries. That way the people who are assisting you can do so effectively.
Your client should consume from the queue using the basic.consume method. Set the channel's "prefetch" value to the maximum number of unacknowledged messages you wish for that consumer to receive at once (you can set it to 1 if you only want one message at a time). Then, do your work and only acknowledge the messages after the desired amount of time has elapsed.
Be certain that this does not result in messages accumulating in queues. You will monitor your RabbitMQ installation, right?
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
When dequeuing from RabbitMQ it seems that the entire queue of messages is taken into memory and then processed, a minimal example can be the tutorial at https://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html.
Giving for example 1000 messages, they are taken all together in the event Received and the event handler manages 1000 messages: instead, I would to take 100 messages at time in order to divide the message consuming among 2 or more servers using a load balancer as HAProxy.
Is it possible or the only possibility is in the event handler make a manual round robin to other servers?
You have to use the prefecth to decide how many messages you want to download to the client.
instead, I would take 100 messages at a time in order to divide the message consuming among 2 or more servers.
it is what is explained here of "Fair dispatch" section, in your case you have to set the prefetch to 100 instead of 1 and attach more consumers to the same queue.
I need to know what is the difference between prefetch count vs no ack in rabbitmq ?
Also
What is the difference between following statements :-
if i set prefetch count say 10 does 10 consumer threads are created ?
Or --
if i register 10 cosumers will it create 10 threads ?
Which of the above is more efficient
To answer this specifically for spring-amqp.
prefetchCount=10 means the broker allows up to 10 unacked message for each consumer; it does not affect the number of threads.
Use concurrentConsumers to create multiple consumers - which will have one thread each.
auto ack means the broker doesn't require acks (so you can lose messages). Spring AMQP also blocks deliveries (to prefetch count) if the listener can't keep up.
Pre-fetch count: How many messages the consumer should read from queue and kept internally rather than picking one message at a time.
No-Ack: Do not acknowledge back that the consumer is done consuming the message.
Those both are used to fine tune your set-up
To address your second part of the question:
If you set prefetch count to 10, 10 consumers won't be created, but your single consumer will fetch 10 messages at a time.
And if you create 10 consumers it will most likely create 10 threads (or processes). It all depends on how you configure it. Most likely you will be wanting to use a thread pool though
I know this question is old, but part of it was never specifically answered, so for anyone who comes here later looking for answers:
If you don't want new messages sent to you as soon as you acknowledge the previous ones, but instead want a message to be sent to you only when you explicitly request one, then you don't want to set up a "consumer" (in RabbitMQ terminology) at all; specifically, you'll want to use AMQP's basic.get operation (which just fetches a single message without creating a consumer) rather the more common basic.consume operation (which registers a consumer that will be sent messages as they become available).
Different libraries and frameworks will have different ways of accomplishing this; for example, in Ruby, using the Bunny client, you can call message = queue.get instead of queue.subscribe do .... In Spring, you'd do something like GetResponse response = channel.basicGet("some.queue", false);.