I'm running RabbitMQ in a private EC2 on Amazon and I'm encountering some issues.
I've many consumers which read from the very same queue, and I experienced an unusual behavior in which, apparently, there is an upper limit on message per seconds read by the consumer.
These are the value I got:
1 consumer, 60 m/s
2 consumers, 30 m/s each, hence, 60 m/s
and so on.. capping at 60 m/s max.
I don't get what could be the bottleneck?
Related
I have a rabbitmq queue that each data which publish to this queue is approximately 1 MB size. Every second 4 or 5 data publish to this queue.
Consumer consume each data one by one(Fetch=1). When i stop the consumer service 30000 queued message became ready to consume. When i start the consumer its consume rate 30/s+. It is just fine for now.
Hovewer in day light publisher never stops publishing and consumer can handle the queue. But at night publisher don't send data anymore(it is not error. It is how it suppose to be). At the first light of the next day publisher starts to publish 7 data per second. This time rabbit queue is started to go up continuesly.
First thought was that consumer can't handle the data. But it can consume 30/s+ data every sec.
I know that consume speed depends on consumer.
BUT.
I think that rabbit have some mechanism that after a time it decrease the consumer speed. Maybe it is lock mechanism maybe internal logs. I couldn't find any solution. Please help
This picture shows the limit of consumer speed.
Thanks to Lutz Horn
How is the consumer implemented? Does it use a push or a pull approach? It only consumes 3.8 messages per second. Edit: The screenshot shows a Consumer utilisation of 0%. This means that RabbitMQ always has to wait for the consumer to be able to handle the next message. RabbitMQ can never just push a published message to the consumer. See https://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/– Lutz Horn
When dequeuing from RabbitMQ it seems that the entire queue of messages is taken into memory and then processed, a minimal example can be the tutorial at https://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html.
Giving for example 1000 messages, they are taken all together in the event Received and the event handler manages 1000 messages: instead, I would to take 100 messages at time in order to divide the message consuming among 2 or more servers using a load balancer as HAProxy.
Is it possible or the only possibility is in the event handler make a manual round robin to other servers?
You have to use the prefecth to decide how many messages you want to download to the client.
instead, I would take 100 messages at a time in order to divide the message consuming among 2 or more servers.
it is what is explained here of "Fair dispatch" section, in your case you have to set the prefetch to 100 instead of 1 and attach more consumers to the same queue.
I was trying to perform LoadTest on the RabbitMQ messaging to see to what extent it can take messages into the queue and transfer it to the target machine over shovel.
Steps i followed:
producer has 20 threads. Each thread sends message to a dedicated queue(Say suppose ProducerQueue1 -- ProducerQueue20).The message is of size 51Mb each. The messages are sent in random interval using java.util.Random(50) seconds.
After each message sent at random seconds(A random second between 1- 50),
there is a sleep of 2 minutes.Therefore each of the producer threads sleep for
2 min after every send.
The messages are sent in a infinite while loop.
There are shovels from each dedicated queue to the consumer side dedicated queues(Say suppose ConsumerQueue1 --- ConsumerQueue20).
The link speed is 100mbps.
Issue observed:
Initially the messages are transferred with no issues, but after some time the NETWORK AT CONSUMER SIDE IS CHOKED.
The reason for choking is that after certain period of time, even if 4/5 out of 20 thread's random second coincides, then the consumer receives close to 250Mb message in one shot. Since the network speed is 100mbps as mentioned above, the network gets choked.
Due to this, the shovels will not be able to exchange heartbeats to stay in "running" state. The leads shovels to move from "running" to "terminated" state. The shovels try to establish a connection depending upon the "reconnect delay".
Due to break in shovels at producer side, the queues at the producer starts getting accumulated.
My Question:
The consumer's rabbitMq memory starts increasing as the queues start accumulating more messages. The memory is crossing the water mark. The purpose of water mark is not served. I have 16gb ram and i have set watermark to 40%(i.e 6.4gb ram). But still the memory shoots up to 10gb and doesnt recover and the producer system hangs.
Can any one please answer my question. and also tell me can there be any other reason for network choking which i mentioned above.
Thanks in advance.
We have a Java application that gets messages from rabbitmq using Spring AMQP.
For some of the queues, the number of consumers are not increasing resulting in slower messages delivery rate.
e.g. even though the max consumers is set to 50, number of consumers remained 6 for most of the time for the load of 9000 messages.
However, this is not the case with other queues. i..e consumers count reached till 35 for other queues.
We are using SimpleMessageListenerContainer's setMaxConcurrentConsumers API for setting max consumers.
Can someone please help me to understand this?
Configuration:
number of concurrent consumers: 4
number of max concurrent consumers: 50
When asking questions like this, you must always show configuration. Edit your question with complete details.
It depends on your configuration. By default, a new consumer is only added once every 10 seconds, and only if an existing consumer receives 10 messages without any gaps.
If that still doesn't answer your question, turn on DEBUG logging. If you can't figure it out from that, post the log (covering at least startConsumerMinInterval milliseconds) someplace like pastebin or dropbox.
All,
I have a problem with the performance with RabbitMQ when consuming messages when there is a large amount of messages to be consumed e.g. 280,000. It seems to go up and down from a performance perspective. The graph illustrated in the diagram taken from the management console demonstrates this where a consumer averages around 40 messages per second but then jumps up to around 120 messages per second:
The pattern will repeat itself again where it will go back to 40 and up to 120 again etc
Also, if I run the same test 1 hour later, the same up and down effect occurs but the range can vastly vary e.g. from 140 to 400 messages per second.
Note: The consumer does nothing with the messages
Note: Single consumer and ConsumerMessagePrefetchCount = 500
In relation to performance I have the following questions:
Is this up and down behaviour normal and expected or should the consumption speed of messages be steady?
Are the numbers that I am quoting expected or should they be much better/worse?
Any help appreciated
Billy
This behavior is quite normal, the queues are designed to be always close to zero messages. 280,000 is an high number, it means that the producer is faster than the consumer(s) so you have to increase the consumers number.
If you have a spike load, 280,000 could be not high number because you have a time to consume the messages.
There are lots techniques to increase the performances, for example:
Increase the consumer threads, (How many threads do you use to
consume the messages?)
Send messages with noAck
PrefetchCount is very important, an high value couldn’t be a right
solution.
The consumers should be steady, but also the producers should be steady, in load spike situation you need more time or more resources.
A few questions:
What rate do you have ?
Do you consume the messages from the same queue?
Do you need the ACK?
Why do you have 280.000 messages to the queue? Is it just a test or
a real situation ?
I hope it can be useful
As said Alexis Richardson (RabbitMQ) :
The easiest way to increase performance is to change what you are measuring ;-)