I'm using RabbitMQ and Celery for a project and I've reached a bottleneck.
My architecture is as follows:
1 RabbitMQ node
between 1 and 7 nodes who will read from RabbitMQ via Celery
I started some performance measurements and by pre-populating Rabbit with 200k messages, the node performs about 600msg/sec.
Starting two nodes with the same pre-populated queue, I get a little under 600msg/sec for both nodes.
Adding more node with the same scenario leads to a drastic loss in throughput, reaching under 400msg/sec with 7 nodes.
I've started to add some settings (some even from the RabbitMQ site) but it lead to no improvement.
My current settings configuration is
[ {kernel, [
{inet_default_connect_options, [{nodelay, true}]},
{inet_default_listen_options, [{nodelay, true}]} ]}, {rabbit, [
{tcp_listeners,[{"0.0.0.0", 5672}]},
{vm_memory_high_watermark, 0.6},
{tcp_listen_options, [binary,
{packet, raw},
{reuseaddr, true},
{backlog, 128},
{nodelay, true},
{exit_on_close, false},
{keepalive, true},
{sndbuf, 32768},
{recbuf,32768}
]} ]} ].
I've been reading blogs and posts by users and they mention a lot bigger throughput than what I am achieving. There are mentions of 100k/sec while I'm barely getting 2.8k/sec.
Any thoughts on what I can do to improve my throughput?
Try using transient queues by setting the following in celeryconfig.py:
CELERY_DEFAULT_DELIVERY_MODE = 'transient'
This prevents RabbitMQ from saving to disk which can improve throughput
Also if you're using RabbitMQ as your result backend you'll probably get better performance by switching to Redis.
My solution was to have a separate queue for each consumer.
Related
I use Symfony Messenger with the RabbitMQ adapter in my application. The logic of the application is that messages from the same queue must be executed sequentially, one after the other (the order is not important), even if the queue is listened to by multiple consumers. In other words, the queue should not give out other messages if at least one is unacked.
To solve the problem, I found in the documentation an option such as x-single-active-consumer (https://www.rabbitmq.com/consumers.html#single-active-consumer).
The configuration is like this:
my-queue:
dsn: amqp://guest:guest#localhost:5672/%2f/my-queue
options:
queues:
my-queue:
arguments:
x-single-active-consumer: true
The UI shows that the queue was created with the SAC (single-active-consumer) flag, which confirms that the configuration is correct. However, when several consumers are started, they receive tasks in parallel, without waiting for the previous task to finish, in fact, as if the single-active-consumer flag wasn't set at all.
Can you tell me what I'm doing wrong?
Preconditions:
symfony/messenger 4.4.*
RabbitMQ 3.8 +
Only what I needed to do was to remove queue in Rabbit Admin and let consumer to register new one.
It is not possible to enable single active consumer with a policy. Here is the reason why. Policies in RabbitMQ are dynamic by nature, they can come and go, enabling and disabling the features they declare. Imagine suddenly disabling single active consumer on a queue: the broker would start sending messages to inactive consumers and messages would be processed in parallel, exactly the opposite of what single active consumer is trying to achieve. As the semantics of single active consumer do not play well with the dynamic nature of policies, this feature can be enabled only when declaring a queue, with queue arguments
https://www.rabbitmq.com/consumers.html#single-active-consumer
You can verify setup from rabbitmq container (or to use rabbitmqadmin whenever you have your rabbit)
rabbitmqadmin export rabbit.definitions.json
...
"queues": [
{
"name": "some-queue",
"vhost": "/",
"durable": true,
"auto_delete": false,
"arguments": {
"x-single-active-consumer": true <--------------
}
}
],
"exchanges": [
{
"name": "my-exchange",
"vhost": "/",
"type": "direct", <---------------------------------
"durable": true,
"auto_delete": false,
"internal": false,
"arguments": {}
}
}
...
edit #1: You many also want to check that exchange is type of direct. Default by symfony/messenger is fanout, which distributes messages to all available consumers.
The fanout exchange is very simple. As you can probably guess from the name, it just broadcasts all the messages it receives to all the queues it knows.
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
config/messenger.yml
...
my-binding:
dsn: 'some-dsn'
options:
exchange:
name: my-echange
type: direct <-----------------------------
Background
I have a distributed system with many machines. I have two types of applications - Producer and Consumer. To be more specific - a single producer and multiple consumers. Each "consumer machine" has multiple consumers.
All the messages in the system are going to same queue. Message looks like this:
{
"Id": "Thisismyid",
"CacheId": "CacheID"
...
}
My consumers are applying a cache strategy in order to process queue messages faster. Once the message was downloaded by the consumer, it being checked if the CacheId is already cached previously. If yes - continue. If no - cache it and continue.
All the consumers on same machine are sharing the same cache repository.
The problem
This structure is "optimal" when I have 1 consumer. Since, same machine cache the items and use it multiple times.
As the number of consumers is going up - the efficiency of the cache is going down. Because its more likely
that an item will be downloaded by node that wasn't has a ready to use cache.
The Question
How to use RabbitMQ to "route" messages with same same CacheId to be processes by same consumer\machine to increase efficiency? What is the "cost" in terms of RabbitMQ resources?
You might be able to do this with a topic exchange: https://www.rabbitmq.com/tutorials/tutorial-five-dotnet.html
But this would quickly get complicated if you have many CacheId.
But I would use a centralized cache instead. Could be Redis: https://redis.io/
I did a base install of RabbitMQ, the latest, without any configuration changes. It is running, and I have 2 million messages sitting on a queue, but I'm only able to pull them off 20 at a time. I was expecting in the thousands. I haven't been able to find anything obvious in the configuration documentation that says explicitly configure this for a high-throughput production environment. Could someone push me in the right direction to figure out why I'm not getting any throughput? I have 1 consumer with a prefetch count of 24. The consumer does nothing with the message, just pulling it off for testing purposes.
I have a base rabbitmq.config file, which only contains this:
[
{rabbit, [{vm_memory_high_watermark, 0.7}]}
].
I have looked through the example config at https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/rabbitmq.config.example but nothing stands out as something that needs to be configured in order to speed up the consumers.
We have been using rabbit for a while, and after a recent deploy to create a new fanout exchange, we got quite spooked by something very curious: in the real time graphs in the management console, and in the REST API, we have queues consistently publishing fewer posts than they are delivering/acking, even though the queue size is hovering around 0!
Sample stats from REST API:
"message_stats": {
"ack": 149063323,
"ack_details": {
"rate": 305.0
},
"deliver": 149089898,
"deliver_details": {
"rate": 318.8
},
"deliver_get": 149089898,
"deliver_get_details": {
"rate": 318.8
},
"disk_reads": 4058297,
"disk_reads_details": {
"rate": 0.0
},
"disk_writes": 149084451,
"disk_writes_details": {
"rate": 227.6
},
"publish": 142374350,
"publish_details": {
"rate": 129.6
},
"redeliver": 5379,
"redeliver_details": {
"rate": 0.0
}
},
The queue has been flatlined at 0 for the time periods we're worrying about.
Here's what the web management console shows:
From what we can tell, we are not getting messages multiple times, so we are currently holding the opinion that this is a bug in the statistics collection or reporting. We have not yet had a window to reboot this node as it is a live system. We have discovered no ill-effects yet.
Gory details: Our queue has 240 consumers over 12 connections, and its being published to via a fanout exchange bound to two queues. Both of those queues are exhibiting this behavior. The publish rates of the queues are matching the publish rates of the exchange. The two queues both have a dead letter exchange/queue combo, and require ack. The fanout exchange, and one of the queues are new, and there have been some changes to how these messages are published (used to be published to the default exchange routed to the old queue, now we publish to the new fanout exchange to hit both the old and new queues). The consumers' code has not changed much at all for the old queue (one fewer insert into a postgres table), and the new queue runs code that is extremely similar to the old queue (except only inserting into that one postgres table).
Are these stats possible to encounter in the wild outside of buggy conditions? Could this be caused by a bad topography? What would such a state entail, what side effects would it have, and what types of setups would create this?
I find that in Kafka 0.72, we can specify either zk.connect or broker.list. But in Kafka 0.8, we can only specify broker.list ,and we can’t specify zk.connect without specifying broker.list. I think, in this case, we can’t balance producer through zookeeper. If anyone use Kafka 0.8, or have some understanding with that? Many thanks!
You can still use a ZooKeeper client to retrieve the broker list:
ZkClient zkClient = new ZkClient("localhost:2108", 4000, 6000, new BytesPushThroughSerializer());
List<String> brokerList = zkClient.getChildren("/brokers/ips");
According to that, you do not have to "hardcode" the broker list on client side and you are flexible as far as the system architecture is concerned. But anyway, this would add the ZooKeeper dependency again which is in fact an disadvantage for producer in several environments.
If you want to get a detailed view to the so called "cluster metadata API" solution, check out this link: https://issues.apache.org/jira/browse/KAFKA-369
Best
pre
In Kafka 0.8, the producer is load balancing through the new cluster metadata API and the use of Zookeeper for that purpose has been removed.