Is it better regarding performance to create multiple exchanges? For instance, if I have multiple file types like zip, json, xml,…
Is it better to create one file exchange (Topic exchange), which accepts all files and map them to the right queues. Here, my concern is that if I get many zip files and the messages accumulate at the file exchange less frequently messages are processed late.
Or to create one exchange per file type (Zip exchange, Json exchange, Xml exchange).
How, does an exchange processes messages? Is there only one process, which handles the mapping between the exchange and the queues or are the multiple processes, which do the mapping in parallel?
RabbitMQ stores the messages to the queues, the exchanges don't store the messages.
Multiple queues can work in parallel.
To have a good performance you have to use different queues and avoid too many bindings between exchanges and queues.
Related
When I look into the design in my new company's push applications backed by RabbitMQ, I find that we have some queues which have millions or even hundred million messages to send to for one push task. Say the queue name is named PUSH_QUEUE
I wonder if I can benefit from the design that I split the queue into several pieces and why:
PUSH_QUEUE_1
PUSH_QUEUE_2
PUSH_QUEUE_3
PUSH_QUEUE_4
PUSH_QUEUE_5
and producer will send to this sharding queue by robin round, consumer subscribe all the queues.
We don't specify any exchange but the default one.
This might be helpful for you: RabbitMQ-Sharding.
I'm trying to scale-out a RabbitMQ messaging system. The current system is very simple - the producer sends a message to a fanout exchange and the message is handle by multiple consumers - classic fanout routing .
I have multiple consumers from different types (e.g: one that print to screen, one that logs to file, one that saves to DB,...).
My challenge - i'm not sure what's the best way to scale-out the consumers. If i add other consumers from the same type - i'll get double logs or double entries in the DB. ... (think about two DB consumers consuming from the same fanout exchange) .
I guess I can create a consumer that publish to a work-queue but I wonder if there's a better "builtin" solution in rabbitmq.
thanks in advance,
zf
If you need to scale consumers in order to consume faster all the messages coming from the fanout exchange you need competive consuming; so you need more consumers attached to the same queue bound to the fanout exchange.
In this way every consumer will consume a batch of messages indipendently from the others. The number of messages inside the batch is defined with the prefetch count property ( http://www.rabbitmq.com/consumer-prefetch.html ).
In this way, in your case, you should be able to scale consumers avoiding double logs and double entries in the DB.
Why do we need routing key to route messages from exchange to queue? Can't we simply use the queue name to route the message? Also, in case of publishing to multiple queues, we can use multiple queue names. Can anyone point out the scenario where we actually need routing key and queue name won't be suffice?
There are several types of exchanges. The fanout exchange ignores the routing key and sends messages to all queues. But pretty much all other exchange types use the routing key to determine which queue, if any, will receive a message.
The tutorials on the RabbitMQ website describes several usecases where different exchange types are useful and where the routing key is relevant.
For instance, tutorial 5 demonstrates how to use a topic exchange to route log messages to different queues depending on the log level of each message.
If you want to target multiple queues, you need to bind them to a fanout exchange and use that exchange in your publisher.
You can't specify multiple queue names in your publisher. In AMQP, you do not publish a message to queues, you publish a message to an exchange. It's the exchange responsability to determine the relevant queues. It's possible that a message is routed to no queue at all and just dropped.
Decoupling queue names from applications is useful for flexibility.
You could establish multiple queues to consume the same message, but queues can't have the same name.
In some cases, message's originator doesn't know the names of queues. (like when you have randomly generated queue names when horizontally scaling a server)
An exchange may be routing messages for more than just one type of consumer. Then you would need some wildcards in your routing keys to route messages to concerned consumers.
all,
Background: now client produces many messages of different labels.Consumers may subscribe some labels of them(different consumer need different labels's msgs).
Now method: I used direct exchange in rabbitmq, consumer declared queue for itself, exchange distribute msgs to these queues.
Problem: there exsit many redundant msgs in queues which effects the performance.
A topic type in activemq supports user a method to distribute a msg to many subscribers, which need not creates large number queues for subscribers.
Does exsit a method in rabbitmq, or some suggestions to solve the problem?
I have a Topic exchange from which I'd like to distribute messages to two queues on two servers part of a cluster, in order to reduce memory pressure on any particular server. My consumers are periodically slow, and I sometimes run into the high memory watermark.
The way I tried to resolve this is by routing messages using an intermediate direct exchange, with two queues bound to the exchange:
a (topic) -> a1 (direct) -> q1/q2 (bound to routing key "a")
But the messages were routed to both queues, as AMQP intends. Anyone has ideas? What I need is an exchange that routes to one and only one queue, even if the routing key matches many queues. I'd prefer not to change my routing keys, but that could be arranged.
I found Selective routing with RabbitMQ, which may mean I'll need to implement my own routing logic. Hopefully, this already exists somewhere else.
You could perhaps use the Shovel plugin - http://www.rabbitmq.com/shovel.html - to move messages from your intermediate exchange to the two queues.
If you set up two shovels, both consuming from a single queue on the direct intermediate exchange, they should be able to fight over the messages coming in (I'm assuming that you don't care too much if the two recipient queues don't get the incoming messages in a strict round robin fashion). The shovels then each publish to one of the two end queues, and can send through the ACKs from the end consumer.