all,
Background: now client produces many messages of different labels.Consumers may subscribe some labels of them(different consumer need different labels's msgs).
Now method: I used direct exchange in rabbitmq, consumer declared queue for itself, exchange distribute msgs to these queues.
Problem: there exsit many redundant msgs in queues which effects the performance.
A topic type in activemq supports user a method to distribute a msg to many subscribers, which need not creates large number queues for subscribers.
Does exsit a method in rabbitmq, or some suggestions to solve the problem?
Related
If I have and exchange with a single queue bound to it. I have two processes each with their own connection. Each connection has its own consumer which is consuming messages from that single queue.
Is it possible that when a message is placed onto the queue, both consumers could be delivered the same message?
Pre-fetch seems to apply across consumers on a single channel, or connection if the global flag is set. What happens with consumers across multiple connections
No. Messages are always round-robin delivered to multiple consumers on the same queue.
One solution is to have two queues bound to the same exchange using the same routing key if you need messages to go to both consumers.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I'm doing some research if rabbitMQ is the way to go for me.
Can I mix publish/subscribe and worker queues?
The goal is to have have a few queues which holds items of for specific process.
Each process can have a lot of workers.
The setup shown in this diagram is possible. If the exchange X is a direct or topic exchange, the 'specific process' would be represented by one of the two queues and routing of messages would be based on the routing key or topic of a message.
Consumers ('workers') would consume messages from the queues using round-robin.
I'm trying to scale-out a RabbitMQ messaging system. The current system is very simple - the producer sends a message to a fanout exchange and the message is handle by multiple consumers - classic fanout routing .
I have multiple consumers from different types (e.g: one that print to screen, one that logs to file, one that saves to DB,...).
My challenge - i'm not sure what's the best way to scale-out the consumers. If i add other consumers from the same type - i'll get double logs or double entries in the DB. ... (think about two DB consumers consuming from the same fanout exchange) .
I guess I can create a consumer that publish to a work-queue but I wonder if there's a better "builtin" solution in rabbitmq.
thanks in advance,
zf
If you need to scale consumers in order to consume faster all the messages coming from the fanout exchange you need competive consuming; so you need more consumers attached to the same queue bound to the fanout exchange.
In this way every consumer will consume a batch of messages indipendently from the others. The number of messages inside the batch is defined with the prefetch count property ( http://www.rabbitmq.com/consumer-prefetch.html ).
In this way, in your case, you should be able to scale consumers avoiding double logs and double entries in the DB.
I have a rabbitmq cluster used as a working queue. There are 5 kinds of consumers who want to consume exactly the same data.
What I know for now is using fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
My question is, does rabbitmq support to push the same data to multi consumers? Just like a message need to be acked for a specified times to be deleted.
I got the following answer from rabbitmq email group. In short, the answer is no... and what I did above is the correct way.
http://rabbitmq.1065348.n5.nabble.com/Does-rabbitmq-support-to-push-the-same-data-to-multi-consumers-td36169.html#a36170
... fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
You can consume with 5 consumers from one queue if you do not want to duplicate messages.
does rabbitmq support to push the same data to multiple consumers
In AMQP protocol terms you publish message to exchange and then broker (RabbitMQ) decide what to do with messages - assume it figured out the queue message intended for (one or more) and then put that message on top of that queue (queues in RabbitMQ are classic FIFO queues which is somehow break AMQP implementation in RabbitMQ). Only after that message may be delivered to consumer (or die due to queue length limit or per-queue or per-message ttl, if any).
message need to be acked for a specified times to be deleted
There are no way to change message body or attributes after message being published (actually, Dead Letter Exchanges extension and some other may change routing key, for example and add,remove and change some headers, but this is very specific case). So if you want to track ack's number you have to re-publish consumed message with changed body or header (depends on where do you plan to store ack's counter, but headers fits pretty nice for this.
Also note, that there are redeliverd message attribute which denotes whether message was already was consumed, but then redelivered. This flag doesn't count redelivers number so it usage is quite limited.
I have a Topic exchange from which I'd like to distribute messages to two queues on two servers part of a cluster, in order to reduce memory pressure on any particular server. My consumers are periodically slow, and I sometimes run into the high memory watermark.
The way I tried to resolve this is by routing messages using an intermediate direct exchange, with two queues bound to the exchange:
a (topic) -> a1 (direct) -> q1/q2 (bound to routing key "a")
But the messages were routed to both queues, as AMQP intends. Anyone has ideas? What I need is an exchange that routes to one and only one queue, even if the routing key matches many queues. I'd prefer not to change my routing keys, but that could be arranged.
I found Selective routing with RabbitMQ, which may mean I'll need to implement my own routing logic. Hopefully, this already exists somewhere else.
You could perhaps use the Shovel plugin - http://www.rabbitmq.com/shovel.html - to move messages from your intermediate exchange to the two queues.
If you set up two shovels, both consuming from a single queue on the direct intermediate exchange, they should be able to fight over the messages coming in (I'm assuming that you don't care too much if the two recipient queues don't get the incoming messages in a strict round robin fashion). The shovels then each publish to one of the two end queues, and can send through the ACKs from the end consumer.