I have a scenario where there are multiple sources sent messages to different queues and every source can send to multiple queues. I also have consumer prefetch equal to 1 for every consumer.
which is better to create one channel with multiple exchanges for every source and each exchange has its own queues or creates different channels for every source and every channel has one exchange?
Related
Good afternoon!
What is the best way to implement the next:
I publish messages about employees. I want that consumer process messages about several employees. For example - I send messages about employees 1,2,3,4,5. I want to process messages in the next way: Consumer 1 - receives messages about employees 1,3,5; Consumer 2 - receives messages about employees 2,4,5
If I create a certain queue for each employee, how can I subscribe the consumer for greater then one queue?
I can send messages in the one queue with the different routing keys (EmployeeID). In this way - how can I filter messages in this queue by the multiple routing keys for different consumers?
I found the solution. I created exchange and the separate queues with the specific binding
TTL can be set on queues, so they will expire after a period of time if they are not used. Is there a similar option for exchanges?
I'm trying to build a social application and each exchange represents a user. Each time someone wants to send a message to this user, he would send the message to the user exchange. If number of the users become large, like 20 million users, there would be 20 million exchanges in the system. I'm afraid that much exchanges degrade the system. Instead I want to only keep exchanges for online users.
By the way the messages are only valuable if the user is online and I don't want to store messages for later delivery.
Having a separate exchange for every user ideed would be overkill. Try a different approach.
Use a single direct exchange.
When a client comes online it creates a new exclusive, auto-delete queue and consumes from it.
The client also binds the single exchange to its queue using the name of the user as the routing key.
Producers publish messages to the single exhange with the name of the user as the routing key of the message.
This will automatically
only keep queues for online users and
discard messages for offline users.
Edit: If a user shall be able to use multiple clients, that's possible using the above approach.
Every client creates a new exclusi auto-delete queue and consumes from it as above.
It binds the single exchange to this queue as above.
Note that it is possible to have multiple bindings from an exchange using identical routing keys. Every client has its own queue and its own binding, even if the routing key on this binding is the same routing key as on another binding created by a different client.
As in kafka offset is used to manage the sequence of data that needed to be transfer to consumer, how this type of management is done in RabbitMQ to manage the data sequence to prevent data loss.
RabbitMQ consumers do not maintain a client side offset like Kafka consumers must. Kafka stores all messages for a configured time period and clients manage their own offsets meaning that different clients can consume messages from different offsets within the queue. This means that different Kafka consumers are not competing consumers unless they coordinate and share their offset.
RabbitMQ is very different. Messages are stored until they get sent to a consumer and then acknowledged or they expire. If there are multiple consumers for one queue then they are competing consumers and one message can only be consumed by one of the many consumers.
RabbitMQ has the concept of a delivery tag which is a monotonically incrementing number that increments per message delivered over a channel. It has no global meaning, it has the scope of a single channel and so is not shared between consumers on different servers, processes. Consumers only need to track this number for message acknowledgement purposes, it is RabbitMQ that is responsible for choosing which message is delivered to which consumer.
We're seeing an issue where consumers of our message queues are picking up messages from queues at the top of the alphabetical range. We have two applications: a producer, and a subscriber. We're using RabbitMQ 3.6.1.
Let's say that the message queues are setup like so:
Our first application, the producer, puts say 100 messages/second onto each queue:
Our second application, the subscriber, has five unique consumer methods that can deal with messages on each respective queue. Each method binds to it's respective queue. A subscriber has a prefetch of 1 meaning it can only hold one message at a time, regardless of queue. We may run numerous instances of the subscriber like so:
So the situation is thus: each queue is receiving 100 msg/sec, and we have four instances of subscriber consuming these messages, so each queue has four consumers. Let's say that the consumer methods can deal with 25 msg/sec each.
What happens is that instead of all the queues being consumed equally, the alphabetically higher queues instead get priority. It's seems as though when the subscriber becomes ready, RabbitMQ looks down the list of queues that this particular ready channel is bound to, and picks the first queue with pending messages.
In our situation, A_QUEUE will have every message consumed. B_QUEUE may have some consumed in certain race conditions, but C_QUEUE/D_QUEUE and especially E_QUEUE will rarely get touched.
If we turn off the publisher, the queues will eventually drain, top to bottom.
Is it possible to configure either RabbitMQ itself or possibly even the channel to use some sort of round robin distribution policy or maybe even random policy so that when a channel has numerous bound queues, all with messages pending, the distribution is even?
to clarify: you have a single subscriber application with multiple consumers in it, right?
I'm guessing you're using a single RabbitMQ Connection within the subscriber app.
Are you also re-using a single RabbitMQ Channel for all of your consumers? If so, that would be a problem. Be sure to use a new Channel for each consumer you start.
Maybe the picture is wrong, but if it's not then your setup is wrong. You don't need 4 queues if you are going to have subscribers that listen to each and every queue. You'd just need one queue, that has multiple instances of the same subscriber consuming from it.
Now to answer, yes (but no need to configure, as long as prefetch is 1), actually rabbitmq does distribute messages evenly. You can find about about that here, and on the same place actually how your setup should look like. Here is a quote from the link.
RabbitMQ just dispatches a message when the message enters the queue.
It doesn't look at the number of unacknowledged messages for a
consumer. It just blindly dispatches every n-th message to the n-th
consumer.
I have a rabbitmq cluster used as a working queue. There are 5 kinds of consumers who want to consume exactly the same data.
What I know for now is using fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
My question is, does rabbitmq support to push the same data to multi consumers? Just like a message need to be acked for a specified times to be deleted.
I got the following answer from rabbitmq email group. In short, the answer is no... and what I did above is the correct way.
http://rabbitmq.1065348.n5.nabble.com/Does-rabbitmq-support-to-push-the-same-data-to-multi-consumers-td36169.html#a36170
... fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
You can consume with 5 consumers from one queue if you do not want to duplicate messages.
does rabbitmq support to push the same data to multiple consumers
In AMQP protocol terms you publish message to exchange and then broker (RabbitMQ) decide what to do with messages - assume it figured out the queue message intended for (one or more) and then put that message on top of that queue (queues in RabbitMQ are classic FIFO queues which is somehow break AMQP implementation in RabbitMQ). Only after that message may be delivered to consumer (or die due to queue length limit or per-queue or per-message ttl, if any).
message need to be acked for a specified times to be deleted
There are no way to change message body or attributes after message being published (actually, Dead Letter Exchanges extension and some other may change routing key, for example and add,remove and change some headers, but this is very specific case). So if you want to track ack's number you have to re-publish consumed message with changed body or header (depends on where do you plan to store ack's counter, but headers fits pretty nice for this.
Also note, that there are redeliverd message attribute which denotes whether message was already was consumed, but then redelivered. This flag doesn't count redelivers number so it usage is quite limited.