I have the following scenario:
One Producer service
A dynamic amount of consumers services
Messages contain tasks with a specific product, so once consumer x handles a message of product y. In the future x should handle all messages of product y. Ideally the producer service should send all messages of product x on a queue which only consumer x reads from.
In order to divide workload evenly, there should be a way that once a new product needs to be managed, that next available consumer takes it.(I suppose a queue which all consumers are reading from)
My approach:
An exchange send new product jobs in a "newProduct" queue to which all the consumers are consuming from.
The consumer y that reads such a message notifies to the producer service (on a separate queue) that he is now in charge of product x.
The producer then sends all messages for product x to a queue proper to consumer y.
When a new consumer service z goes online, it notifies the producer service on a therefore specific queue that he is online such that the producer can create a binding in the exchange for z's proper queue.
Questions:
is my approach a good way to solve the problem, or am I missing rabbitmq solutions that would solve the problem in a less complicated way ?
How do I add a new queue during runtime to the exchange ?
An exchange send new product jobs in a "newProduct" queue to which all
the consumers are consuming from.
This looks good to me.
The consumer y that reads such a message notifies to the producer
service (on a separate queue) that he is now in charge of product x.
This is also fine, I guess if producer did not receive notification
that product X is taken care of it will need to do something. The
producer then sends all messages for product x to a queue proper to
consumer y.
I'd send all messages for product X with the same routing key, like product-X. Which is what you probably mean here. I'd avoid telling producer who exactly handles the product-X now. For better separation of concerns and simplicity producers should know as less as possible about consumers and their queues and vice versa.
When a new consumer service z goes online, it notifies the producer
service on a therefore specific queue that he is online such that the
producer can create a binding in the exchange for z's proper queue.
You could do it this way, but I'd do it differently:
When consumer goes online, it will create needed queues (or subscribe to existing queues) by itself.
I see it like this:
Consumer comes online and subscribes to newProduct queue.
When received a message to handle product Z:
Creates a new queue for itself with binding key product-Z
Notifies producer that product Z is now being handled
Producer starts to send messages with routing key product-Z and they end up in Consumer's queue.
Make sure your consumer has some High Availability, otherwise you may end up in the situation when your consumer started to handle some of the messages and then gone dead, while producer is continuing to send messages for now unhandled product.
Related
At the moment we have number of publishers (micro-services) which publish their messages to exchange. Each message has a serviceId attribute. The queue is connected to a single subscriber (micro-service) which processes the queue messages, processing of a single message is a costly operation (takes about 20-30 secs).
Currently we have the following situation: service A publishes ~200 messages, after some seconds service B publishes 2 messages. So the subscriber will process these 2 messages only after the first 200 will be processed.
We want to process the messages in the order they came to the queue, but with respect to the source serviceId.
Obvious solution is to split the queue to a separate queues (one per publisher) and subscribe to each queue separately, but the number of publishers can change, we need to request them dynamically and subscribe (unsubscribe) to them.
Another approach is to replicate our subscriber app to have one to one relationship between publisher and subscriber, but this will require more system resources.
What would be the best approach to handle this situation?
Thanks!
/!\ Be careful, publishers publish to an exchange, not to a queue.
We want to process the messages in the order they came to the queue,
but with respect to the source serviceId.
If I understand well, you want to load balance your messages according to a serviceId, and serviceIds are not known in advance.
The solution I would suggest here is to have a direct exchange, with routing keys such as xxxxx.<serviceId>. Then, you can bind one queue by serviceId (that is: one queue for service A, one for service B, ...), each consumer consuming on all queues.
Then you have to handle the publisher subscription: I would make a publisher publish a "hello" message, this message being consumed by each consumer, which in turn bind a new queue for that service (using xxxxx.<newServiceId>), and finally publish a response back (so that the publisher can start sending messages).
Note: each service queue is the same for all consumers, resulting in the worker configuration (see this tutorial)
Hope this helps.
Let's suppose we have one producer, one queue and some consumers which are subscribed on queue.
Producer -> Queue -> Consumers
Queues contains messages about life events. These messages should receive all consumers.
When queue will be erased?
When all consumers get message?
Or when one of consumers confirm message with flag ack (true)?
And how to manage priority, who from consumers must to get message first/last (don't confuse with message priority).
As instance I have 10 consumers and I want that the fifth consumer get message first, remaining consumers later after specified time.
Be careful: when there are many consumers on one queue, only one of them will receive a given message, provided that it is consumed and acked properly. You need to bind as many queues as consumers to an exchange to have all consumers receive the message.
For your priority question, there is no built-in mecanism to have consumers receive the same message with a notion of priority: consumer priority exists (see https://www.rabbitmq.com/consumer-priority.html), but it is made to have consumer receive a given message before the others on a given queue, so the other consumers won't receive this message. It you need to orchestrate the delivery of your messages, you have to think of a more complex system (maybe a saga or a resequencer?).
Note that you can delay messages using this pattern. Again, this requires having multiple queues.
Finally, there are many scenarios when a queue is deleted. Take a look at the documentation, these are well explained.
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
I've defined one topic exchange (alarms) and multiple queues, each with its own routing key:
allAlarms, with routing key alarms.#: I want this to be used for receiving all alarms in a monitoring application
alarms_[deviceID], with routing key alarms.[deviceID], where the number of devices can vary at any given time
When sending an alarm from the device, I publish it using the routing key alarms.[deviceID]. The monitoring app, however, only consumes from the allAlarms queue. This leads to the following problem:
The messages in the allAlarms queue have been consumed, while the messages in the remaining queues are ready. Is there a better way of handling messages from multiple consumers? Ideally, I'd like to be able to also send commands back to the devices using the same queues where the devices publish their alarms.
It looks like you have consumers bound to the allAlarms queue but not to any of the alarms_[deviceID] queues.
In AMQP, a single consumer is bound to a single queue by name (and each queue can have multiple consumers bound to it). Messages are delivered to the consumers of a queue in round robin such that for a given message in a queue there is exactly one consumer that will receive the message. That is, consumers cannot listen to multiple queues.
Since you're using a topic exchange, you're correctly routing a single message to multiple queues via the routing key and queue bindings. This means that you can have a consumer for each queue and when a message is delivered to the exchange, each queue will get a copy of the message and each queue will deliver the message to exactly one consumer on each queue.
Thus, if allAlarms is consuming messages, it's because it has a consumer attached to the queue. If any of the alarms_[deviceID] are not consuming messages then they must not have consumers bound to those individual queues. You have to start up consumers for each alarms_[deviceID] by name. That will allow you to also have different consumer logic for different queues.
One last thing:
Ideally, I'd like to be able to also send commands back to the devices using the same queues where the devices publish their alarms.
You don't want to do this using the same queue because there's nothing that will stop the non-device consumers on the queue from picking up those messages.
I believe you're describing RPC over RabbitMQ. For that you will want to publish the messages to the alarms queues with a reply-to header which is the name of a temporary queue. This temp queue is a single-use queue that the consumer will publish to when it's done to communicate back to the device. The device will publish to the alarms exchange and then immediately start listening to the temp queue for a response from the consumer.
For more info on RPC over RabbitMQ check out this tutorial.
I don't think you need any of the queues for the devices - the alarm_[deviceid] queues.
You don't have any consumer code set up on these queues, and the messages are backed up and waiting for you to consume them.
You also haven't mentioned a need to consume messages from these queues. Instead, you are only consuming messages form the alarmAll queue.
Therefore, I would drop all of the alarm_[deviceid] queues and only have the alarmAll queue.
Just publish the alarms through your exchange and route them all to the alarmAll queue and be done with it. No need for any other routing or queues.
I have trouble understanding the routing in RabbitMQ. Consider I have several producers (let call them clients) that produce messages to the queue. E.g., clients A, B, and C send messages to queue X1.
Let the consumer respond to all messages sending responses back to the queue. E.g., consumer gets message from queue X1, does something, and sends responses to the queue X1.
How can, client A determine where are in the queue X1 messages sent to it and where are messages sent to clients B or C?
I can't declare one queue per connection because of large number of connections expected (~10^6). So I'm in trouble here. Any suggestions? Thanks.
I think you need to look at the RPC tutorial. From your description it sounds like that is what you want to do. However that would probably require you to declare more queues than you want.
Approaching this a different way. I cannot understand why you would send a reply back to the producer not only by the same exchange but the same queue that the consumers are consuming from.
Would it not make sense to have producers P1,P2 and P3 send to exchange X1 with routing key "abc.aaa.xyz" / "abc.bbb.xyz" / "abc.ccc.xyz". Then have queues Q1, Q2 and Q3 bound to X1 with binding keys ".aaa." / ".bbb." / ".ccc." or just Q1 with binding key "abc.*.xyz" (I am unclear on exactly what you want so just making some suggestions). Which are consumed by Consumers C1, C2 and C3
When the Consumer has finished processing the message then it will send a message to X2, with routing key that identifies itself. The producers will consume from queues bound to X2.
The point I am trying to make is that you do not want more than one consumer reading from a queue. There is only one case in which you want that and that is a task queue. I am not clear on your use case so you may want a task queue. If you do then you should still not have your producers reading from the same task queue as your consumers. Aside from task queues you should have one consumer read from one queue. You may have many queues to one exchange and even many bindings from one queue to one exchange.
I hope this helps