How to remove message from a queue , in a bining configuration, once the message has been consumed?
Example:
I have
topic.1, topic.2, topic.3,...etc
I have two binding, like my_topic which is binded to topic.1 and all_topic which is binded to topic.#
my_topic is high priority, and messages are consumed normally, but remains in the topic.# binding. So how can I remove them from the topic.# so they are not consumed twice?
Thanks in advance
Split your topics in more queues. So instead of having one queue subscribed to messages with routing key "users.*", create one for each routing key: "users.created", "users.updated", "users.deleted". Then you can create new consumers to bind to this queue on the fly when you have more load in one kind of message.
Related
I am using RabbitMq 3.7.16 and I have an exchange that I want to bind to one of two possible queues.
My use case is having the exchange bound to first queue and switching it to second queue, and then switch the binding back.
My Current Implementation
The exchange is of type direct and routing key for both is "".
When switching binding I perform Bind(second-queue) and then Unbind(first-queue).
When switching back I do the opposite.
The problem
I've few milliseconds where both queues are bound to the exchange and thus receiving the same messages. I want every message to get to exactly one of the possible queues.
What is the right way to do such thing with RabbitMQ?
My Solution
Have two exchanges, one bound to first-queue and the other to second-queue.
I defined 'alternate-exchange' property to first-exchange that will forward any message that cannot be routed to second-exchange.
When I want the messages to arrive to second-queue I simply unbind first-queue from the first-exchange. This make messages unroutable in the exchange and thus forwarded to second-exchange. When I want the messages to arrive again to first-queue I bind it back up.
Using RabbitMQ 3.7.16, with spring-amqp 2.2.3.RELEASE.
Multiple clients publish messages to the DataExchange topic exchange in our RabbitMQ server, using a unique routing key. In the absence of any bindings, the exchange will route all the messaged to the data.queue.generic through the AE.
When a certain client (client ID 1 and 2 in the diagram) publishes lots of messages, in order to scale the consumption of their messages independently from other clients, we are starting consumers and assign them to only handle a their client ID. To achieve this, each client-consumer is defining a new queue, and it binds it to the topic exchange with the routing key events.<clientID>.
So scaling up is covered and works well.
Now when the messages rate for this client goes down, we would like to also scale down its consumers, up to the point of removing all of them. The intention is to then have all those messages being routed to the GenericExchange, where there's a pool of generic consumers taking care of them.
The problem is that if I delete data.queue.2 (in order to remove its binding which will lead to new messages being routed to the GenericExchange) all its pending messages will be lost.
Here's a simplified architecture view:
It would be an acceptable solution to let the messages expire with a TTL in the client queue, and then dead letter them to the generic exchange, but then I also need to stop the topic exchange from routing new messages to this "dying" queue.
So what options do I have to stop the topic exchange from routing messages to the client queue where now there's no consumer connected to it?
Or to explore another path - how to dead letter messages in a deleted/expired queue?
If the client queue is the only one with a matching binding as your explanation seems to suggest, you can just remove the binding between the exchange and the queue.
From then on, all new messages for the client will go through the alternate exchange, your "generic exchange", to be processed by your generic consumers.
As for the messages left over in the client queue, you could use a shovel to send them back to the topic exchange, for them to be routed to the generic exchange.
This based on the assumption the alternate exchange is internal. If it's not internal, you can target it directly with the shovel.
As discussed with Bogdan, another option to resolve this while ensuring no message loss is occuring is to perform multiple steps:
remove the binding between the specific queue and the exchange
have some logic to have the remaining messages be either consumed or rerouted to the generic queue
if the binding removal occurs prior to the consumer(s) disconnect, have the last consumer disconnect only once the queue is empty
if the binding removal occurs after the last consumer disconnect, then have a TTL on messages with alternate exchange as the generic exchange
depending on the options selected before, have some cleanup mecanism to remove the lingering empty queues
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
I have a scenario as below
Exchange is of type "topic" and i have two queues(queue1 and queue2) binded to it with routing key #.
Each queue is having 1 consumer. Now, when i send a message from publisher both queues are getting my message since its binded to routing key # .
Is there any way, we can send messages to only one queue on round robin basis ?
Since, each message is getting into both queues , i am getting duplication of messages.
If you are looking for round robin consuming you shoud see this pattern:
https://www.rabbitmq.com/tutorials/tutorial-two-java.html
So basically you publish to one queue and then you add consumers to the same queue.
through the QoS = 1 you have the round-robin. In this way you can add more consumers without create new queues
Is it at all possible to declare a binding between an existing exchange and a non-existing queue, so that when the queue (eventually) gets created by some other means in the future messages will start to get forwarded to it?
Is it at all possible to declare a binding between an existing exchange and a non-existing queue,
this is not possible.
you can only bind an exchange to an existing queue. you can only set up a consumer to get messages from an existing queue.
so that when the queue (eventually) gets created by some other means in the future messages will start to get forwarded to it?
sort of... when you create a queue and binding, messages will start flowing to that queue. but only new messages. old messages are lost and will not flow to that queue.
If you are dynamically creating queues and bindings for your consumers, then your consumer should be the one to declare the queues. The problem, as you've probably run in to, is that you will not have any messages in the queue until the queue is created and bound.
If you need messages to be there before the consumer connects, then some other code needs to set up the queue and binding before the consumer connects and starts consuming from the queue.