I have producer app with 2 separate instances (p1, p2), and consumer app with 2 separate instances (c1, c2).
Producer p1 connects to exchange with topic= t1, queueName =name1.
Consumer c1 connects to exchange with topic= t1, queueName =name1.
Producer p2 connects to exchange with topic= t2, queueName =name1.
Consumer c2 connects to exchange with topic= t2, queueName =name1.
I see in RabbitMQ GUI that I have 2 exchanges but only 1 queue.
Instead that c1 will receive messages from p1 only, and c2 will receive messages from p2 only, RabbitMQ is doing round robin on messages between c1 and c2. So the messages I send from p2 are being received both by c1 and c2.
I thought that in RabbitMQ the correlation is multiple queues per exchange, and the behavior here is unexpected. Why?
You can have multiple queues for every exchange, it's true; but the routing key is a queue matter, not a consumer matter.
The routing key will be used by rabbit to send the message to the right queue; once the message is received on a topic exchange, the message will be sent to all the queues binded to that specific topic. You have only one queue here, that's why both C1 and C2 get the message.
Check this link for a clear example.
If you need to separate C1 and C2, you need to bind them to 2 different queues, not to the same one.
Related
I'm building a SpringCloud Stream based application and exchange type is topic and message is sent to 2 queue consumer groups from the topic exchange. The scenario is something like this:
Service A in my application wants to send message of type appointments to service B and service C via an exchange named as: appointments-request based on different use case scenarios such as book, cancel, update etc.
So messages with a key appointments.book.B or appointments.cancel.B should go to consumer queue group appointments.B
messages with a key appointments.book.C or appointments.cancel.C should go to consumer queue group appointments.C
How to achieve this successfully?
Configuration of Producer Service:
spring.cloud.stream.bindings.output.destination=appointments-request
spring.cloud.stream.bindings.input.destination=appointments-reply
spring.cloud.stream.rabbit.bindings.output.producer.exchangeType=topic
spring.cloud.stream.rabbit.bindings.output.producer.routingKeyExpression=
appointments.#.#
Configuration of Consumer Service B:
spring.cloud.stream.rabbit.bindings.input.consumer.exchangeType=direct
spring.cloud.stream.rabbit.bindings.input.consumer.group=
appointments.docmgmt
spring.cloud.stream.rabbit.bindings.input.consumer.bindingRoutingKey=
appointments.docmgmt
spring.cloud.stream.rabbit.bindings.input.consumer.routingKeyExpression=
appointments.#.docmgmt
Producer Service A has the below method to set routing key
public boolean send(AppointmentEvent appointmentEvent)
{
logger.info("Sending event {} ",appointmentEvent);
return this.source.output().
send(MessageBuilder.withPayload(appointmentEvent).
setHeader(SimpMessageHeaderAccessor.DESTINATION_HEADER,
"appointments.book.docmgmt").build());
}
My communication between services is not working.
appointments.#.#
You can't use wildcards on the producer side.
You need something like
spring.cloud.stream.rabbit.bindings.output.producer.routingKeyExpression=headers['routingKey']
And then the producer sets the routingKey header to the desired value for each message.
You shouldn't really use the Simp headers; that is for STOMP; use your own header.
i am trying to implement below scenario in my application
Exachange e1 -> Queue q1
DLX exchange e2 -> Queue q2
Also i have mentioned DLE and DLK in queue-q1 then message moving to queue-q2 on rejection/failure/timeout.
But how does i resend/retry message from queue-q2 to original queue-q1?
You can do that manually in your application after some analyze and filtering logic. Or you can make some TTL on that queue-q2 to let not consumed messages to be expired. And you also need to specify in this queue a x-dead-letter-exchange as a name for the Exachange e1 for desired recycling.
See more info yin this article:
Create the dead letter exchange, which is just a normal exchange with a special name
Create a retry_message queue and have all messages published to the dead letter exchange route here
When you setup the retry_message queue, be sure to default the following parameter values of the queue
x-message-ttl: 30000 – This will set a ttl on any message published to the queue. When the ttl expires, the message will be republished to the exchange specified in the x-dead-letter-exchange parameter.
x-dead-letter-exchange: original_exchange_name – This is where the message will get republished to once the message ttl expires. We normally want this be the name of the exchange where the message was originally published.
Suppose we have 3 nodes in a cluster.
node1,node2,node3
In node1 we have a
exchange e1 bounded to a queue q1 with binding key =key1
It is attached to a consumer1.
In node2 we have a
exchange e2 bounded to a queue q2 with binding key =key2
It is attached to a consumer2.
Can consumer2 read messages from q1 in cluster ? If not how can this be implemented ?
you can read rabbitMQ route totorial.Though it's using python,the concept would be the same.In the Putting it all together part the consumer 2 can receive info,error and warning from queue 2 while the consumer 1 get error from queue 1.
In your case,c2 can't read message from queue 1 now.To implement,the exchange setting don't need to change.Just bind queue 2 with exchange 1 key 1.
Say I have this pub/sub pattern implemented:
So basically I deliver a message to each C, who subscribed to exchange X.
I have instances of P, and a lot subscribers like C. Let's define C10, C11, C12, C13 as a group C1 and C20, C21, C22, C23 as a group C2.
How do I deliver a message so only one C will receive a message from each group? (I'm perfectly fine with round robin)
Just go to topics tutorial.
Routing key should look like C.C1 or C.C2.
Basically, send messages with routing key C.* ( so it they will go to C.C1 xor C.C2) , and subscribe each consumer to C.C1 xor C.C2. RMQ will distribute messages to all consumers subscribed to C.CN routing key in round-robin fashion.
So appears what I needed was a fanout exchange with named queues instead of exclusive ones.
Each C service declares a non-exclusive named queue and binds it to the exchange. And binds a consumer to that queue.
If two services would declare a same queue, and bind a consumer to it, they end up being round-robin'ed.
What will happen to messages posted to a virtual topic when there are no consumers listening ? Will the broker hold them for a certain while until a subscriber is available ?
More specifically :
At T0 and T1 messages M0 and M1 are posted. At T2, consumer C1 connects, will he receive M0 and M1 ? Obviously messages M2 and M3 posted at T3 and T4 will be received by C1, but what will a new Consumer, C2, that connects at T5 receice ? All messages, M2 and M3, or none ?
It depends on the nature of the topic:
if the topic is durable (has durable consumers subscribing to it), the broker will hold the messages in the topic until all the durable consumers consumes the messages.
if the topic is non-durable (no durable consumers), the message will not even be sent to the topic, as there will be no durable subscription.
For your example, I'll consider that you are using durable subscriptions / consumers:
Case 1:
T-2 C1 and C2 make durable subscription to the topic
T-1 C1 and C2 disconnect
T0: M0 is posted
T1: M1 is posted
T2: C1 connects. C1 receives M0 and M1
T3: M3 is posted. C1 receives M3
T4: M4 is posted. C1 receives M4
T5: C2 connects, C2 receives M0, M1, M2, M3, M4
That's because they are holding durable subscriptions
You need to be very careful when using durable topics / queues: if the consumer doesn't unsubscribe, the broker will hold the messages until the message store explodes. You will need to make sure it doesn't happen (by setting eviction policies and / or putting a Time to Live on the messages).
Of course the previous example will vary depending when the consumer does the durable subscription.
If you are using non-durable topics:
T-2 C1 and C2 make normal subscription to the topic
T-1 C1 and C2 disconnect
T0: M0 is posted
T1: M1 is posted
T2: C1 connects. C1 does not receive anything
T3: M3 is posted. C1 receives M3
T4: M4 is posted. C1 receives M4
T5: C2 connects, C2 does not receive anything
There are two ways to allow messages published to a virtual topic to suivive. The first one is through the durable subscriber and the other is that the publisher sends messages with delivery mode "PERSISTENT". When messages are published with the delivery mode of "PERSISTENT", the message will be saved on disk, otherwise, it will be save in-memory.
Why can't there be a observer/observable pattern - taking the example above:
When M0 is posted, C1 and C2 (consumers subscribed) are woken and can consume the event? I see this pattern better than the durable and non-durable - a hybrid approach.