RabbitMQ — Why does the wrong subscriber get a published message? - rabbitmq

I have two services, Manager and Collector.
Manager is subscribed to Queue COLLECTED_USER with routingKey user.collected and invokes a UserCollected handler.
Collector is subscribed to Queue COLLECT_USER with routingKey user.collect and invokes a CollectUser handler.
There can be multiple collectors so I have set exclusive to false (see below for code).
There are also other services that listen for events like
user.created,
user.updated,
user.deleted
In addition there are services that listen for more general events like
#.created
user.#
and so on.
So I am using a topic exchange.
Setup
| exchange | type | routingKey | queueName |
| -------- | ----- | -------------- | ------------- |
| MY_APP | topic | user.collect | COLLECT_USER |
| MY_APP | topic | user.collected | COLLECTED_USER |
What should happen:
Manager publishes message with routingKey user.collect
Collector gets the user.collect message and invokes a CollectUser handler
Collector's CollectUser handler does work, then publishes a message with routingKey user.collected
Manager gets the user.collected message and invokes the UserCollected handler
What actually happens:
Manager publishes message with routingKey user.collect (correct)
Collector gets the user.collect message and invokes a CollectUser handler (correct)
Manager also gets the user.collect message and invokes the UserCollected handler with the wrong data. (wrong)
Collector's CollectUser handler does work, then publishes a message with routingKey user.collected (correct)
Manager gets the user.collected message and invokes the UserCollected handler (correct)
My Question
Why does the Manager get the user.collect message, given:
It's listening on the COLLECTED_USER queue not the COLLECT_USER queue, and
The Collector, which is listening on the COLLECT_USER queue, has already handled the message.
Implementation details
I create the subscribers and publishers as follows (trimmed for relevance)
Creating a Subscriber
given the AMQP url and params url, exchange, type, routingKey, queueName and handler
const connection = await amqp.connect(url)
const channel = await connection.createChannel()
channel.assertExchange(exchange, type, { durable: true })
const result = await channel.assertQueue(queueName, { exclusive: false })
channel.bindQueue(result.queue, exchange, routingKey)
channel.prefetch(1)
channel.consume(result.queue, handler)
Creating a Publisher
given the AMQP url and params url, exchange, and type
const connection = await amqp.connect(url)
const channel = await connection.createChannel()
await channel.assertExchange(exchange, type, { durable: true })
Publishing
given the channel and params exchange, routingKey, and message
await channel.publish(exchange, routingKey, message)
Note
This question is a follow-on from RabbitMQ — Why are my Routing Keys being ignored when using topic exchange
.

I finally worked out what my problem was. A dirty exchange. While experimenting with this I'd inadvertently added an exchange that was routing messages to the wrong queue, and this was causing my confusion.
To fix it I fired up the RabbitMQ admin GUI and deleted all of the queues and let my code create the ones it needed. There was no issue with the code as outlined above.

Related

How to set routing key for producer and consumer successfully

I'm building a SpringCloud Stream based application and exchange type is topic and message is sent to 2 queue consumer groups from the topic exchange. The scenario is something like this:
Service A in my application wants to send message of type appointments to service B and service C via an exchange named as: appointments-request based on different use case scenarios such as book, cancel, update etc.
So messages with a key appointments.book.B or appointments.cancel.B should go to consumer queue group appointments.B
messages with a key appointments.book.C or appointments.cancel.C should go to consumer queue group appointments.C
How to achieve this successfully?
Configuration of Producer Service:
spring.cloud.stream.bindings.output.destination=appointments-request
spring.cloud.stream.bindings.input.destination=appointments-reply
spring.cloud.stream.rabbit.bindings.output.producer.exchangeType=topic
spring.cloud.stream.rabbit.bindings.output.producer.routingKeyExpression=
appointments.#.#
Configuration of Consumer Service B:
spring.cloud.stream.rabbit.bindings.input.consumer.exchangeType=direct
spring.cloud.stream.rabbit.bindings.input.consumer.group=
appointments.docmgmt
spring.cloud.stream.rabbit.bindings.input.consumer.bindingRoutingKey=
appointments.docmgmt
spring.cloud.stream.rabbit.bindings.input.consumer.routingKeyExpression=
appointments.#.docmgmt
Producer Service A has the below method to set routing key
public boolean send(AppointmentEvent appointmentEvent)
{
logger.info("Sending event {} ",appointmentEvent);
return this.source.output().
send(MessageBuilder.withPayload(appointmentEvent).
setHeader(SimpMessageHeaderAccessor.DESTINATION_HEADER,
"appointments.book.docmgmt").build());
}
My communication between services is not working.
appointments.#.#
You can't use wildcards on the producer side.
You need something like
spring.cloud.stream.rabbit.bindings.output.producer.routingKeyExpression=headers['routingKey']
And then the producer sets the routingKey header to the desired value for each message.
You shouldn't really use the Simp headers; that is for STOMP; use your own header.

Spring Cloud Stream - publish and consume from same queue with same routing key with different headers

Producer
exchange name : xyz,
exchange type: direct,
queue: myqueue,
routingkey : myroutingkey,
producer can send different Types of Object to the queue but to distinguish b/w the objects he will send header['type'] = 'type1' along with 'myroutingkey' which is already binded withe queue.
send message : myroutingkey + header[type] = type1 to queue:
myqueue
send message : myroutingkey + header[type] = type2 to queue:
myqueue
Consumer
#StreamListner1 - consume type1 message from queue:
myqueue with routingkey : myroutingkey
#StreamListner2 - consume type2 message from queue:
myqueue with routingkey : myroutingkey
I am using Rabit MQ to achieve above requirement.
The above requirement is possible to do with spring cloud stream.if it is possible what need to do the publisher and consumer side.give me some example.
Yes, it is possible by using condition attribute on #StreamListener. For example,
#StreamListener(target = Sink.INPUT, condition = "headers['type']=='bogey'")
The details are here.

Spring AMQP- How does i retry/re-queue message from DLX queue to original queue?

i am trying to implement below scenario in my application
Exachange e1 -> Queue q1
DLX exchange e2 -> Queue q2
Also i have mentioned DLE and DLK in queue-q1 then message moving to queue-q2 on rejection/failure/timeout.
But how does i resend/retry message from queue-q2 to original queue-q1?
You can do that manually in your application after some analyze and filtering logic. Or you can make some TTL on that queue-q2 to let not consumed messages to be expired. And you also need to specify in this queue a x-dead-letter-exchange as a name for the Exachange e1 for desired recycling.
See more info yin this article:
Create the dead letter exchange, which is just a normal exchange with a special name
Create a retry_message queue and have all messages published to the dead letter exchange route here
When you setup the retry_message queue, be sure to default the following parameter values of the queue
x-message-ttl: 30000 – This will set a ttl on any message published to the queue. When the ttl expires, the message will be republished to the exchange specified in the x-dead-letter-exchange parameter.
x-dead-letter-exchange: original_exchange_name – This is where the message will get republished to once the message ttl expires. We normally want this be the name of the exchange where the message was originally published.

In RabbitMQ how to consume multiple message or read all messages in a queue or all messages in exchange using specific key?

I want to consume multiple messages from specific queue or a specific exchange with a given key.
so the scenario is as follow:
Publisher publish message 1 over queue 1
Publisher publish message 2 over queue 1
Publisher publish message 3 over queue 1
Publisher publish message 4 over queue 2
Publisher publish message 5 over queue 2
..
Consumer consume messages from queue 1
get [message 1, message 2, message 3] all at once and handle them in one call back
listen_to(queue_name , num_of_msg_to_fetch or all, function(messages){
//do some stuff with the returned list
});
the messages are not coming at the same time, it is like events and i want to collect them in a queue, package them and send them to a third party.
I also read this post:
http://rabbitmq.1065348.n5.nabble.com/Consuming-multiple-messages-at-a-time-td27195.html
Thanks
Don't consume directly from the queue as queues follow round robin algorithm(an AMQP mandate)
Use shovel to transfer the queue contents to a fanout exchange and consume messages right from this exchange. You get all messages across all connected consumers. :)
If you want to consume multiple messages from specific queue, you can try as below.
channel.queueDeclare(QUEUE_NAME, false, false,false, null);
Consumer consumer = new DefaultConsumer(channel){
#Override
public void handleDelivery(String consumerTag,
Envelope envelope,
AMQP.BasicProperties properties,
byte[] body)
throws IOException {
String message = new String(body, "UTF-8");
logger.info("Recieved Message --> " + message);
}
};
You might need to conceptually separate domain-message from RMQ-message. As a producer you'd then bundle multiple domain messages into a single RMQ-message and .produce() it to RMQ. Remember this kind of design introduces timeouts and latencies due to the existence of a window (you might take some impression from Kafka that does bundling to optimize I/O at the cost of latency).
As a consumer then, you'd have a consumer, with typical .handleDelivery implementation that would transform the received body for the processing: byte[] -> Set[DomainMessage] -> your listener.

rabbitmq multiple consumer and multiple publisher

Want to know the behavior of rabbitmq multiple publisher and consumer.
Does rabbitmq server gives one message to any one of the consumer at a time and other consumers are ideal at that time?
OR
Consumers pick any unattended message from queue, so that at a time, more than one consumers are consuming the message from queue?
Basically I am designing a database queue and do not want more than one inserts at a time.
A message from the queue will be delivered to one consumer only. Ie: once the message makes its way to the queue - it won't be copied (broadcasted) to multiple consumers.
If you want to do broadcast - you have to use multiple queues.
See this tutorial for more details:
http://www.rabbitmq.com/tutorial-two-python.html
yes , RabitMQ supports multiple publisher and consumer.
Multiple Publisher
For publishing a messsge to rabbitmqyou need to declare a factory and do a connection to the rabbitmq server.
then decare a chennel to rabbitmq
ConnectionFactory FACTORY = new ConnectionFactory
FACTORY.setUsername ("guest")
FACTORY.setPassword ("guest")
FACTORY.setVirtualHost ("\")
FACTORY.setPort (5572)
FACTORY.setHost ("localhost")
Connection connection=FACTORY.newConnection
Channel channel = connection.createChannel
the basic key to route a message is a routing key
channel.basicPublish(EXCHANGE_NAME, "Queue1", MessageProperties.PERSISTENT_TEXT_PLAIN, "msg1".getBytes)
channel.basicPublish(EXCHANGE_NAME, "Queue2", MessageProperties.PERSISTENT_TEXT_PLAIN, "msg2".getBytes)
these two messages will be published to a seperate queue as per the routing key as mention queue1 and queue2
2.Multiple Consumer
for multiple consumer we declare a queue and bind to a particular routing key
the the message to that routing key will be publishe to respected queue.
channel.exchangeDeclare(EXCHANGE_NAME, "direct", durable)
channel.queueDeclare("q1", durable, false, false, null)
channel queueBind ("q1", EXCHANGE_NAME,"queue1")// routing key = "queue1"
val q1Consumer = new QueueingConsumer(channel)
channel basicConsume ("q1", false, q1Consumer)
like this u can consume messages from first queue
and same goes for second queue but specify the routing key as "queue2"
channel.exchangeDeclare(EXCHANGE_NAME, "direct", durable)
channel.queueDeclare("q2", durable, false, false, null)
channel queueBind ("q2", EXCHANGE_NAME,"queue2") // routing key = "queue2"
val q2Consumer = new QueueingConsumer(channel)
channel basicConsume ("q2", false, q2Consumer)