How do you consume using Spring Cloud Stream (that is using AMQP) messages sent with MQTT to RabbitMQ?
With MQTT on Rabbit, all messages land on an exchange called "amq.topic".
On the consumer side, with Spring Cloud Stream, a queue is created for each destination and an exchange of type "topic" with the name of the destination; the created queue is bound to the exchange.
Now,
I cannot manually bind my queue to the "amq.topic", because it is exclusive:
cannot obtain exclusive access to locked queue '...' in vhost '/'. It could be originally declared on another connection or the exclusive property value does not match that of the original declaration.
I cannot listen directly from the exchange "amq.topic", because, well, you have to listen to a queue...
I created a "tmp" queue bound to "amq.topic" where messages are delivered, but I cannot use it as destination, because RMQ will create a new queue called "tmp.SOME_CLIENT_ID", bound to an exchange called "tmp", that has nothing to do with my "tmp" queue.
Any idea would be welcome!
See Using Existing Queues/Exchanges.
By default, the binder will automatically provision a topic exchange with the name being derived from the value of the destination binding property <prefix><destination>. The destination defaults to the binding name, if not provided. When binding a consumer, a queue will automatically be provisioned with the name <prefix><destination>.<group> (if a group binding property is specified), or an anonymous, auto-delete queue when there is no group. The queue will be bound to the exchange with the "match-all" wildcard routing key (#) for a non-partitioned binding or <destination>-<instanceIndex> for a partitioned binding. The prefix is an empty String by default. If an output binding is specified with requiredGroups, a queue/binding will be provisioned for each group.
There are a number of rabbit-specific binding properties that allow you to modify this default behavior.
If you have an existing exchange/queue that you wish to use, you can completely disable automatic provisioning as follows, assuming the exchange is named myExchange and the queue is named myQueue:
spring.cloud.stream.bindings.<binding name>.destination=myExhange
spring.cloud.stream.bindings.<binding name>.group=myQueue
spring.cloud.stream.rabbit.bindings.<binding name>.consumer.bindQueue=false
spring.cloud.stream.rabbit.bindings.<binding name>.consumer.declareExchange=false
spring.cloud.stream.rabbit.bindings.<binding name>.consumer.queueNameGroupOnly=true
If you want the binder to provision the queue/exchange, but you want to do it using something other than the defaults discussed here, use the following properties. Refer to the property documentation above for more information.
spring.cloud.stream.rabbit.bindings.<binding name>.consumer.bindingRoutingKey=myRoutingKey
spring.cloud.stream.rabbit.bindings.<binding name>.consumer.exchangeType=<type>
spring.cloud.stream.rabbit.bindings.<binding name>.producer.routingKeyExpression='myRoutingKey'
Related
I am integrating a NestJS application with RabbitMQ and i followed this tutorial
From the tutorial i can see that message consumer can connect to the rabbitMQ broker using a queue name, but actually i want to create a temporary queue that connects to an exchange. I cannot find an option to specify the exchange name in the queue options specified in the tutorial. can someone please point to the right configuration to use to specify an exchange instead of queue?
is it possible to specify an exchange name inside the 'queueOptions' structure?
From what I understand from your tutorial (I don't know NestJS), createMicroservice only connects to a queue (or create it if needed). In other words, it performs assertQueue operation:
var queue = 'task_queue';
channel.assertQueue(queue, {
durable: false
});
If you want to bind a queue to an existing exchange, you need to perform bindQueue operation:
channel.bindQueue(queue, exchange, '');
Thus you can't bind to an exchange using queueOptions. Take a look at the official Rabbitmq documentation, especially the "Temporary queues" and following "Bindings" sections.
Is there a way to configure some policy to redirect messages from one exchange to another exchange, if the current exchange doesn't have any binded queue?
I got a situation where RabbitMQ after restart loose some bindings and my messages were gone.
Yes, the feature is called Alternate Exchanges, and it works like this:
You set a "policy" which matches exchange A, with a key of 'alternate-exchange' and a value naming exchange B. Alternatively, you can set it as a property directly when creating the exchange.
Any message routed to exchange A which doesn't match any binding will be routed to exchange B.
Exchange B processes it in the normal way, and can even have an Alternate Exchange of its own.
I use rabbitmq as the middleware in cloud stream ,when I configure the application.yml
spring:
cloud:
stream:
bindings:
usertest: #channelName
content-type: application/json
group: testGroup
destination: topic888 #destination，
binder: rabbit1
I can't understand the key meaning of destination.
official definition is :
destination
The target destination of a channel on the bound middleware (e.g., the RabbitMQ exchange or Kafka topic). If the channel is bound as a consumer, it could be bound to multiple destinations and the destination names can be specified as comma separated String values. If not set, the channel name is used instead.
Also,I found when consumer and producer have different destinations , It did't stop consumer from consuming.
The destination depends on the binder type. For kafka, it's the topic; for RabbitMQ, it's an exchange.
The producer publishes to the exchange.
The consumer binds a queue to the exchange. Anonymous consumers bind a temporary, auto-delete queue, consumers with a group bind a queue called topic888.testGroup in your case (destination.group).
Also,I found when consumer and producer have different destinations , It did't stop consumer from consuming.
I don't see how that's possible, with the binder doing the provisioning, for the reasons discussed above.
It's possible if you manually bind a queue with the correct name to a different exchange that the producer is publishing to.
Using a SimpleMessageListenerContainer that is attached to multiple queues and configured with a ChannelAwareMessageListener. Is it possible to determine which queue a message has been consumed from? In particular if the message was routed to the queue from an Exchange.
It looks that if a message is sent directly to a queue that the MessageProperties#getReceivedRoutingKey will contain the queue name but if the message is routed to a queue via an Exchange then this information contains the routing key that was used.
I'm looking for a mechanism that would allow this information to be extracted correctly regardless of how the message was delivered to the queue. Or a mechanism to enrich the information with a header containing this information on the RabbitMQ side.
I had a similar issue where I wanted to add the queue name to slf4j MDC context.
The only solution I have found is to subclass SimpleMessageListenerContainer and set a ThreadLocal variable for the queue name or in my case the MDC context (which is basically threadlocals).
Because SimpleMessageListenerContainer still doesn't know exactly which queue (you can bind multiple queues to a container) you will have to allow only single queue per container which in my opinion is what you should do regardless.
In my companies own code base we have a magical SimpleMessageListenerContainerFactory that does the creation of custom SimpleMessageListenerContainer based on routing annotations (think spring mvc #RequestMapping for amqp). If there is interest perhaps we can expedite opensourcing it.
I have exactly 2 types of messages that I want to be sent via RabbitMQ. So I have 2 options how I can do this:
sent a message to default empty-named exchange with routing_key corresponding to the queue name
use direct exchange's routing_key parameter corresponding to consumer's routing_key parameter in queue binding
So which option is preferable and why?
A default exchange is a direct exchange. RabbitMQ creates the default exchange by default, but it uses an empty string for the name. If you look at the RabbitMQ AMQP concepts page, under Default Exchange:
The default exchange is a direct exchange with no name (empty string)
pre-declared by the broker.
You can see this by running rabbitmqctl list_exchanges as well:
direct
Foo direct < Same thing as the above
amq.direct direct
amq.fanout fanout
...and so on
As far as I'm aware, there isn't any benefits of using one over the other. I would stick with the default exchange if you only need to route based on routing keys.
Let's say you direct-bind to an exchange broadcasting logs to routing keys of "info", "warn", and "error". Using the default exchange, you would need to create three different queues with those names to receive all logs; and adjustments to which log levels you receive would require changing your queue declarations. By using a named exchange, you can simply change your queue's bindings and continue processing things as normal.
In short, it provides one extra level of abstraction.
As I see it, the default direct exchange give the possibility for the consumers and the producers to not know about each other, by binding a queue (used by a consumer) to an exchange (used by a producer) implicitly using the queue's name.
I use the default direct-exchange for a specific case: the consumer and producers don't know about each other. In my case, each consumer have its proper queue. From the producer, I cannot know by advance which queues are going to be declared and used, as it depends on the consumers. So it is impossible to define the bindings between a custom direct-exchange and the queues on the producer side.
One way to solve it with a custom (user-defined) direct-exchange would be to define the binding-key on consumer side. But it would means to know about the producer from the consumer side as I need to know the exchange name used by the producer.
Therefore, automatically binding a queue by its name on the default direct-exchange makes it possible, in my case, to only declare a queue on consumer side and send message to it from producer by only knowing the queue's name.
Of course it implies to know the queue's name at runtime, when invoking the producer, as it would be required to know the binding-key of a custom direct-exchange (in my case, the queue's name is given by the application using the producer). But when configuring the broker, producers and consumers don't have to know about each other.
some description in rabbitmq web manager.
//default exchange 's Bindings
Default exchange
The default exchange is implicitly bound to every queue,
with a routing key equal to the queue name.
It is not possible to
explicitly bind to, or
unbind from the default exchange.
It also cannot be deleted.