In Cloud Stream for RabbitMQ I want to set a consumer that will consume from a Queue only the messages that have been forwarded with a specific routing key. Is this possible?
Here are my application.properties.
Producer:
spring.cloud.stream.rabbit.bindings..consumer.bindQueue=false
spring.cloud.stream.rabbit.bindings..consumer.declareExchange=false
spring.cloud.stream.bindings.producer1-out-0.destination=exchange1
spring.cloud.stream.rabbit.bindings.producer1-out-0.producer.routingKeyExpression='routing.key.1'
Consumer:
spring.cloud.stream.rabbit.bindings..consumer.bindQueue=false
spring.cloud.stream.rabbit.bindings..consumer.declareExchange=false
spring.cloud.stream.bindings.consumer1-in-0.destination=exchange1
spring.cloud.stream.bindings.consumer1-in-0.group=reports
spring.cloud.stream.rabbit.bindings.consumer1-in-0.consumer.queueNameGroupOnly=false
spring.cloud.stream.rabbit.bindings.consumer1-in-0.consumer.bindingRoutingKeyDelimiter=,
spring.cloud.stream.rabbit.bindings.consumer1-in-0.consumer.bindingRoutingKey='routing.key.1'
Exchanges and bindings are defined directly in RabbitMQ.
My expectation was if I change spring.cloud.stream.rabbit.bindings.consumer1-in-0.consumer.bindingRoutingKey to something else the consumer will stop consuming but it's not the case.
No; unlike JMS, RabbitMQ has no concept of a message selector; you will get all messages.
The canonical way to solve this issue is to use a different queue for each RK.
bindingRoutingKey is meaningless when bindQueue is false and, in any case, all it does is specify which routing key to use when binding the queue to the exchange.
Related
Using RabbitMQ 3.7.16, with spring-amqp 2.2.3.RELEASE.
Multiple clients publish messages to the DataExchange topic exchange in our RabbitMQ server, using a unique routing key. In the absence of any bindings, the exchange will route all the messaged to the data.queue.generic through the AE.
When a certain client (client ID 1 and 2 in the diagram) publishes lots of messages, in order to scale the consumption of their messages independently from other clients, we are starting consumers and assign them to only handle a their client ID. To achieve this, each client-consumer is defining a new queue, and it binds it to the topic exchange with the routing key events.<clientID>.
So scaling up is covered and works well.
Now when the messages rate for this client goes down, we would like to also scale down its consumers, up to the point of removing all of them. The intention is to then have all those messages being routed to the GenericExchange, where there's a pool of generic consumers taking care of them.
The problem is that if I delete data.queue.2 (in order to remove its binding which will lead to new messages being routed to the GenericExchange) all its pending messages will be lost.
Here's a simplified architecture view:
It would be an acceptable solution to let the messages expire with a TTL in the client queue, and then dead letter them to the generic exchange, but then I also need to stop the topic exchange from routing new messages to this "dying" queue.
So what options do I have to stop the topic exchange from routing messages to the client queue where now there's no consumer connected to it?
Or to explore another path - how to dead letter messages in a deleted/expired queue?
If the client queue is the only one with a matching binding as your explanation seems to suggest, you can just remove the binding between the exchange and the queue.
From then on, all new messages for the client will go through the alternate exchange, your "generic exchange", to be processed by your generic consumers.
As for the messages left over in the client queue, you could use a shovel to send them back to the topic exchange, for them to be routed to the generic exchange.
This based on the assumption the alternate exchange is internal. If it's not internal, you can target it directly with the shovel.
As discussed with Bogdan, another option to resolve this while ensuring no message loss is occuring is to perform multiple steps:
remove the binding between the specific queue and the exchange
have some logic to have the remaining messages be either consumed or rerouted to the generic queue
if the binding removal occurs prior to the consumer(s) disconnect, have the last consumer disconnect only once the queue is empty
if the binding removal occurs after the last consumer disconnect, then have a TTL on messages with alternate exchange as the generic exchange
depending on the options selected before, have some cleanup mecanism to remove the lingering empty queues
I am integrating several .Net modules using pub/sub messaging using RabbitMQ and MassTransit. Most of the message subscription shall be durable. But some shall be transient. When a consumer dies the messages shall not be stored and already queued messages shall be discarded.
In each module I create 1 bus with 2 receive endpoints. One is configured as durable and non-auto-delete. The other one is configured as non-durable and auto-delete. Each gets its own set of consumers. This works as expected.
Now I am trying to implement request/response messages. Here comes the problem because now the sender has to decide to which exchange to route to. And that is wrong as I want receiver to decide whether to use durable or transient queue.
My questions:
Is there a better way how to support durable and transient subscription at the same time?
Why is MassTransit binding message exchange to an endpoint exchange that is bound to an endpoint queue? Why cannot the message exchange be directly bound to the endpoint queue?
Lets assume that all request consumers in one module are either durable or transient. Is it possible to declare one "module"-exchange which is then bound to either durable or transient queue? So the sender addresses the module exchange and module decides to which queue to bind. How to convince MassTransit to do so?
A module is using durable subscriptions that survive through restarts of module and also broker. After some time, admin (so in run-time of the system) decides to disconnect this module from the system. Can the module somehow unsubscribe everything and let MassTransit to remove the durable exchanges and queues?
Your question starts with request/response sent to an unknown endpoint, and ends with removing exchanges. These are different things, I suppose.
I cannot answer point-by-point, just will try to clear up things.
Request/response by definition requires you to know where you send stuff. As per MassTransit convention, the endpoint address is always an exchange/queue pair address. Therefore, you cannot let receiver decide who will handle this message, it will be delivered to the exchange/queue of the endpoint where you send it to.
About the "unsubscribe" - MassTransit deletes nothing. You have to clean up the binding that is not being used manually or by using the management API.
I'm trying to setup RabbitMQ in a model where there is only one producer and one consumer, and where messages sent by the producer are delivered to the consumer only if the consumer is connected, but dropped if the consumer is not present.
Basically I want the queue to drop all the messages it receives when no consumer is connected to it.
An additional constraint is that the queue must be declared on the RabbitMQ server side, and must not be explicitly created by the consumer or the producer.
Is that possible?
I've looked at a few things, but I can't seem to make it work:
durable vs non-durable does not work, because it is only useful when the broker restarts. I need the same effect but on a connection.
setting auto_delete to true on the queue means that my client can never connect to this queue again.
x-message-ttl and max-length make it possible to lose message even when there is a consumer connected.
I've looked at topic exchanges, but as far as I can tell, these only affect the routing of messages between the exchange and the queue based on the message content, and can't take into account whether or not a queue has connected consumers.
The effect that I'm looking for would be something like auto_delete on disconnect, and auto_create on connect. Is there a mechanism in rabbitmq that lets me do that?
After a bit more research, I discovered that one of the assumptions in my question regarding x-message-ttl was wrong. I overlooked a single sentence from the RabbitMQ documentation:
Setting the TTL to 0 causes messages to be expired upon reaching a queue unless they can be delivered to a consumer immediately
https://www.rabbitmq.com/ttl.html
It turns out that the simplest solution is to set x-message-ttl to 0 on my queue.
You can not doing it directly, but there is a mechanism not dificult to implement.
You have to enable the Event Exchange Plugin. This is a exchange at which your server app can connect and will receive internal events of RabbitMQ. You would be interested in the consumer.created and consumer.deleted events.
When these events are received you can trigger an action (create or delete the queue you need). More information here: https://www.rabbitmq.com/event-exchange.html
Hope this helps.
If your consumer is allowed to dynamically bind / unbind a queue during start/stop on the broker it should be possible by that way (e.g. queue is pre setup and the consumer binds the queue during startup to an exchange it wants to receive messages from)
This is the scenario - There are multiple app servers. Browser can connect via websocket to any app server.
The app servers (consumers) are all listening on a particular queue. As soon as a web socket connection is received, the particular app server binds the queue with a routing key {userId} to a direct exchange.
I want a message sent to the direct exchange with the routing key {userId} to be received by only the particular app server where the binding has occured.
Is a direct exchange the right exchange to use in this case? Or should some other type of exchange be used?
I'm using spring-amqp to create dynamic bindings when a websocket comes in
// create the RabbitMq queue and bind to it
String routingKey = MessageConstants.getRoutingKeyForUserRecommendationQueue(user);
Binding userRecommendationBinding = BindingBuilder.bind(userRecommendationsQueue).
to(directExchange).with(routingKey);
amqpAdmin.declareBinding(userRecommendationBinding);
Send message to a particular consumer in a queue
this is not possible. any consumer connected to a queue has a chance of consuming any given message in the queue
I want a message sent to the direct exchange with the routing key {userId} to be received by only the particular app server where the binding has occured.
you can do this by creating exclusive / autoDelete queues for your consumer, with a binding that directs all messages for that consumer to that queue.
Is a direct exchange the right exchange to use in this case?
either a direct exchange or a topic exchange is fine. direct exchange is slightly easier to understand, but topic exchange is more flexible
Actually you go right way.
And yes: Direct Exchange with an appropriate binding should save you.
See more info in the RabbitMQ Tutorial: http://www.rabbitmq.com/tutorials/tutorial-four-java.html
Also take a look into Spring AMQP Samples on the matter: https://github.com/spring-projects/spring-amqp-samples/tree/master/rabbitmq-tutorials
UPDATE
Unfortunately that is not what is happening. The messages seem to go randomly to any consumer, and not just the consumer that created the binding.
M-m-m. That's possible, because we route only my the key, but after that the message is placed to the queue, which may have several consumers on different machines.
In this case yes: the dynamic binding doesn't help.
You should consider to create an unique new queue (auto-deleted is fine) and bind and listen exactly from that. The SimpleMessageListenerContainer supports addQueues() at runtime to start a new consumer for a new queue.
I think that should work for you.
You still shouldn't do anything on the producer side: the same exhchange and routingKey logic.
Is it possible to bind a single queue to many topics using RabbitMQ STOMP client?
Each time a client sending SUBSCRIBE frame server creates a new queue for it, it makes usage of "prefetch-count" useless for me, because it applies to each subscription individually.
I am just looking for any way to get messages with many topics in the single queue via RabbitMQ Web-STOMP. Any ideas?
See Documentation:User generated queue names for Topic and Exchange destinations
The header x-queue-name specified queue name should binding to same queue if there exist, but will exist multiple subscription on client.
The different between AMQP and STOMP concept not compatible in some ways.