Is is possible to configure a RabbitMQ exchange or a queue in such a way that at most one message with a given routing key is pending at any time? If new message arrives, the old one would be dropped and the new one enqueued.
If such option is not available, what would be the best way to implement this at the application level? I.e. when application receives a message how can it check if there any more pending messages?
You need to install Last Value Cache and enable it. Your exchange will be type "x-lvc", which inherits from the direct exchange type.
each time you connect to MQ, create a queue and bind to this exchange. It will deliver the most recent message to the queue. It is perfect for making sure you get only the most uptodate message. All other messages sent to this exchange are discarded unless there is a queue connected. So once connected you will continue to receive updates.
here are installation instructions:
https://github.com/simonmacmullen/rabbitmq-lvc-plugin
here is a similar question:
RabbitMQ messaging - initializing consumer
Related
I wonder if the following scenario is possible:
Create an exchange of type direct
Publish a message to that exchange with routing key rk1
After that:
Create a queue which accepts messages with routing key rk1
Consume message published to exchange
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
So basically I want to be able to produce messages when there are no consumers present. And consume them some time later.
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
Yes, this is correct, but it's only part of the story.
A message queue is the storage location for messages published to the server. A consumer is a designated connection set to receive messages put into a queue. The exchange is simply a location to push messages. It contains the routing semantics to determine which messages wind up in the queues on the server. When a message cannot be routed to a queue and/or consumer, there are various semantics that can apply, but the default is that the message is dropped.
Options for dealing with unroutable messages:
Alternate exchange - designates a different exchange where messages can be dumped if they cannot be routed to a queue on the current exchange. This can be thought of similar to how TCP/IP works when a destination host is not reachable on the current subnet, and traffic is forwarded to the gateway. Note that a queue must be bound to the alternate exchange for the message to be dumped into. A typical case might be to have it configured as fanout exchange with one queue to trap all messages sent into the alternate exchange.
Mandatory or Immediate - return a message back to the sender if it can't be delivered. The server does not store the message.
Mandatory designates that the message must be deliverable to a queue at the time it is published. If the message is not routable, the publisher will receive a basic.return.
Immediate designates that, in addition to being deliverable, must be immediately routed to a consumer on a particular queue (e.g. it's not good enough that it be dumped in a queue for pickup later - it has to be delivered to the end consumer right now.
In every case, if there is no queue, the server cannot store the message.
The entity queue is the one that is supposed to hold the messages , so without a queue the messages will be lost.
However in case you do not create any exchange with appropriate routing key you may leverage dead lettering feature in rabbitmq.
Another solution could be to declare the queue with the binding after the exchange and before publishing the message; this way the message will be routed and stored, but you may have to add some TTLs ( https://www.rabbitmq.com/ttl.html ).
Does anyone know what is the easiest way to check if a consumer is active for a particular queue? I could hit the api on localhost manually, but I'd love a way to do this with Pika.
For example, I have a queue xyz I want to know how many consumers are listening to that queue.
You can do a queue.declare with passive=true for the same queue you want to get the consumer count from. This will return a response (queue.declare-ok) which includes the consumer count.
From the protocol reference documentation:
bit passive
If set, the server will reply with Declare-Ok if the queue already
exists with the same name, and raise an error if not. The client can
use this to check whether a queue exists without modifying the server
state.
I'm trying to setup RabbitMQ in a model where there is only one producer and one consumer, and where messages sent by the producer are delivered to the consumer only if the consumer is connected, but dropped if the consumer is not present.
Basically I want the queue to drop all the messages it receives when no consumer is connected to it.
An additional constraint is that the queue must be declared on the RabbitMQ server side, and must not be explicitly created by the consumer or the producer.
Is that possible?
I've looked at a few things, but I can't seem to make it work:
durable vs non-durable does not work, because it is only useful when the broker restarts. I need the same effect but on a connection.
setting auto_delete to true on the queue means that my client can never connect to this queue again.
x-message-ttl and max-length make it possible to lose message even when there is a consumer connected.
I've looked at topic exchanges, but as far as I can tell, these only affect the routing of messages between the exchange and the queue based on the message content, and can't take into account whether or not a queue has connected consumers.
The effect that I'm looking for would be something like auto_delete on disconnect, and auto_create on connect. Is there a mechanism in rabbitmq that lets me do that?
After a bit more research, I discovered that one of the assumptions in my question regarding x-message-ttl was wrong. I overlooked a single sentence from the RabbitMQ documentation:
Setting the TTL to 0 causes messages to be expired upon reaching a queue unless they can be delivered to a consumer immediately
https://www.rabbitmq.com/ttl.html
It turns out that the simplest solution is to set x-message-ttl to 0 on my queue.
You can not doing it directly, but there is a mechanism not dificult to implement.
You have to enable the Event Exchange Plugin. This is a exchange at which your server app can connect and will receive internal events of RabbitMQ. You would be interested in the consumer.created and consumer.deleted events.
When these events are received you can trigger an action (create or delete the queue you need). More information here: https://www.rabbitmq.com/event-exchange.html
Hope this helps.
If your consumer is allowed to dynamically bind / unbind a queue during start/stop on the broker it should be possible by that way (e.g. queue is pre setup and the consumer binds the queue during startup to an exchange it wants to receive messages from)
I have the following problem.
My program sends messages directly to the Queue (without exchange). I need to monitor incoming of new messages and send them to other Queue without removing them from source queue.
I don't have access to program code, so I'm not able to publish messages to exchange first.
Is it possible to solve this problem using the management web interface of RabbitMQ?
I tried to use shovel plugin, but it removes all messages from source queue after ack.
First to clear up few things:
My program sends messages directly to the Queue (without exchange) This is not true, at the very least (and most likely in this case) nameless exchange is used.
removes all messages from source queue after ack
this is by design and therefore perfectly fine.
You should never keep messages in the queue, queue is made to be consumed. As Derick Bailey says here
RabbitMQ is not a database. RabbitMQ is a message broker and queueing system.
on the same link you will find your answer. I cannot give a concrete one since you didn't provide motivation, but whatever it is keeping messages in the queue is never good!
Maybe you want to log/store your message first and then process it with the consequence of processing being some 3rd action or whatever...
This is the scenario - There are multiple app servers. Browser can connect via websocket to any app server.
The app servers (consumers) are all listening on a particular queue. As soon as a web socket connection is received, the particular app server binds the queue with a routing key {userId} to a direct exchange.
I want a message sent to the direct exchange with the routing key {userId} to be received by only the particular app server where the binding has occured.
Is a direct exchange the right exchange to use in this case? Or should some other type of exchange be used?
I'm using spring-amqp to create dynamic bindings when a websocket comes in
// create the RabbitMq queue and bind to it
String routingKey = MessageConstants.getRoutingKeyForUserRecommendationQueue(user);
Binding userRecommendationBinding = BindingBuilder.bind(userRecommendationsQueue).
to(directExchange).with(routingKey);
amqpAdmin.declareBinding(userRecommendationBinding);
Send message to a particular consumer in a queue
this is not possible. any consumer connected to a queue has a chance of consuming any given message in the queue
I want a message sent to the direct exchange with the routing key {userId} to be received by only the particular app server where the binding has occured.
you can do this by creating exclusive / autoDelete queues for your consumer, with a binding that directs all messages for that consumer to that queue.
Is a direct exchange the right exchange to use in this case?
either a direct exchange or a topic exchange is fine. direct exchange is slightly easier to understand, but topic exchange is more flexible
Actually you go right way.
And yes: Direct Exchange with an appropriate binding should save you.
See more info in the RabbitMQ Tutorial: http://www.rabbitmq.com/tutorials/tutorial-four-java.html
Also take a look into Spring AMQP Samples on the matter: https://github.com/spring-projects/spring-amqp-samples/tree/master/rabbitmq-tutorials
UPDATE
Unfortunately that is not what is happening. The messages seem to go randomly to any consumer, and not just the consumer that created the binding.
M-m-m. That's possible, because we route only my the key, but after that the message is placed to the queue, which may have several consumers on different machines.
In this case yes: the dynamic binding doesn't help.
You should consider to create an unique new queue (auto-deleted is fine) and bind and listen exactly from that. The SimpleMessageListenerContainer supports addQueues() at runtime to start a new consumer for a new queue.
I think that should work for you.
You still shouldn't do anything on the producer side: the same exhchange and routingKey logic.