I would require a queue that does not accumulate in "Ready" state.
That is, if there is a subscriber to the queue, pass the messages from the exchange directly to it, and if there is no subscriber, just waste the messages, do not accumulate them in "Ready" state.
I know that an exchange of type fanout does this, but I want to be able to use the routing key and a fanout exchange ignores the routing keys. Moreove my scenario is not a broadcast one, it is 1 publisher and 1 (sometimes absent) subscriber.
Is it possible to have a direct exchange to which the queue does not accumulate messages in "ready" state if there is no subscriber?
ok I think I must declare the queue with auto-delete to true
and set the mandatory bit to true on the BasicPublish calls
it seems to work so far...
Related
RabbitMQ: version 3.11.2
I wish to configure a fanout exchange for which there will be two consumers and a single producer. Each of the two consumers can go offline for several minutes at a time; in the worst case scenario, one of the consumers could go offline for hours.
QUESTION: how should the fanout exchange and/or consumer queues be configured such that no message is ever lost, even if and when one of the consumers goes offline for several minutes or hours?
It is enough to bind queues to your exchange.
See here for more details https://www.rabbitmq.com/tutorials/tutorial-three-python.html
result = channel.queue_declare(queue='')
At this point result.method.queue contains a random queue name. For example it may look like amq.gen-JzTY20BRgKO-HjmUJj0wLg.
Secondly, once the consumer connection is closed, the queue should be deleted. There's an exclusive flag for that:
result = channel.queue_declare(queue='', exclusive=True)
channel.queue_bind(exchange='logs',
queue=result.method.queue)
Note:
If you need to handle the offline consumers, you need persistent queues and persistent messages instead of exclusive and auto-delete queues
Using RabbitMQ 3.7.16, with spring-amqp 2.2.3.RELEASE.
Multiple clients publish messages to the DataExchange topic exchange in our RabbitMQ server, using a unique routing key. In the absence of any bindings, the exchange will route all the messaged to the data.queue.generic through the AE.
When a certain client (client ID 1 and 2 in the diagram) publishes lots of messages, in order to scale the consumption of their messages independently from other clients, we are starting consumers and assign them to only handle a their client ID. To achieve this, each client-consumer is defining a new queue, and it binds it to the topic exchange with the routing key events.<clientID>.
So scaling up is covered and works well.
Now when the messages rate for this client goes down, we would like to also scale down its consumers, up to the point of removing all of them. The intention is to then have all those messages being routed to the GenericExchange, where there's a pool of generic consumers taking care of them.
The problem is that if I delete data.queue.2 (in order to remove its binding which will lead to new messages being routed to the GenericExchange) all its pending messages will be lost.
Here's a simplified architecture view:
It would be an acceptable solution to let the messages expire with a TTL in the client queue, and then dead letter them to the generic exchange, but then I also need to stop the topic exchange from routing new messages to this "dying" queue.
So what options do I have to stop the topic exchange from routing messages to the client queue where now there's no consumer connected to it?
Or to explore another path - how to dead letter messages in a deleted/expired queue?
If the client queue is the only one with a matching binding as your explanation seems to suggest, you can just remove the binding between the exchange and the queue.
From then on, all new messages for the client will go through the alternate exchange, your "generic exchange", to be processed by your generic consumers.
As for the messages left over in the client queue, you could use a shovel to send them back to the topic exchange, for them to be routed to the generic exchange.
This based on the assumption the alternate exchange is internal. If it's not internal, you can target it directly with the shovel.
As discussed with Bogdan, another option to resolve this while ensuring no message loss is occuring is to perform multiple steps:
remove the binding between the specific queue and the exchange
have some logic to have the remaining messages be either consumed or rerouted to the generic queue
if the binding removal occurs prior to the consumer(s) disconnect, have the last consumer disconnect only once the queue is empty
if the binding removal occurs after the last consumer disconnect, then have a TTL on messages with alternate exchange as the generic exchange
depending on the options selected before, have some cleanup mecanism to remove the lingering empty queues
I have an a queue that has x-expires set. The issue I am having is that I need to do further processing on the messages that are in the queue IF the queue expires. My initial idea was to set x-dead-letter-exchange on the queue. But, when the queue expires, the messages just vanish without making it to the dead-letter exchange.
How can I dead-letter, or otherwise process, messages that are in a queue that expires?
As suggested in the comments, you cannot do this by relying only on the x-expire feature. But a solution that worked in a similar case I had was to:
Use x-message-ttl to make sure messages die if not consumed in a timely manner,
Assign a dead letter exchange to the queue where all those messages will be routed,
Use x-expires to set the queue expiration to a value higher than the TTL of the messages,
(and this is the tricky part) Assuming you have control over your consumers, before the last consumer goes offline, delete the binding to your "dying" queue, potentially through a REST API call - this will prevent new messages from being routed to the queue.
This way the messages that were published before the last consumer died were already processed, existing messages will be dead-lettered before the queue expires, and new messages cannot come into the queue.
You need to add a new dead letter queue that is bound to your dead letter exchange with the binding routing key set as the original queue name. In this way all expired messages sent to the dead letter exchange are routed to the dead letter queue.
I'm trying to setup RabbitMQ in a model where there is only one producer and one consumer, and where messages sent by the producer are delivered to the consumer only if the consumer is connected, but dropped if the consumer is not present.
Basically I want the queue to drop all the messages it receives when no consumer is connected to it.
An additional constraint is that the queue must be declared on the RabbitMQ server side, and must not be explicitly created by the consumer or the producer.
Is that possible?
I've looked at a few things, but I can't seem to make it work:
durable vs non-durable does not work, because it is only useful when the broker restarts. I need the same effect but on a connection.
setting auto_delete to true on the queue means that my client can never connect to this queue again.
x-message-ttl and max-length make it possible to lose message even when there is a consumer connected.
I've looked at topic exchanges, but as far as I can tell, these only affect the routing of messages between the exchange and the queue based on the message content, and can't take into account whether or not a queue has connected consumers.
The effect that I'm looking for would be something like auto_delete on disconnect, and auto_create on connect. Is there a mechanism in rabbitmq that lets me do that?
After a bit more research, I discovered that one of the assumptions in my question regarding x-message-ttl was wrong. I overlooked a single sentence from the RabbitMQ documentation:
Setting the TTL to 0 causes messages to be expired upon reaching a queue unless they can be delivered to a consumer immediately
https://www.rabbitmq.com/ttl.html
It turns out that the simplest solution is to set x-message-ttl to 0 on my queue.
You can not doing it directly, but there is a mechanism not dificult to implement.
You have to enable the Event Exchange Plugin. This is a exchange at which your server app can connect and will receive internal events of RabbitMQ. You would be interested in the consumer.created and consumer.deleted events.
When these events are received you can trigger an action (create or delete the queue you need). More information here: https://www.rabbitmq.com/event-exchange.html
Hope this helps.
If your consumer is allowed to dynamically bind / unbind a queue during start/stop on the broker it should be possible by that way (e.g. queue is pre setup and the consumer binds the queue during startup to an exchange it wants to receive messages from)
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html