We are sending amqp messages to rabbitMQ and are setting the message-ttl property.
If messages got expired, they are moved to the defined DLQ.
Is it possible to have expired messages moved to a seperate DLQ so that they do not interfere with other messages moved to DLQ because of more serious reasons?
Yes, this is possible.
You need to set a Dead Letter Exchange on your queue, and configure the message routing key to change when the messages get expired. Use the x-dead-letter-routing-key arg for this.
Then bind a new queue to your DLX with the dead letter routing key you just defined.
Expired messages will then be sent by RabbitMQ to the DLX, which will route them to the queue you have explicitly defined only for expired messages.
More about this here: https://www.rabbitmq.com/dlx.html.
Related
Using Active MQ 5.15.4 and following the doc from http://activemq.apache.org/virtual-destinations.html, when sending to a VirtualTopic, the messages get sent to all connected queues, but they never get dequeued from the virtual topic where they were sent.
Do we need to manually clean the virtual topic?
What is the reason of having the messages kept in the topic? Is it that they can be re-sent later on? But when a new queue gets linked to the virtual topic, all existing enqueued messages are not sent to it.
Have not tested this, but are the messages in the connected queues respecting the persistence flag for the message sent in the virtual topic?
If there is no consumer on the Virtual Topic itself then the only messages retained are the one's placed on the subscription queues for the Virtual Topic consumers. For example if you send to VirtualTopic.FOO and there are no subscriptions on that Topic or the named Virtual Topic consumer queues such as Consumer.A.VirtualTopic.FOO then the message would be completely discarded. If there was some consumer on the consumer queue at some point then messages sent to the Topic are then forwarded to the Queue but the Topic itself retains nothing.
If there are consumers on the Virtual Topic itself they would get messages sent to them or held for them up to the configured pending message limit etc etc.
The Consumer Queues will respect the persistent value specified by the MessageProducer that sent them.
We have observed the following behavior of RabbitMQ and are trying to understand if it is correct and how to resolve it.
Scenario:
A (persistent) message is delivered into a durable queue
The (single) Consumer (Spring-AMQP) takes the message and starts processing => Message goes from READY to UNACK
Now the broker is shut down => Client correctly reports "Channel shutdown"
The consumer finishes the processing, but can not acknowledge the message as the broker is still down
Broker is started again => Client reconnects
As a result, one message remains unack'ed forever (or until the client is restarted).
Side note: In the Rabbit Admin UI, I can see that two channels are existing now. The "dead" one that was created before the broker restart, containing the unacked message and a new one that is healthy.
Is this behavior expected to be like that? It seems to me "correct" in the way, that RabbitMQ can not know after the broker restart, whether the message processing was completed or not. But what solution would exist than to get that unacked message back into the queue and to heal the system without restarting the consumer process?
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
Is this behavior expected to be like that? It seems to me "correct" in the way, that RabbitMQ can not know after the broker restart, whether the message processing was completed or not.
Yes, you are observing expected behavior. RabbitMQ will re-enqueue the message once it determines that the consumer is really dead. Since your consumer re-connects with what must be the same consumer tag as before, it is up to that process to ack or nack the message.
I am using rabbitmq client (https://github.com/rabbitmq/rabbitmq-objc-client). I wanted to notify whether or not a message is sent to the rabbitmq server, but I couldn't find anything related to whether or not the message is really sent.
Could someone tell me how to know if I publish a message to a queue and the message really arrives to the rabbitmq server?. Thanks in advance !
Kinh
Publisher acknowledgments are on the amqp level, handled by RMQ itself. In the "API" level you may get a exception or a return value or some indication depends on the library.
Quote from the aforementioned link:
For unroutable messages, the broker will issue a confirm once the
exchange verifies a message won't route to any queue (returns an empty
list of queues). If the message is also published as mandatory, the
basic.return is sent to the client before basic.ack. The same is true
for negative acknowledgements (basic.nack).
For routable messages, the basic.ack is sent when a message has been
accepted by all the queues. For persistent messages routed to durable
queues, this means persisting to disk. For mirrored queues, this means
that all mirrors have accepted the message.
Using web-stomp with RabbitMQ and web-socket (SockJS not used), after sending a message to a queue, how can the consumer be notified by the broker or monitor that the sent message has been consumed?
I've experimented with subscribing to the queue which makes the client a consumer and the goal is not to receive the message for processing, but to know when a consumer elsewhere has picked up, acknowledged the message and is no longer in the queue.
In retrospect, I really do believe I'm breaking the spirit of "send it and forget it" with this question's approach.
The better approach will be to subscribe to another queue that will receive the "finished processing" message that will be sent from the processor. The client can take the appropriate actions from there.
I've got some trouble understanding the confirm of RabbitMQ, I see the following explanation from RabbitMQ:
Notes
The broker loses persistent messages if it crashes before said
messages are written to disk. Under certain conditions, this causes
the broker to behave in surprising ways. For instance, consider this
scenario:
a client publishes a persistent message to a durable queue
a client consumes the message from the queue (noting that the message is persistent and the queue durable), but doesn't yet ack it,
the broker dies and is restarted, and
the client reconnects and starts consuming messages.
At this point, the client could reasonably assume that the message
will be delivered again. This is not the case: the restart has caused
the broker to lose the message. In order to guarantee persistence, a
client should use confirms. If the publisher's channel had been in
confirm mode, the publisher would not have received an ack for the
lost message (since the consumer hadn't ack'd it and it hadn't been
written to disk).
Then I am using this http://hg.rabbitmq.com/rabbitmq-java-client/file/default/test/src/com/rabbitmq/examples/ConfirmDontLoseMessages.java to do some basic test and verify the confirm, but get some weird results:
The waitForConfirmsOrDie method doesn't block the producer, which is different from my expectation, I suppose the waitForConfirmsOrDie will block the producer until all the messages have been ack'd or one of them is nack'd.
I remove the channel.confirmSelect() and channel.waitForConfirmsOrDie() from publisher, and change the consumer from auto ack to manual ack, I publish all messages to the queue and consume messages one by one, then I stop the rabbitmq server during the consuming process, what I expect now is the left messages will be lost after the rabbitmq server is restarted, because the channel is not in confirm mode, but I still see all other messages in the queue after the server restart.
Since I am new to RabbitMQ, can anyone tells me where is my problem of the confirm understanding?
My understanding is that "Channel Confirmation" is for Broker confirms it successfully got the message from producer, regardless of consumer ack this message or not. Depending on the queue type and message deliver mode, see http://www.rabbitmq.com/confirms.html for details,
the messages are confirmed when:
it decides a message will not be routed to queues
(if the mandatory flag is set then the basic.return is sent first) or
a transient message has reached all its queues (and mirrors) or
a persistent message has reached all its queues (and mirrors) and been persisted to disk (and fsynced) or
a persistent message has been consumed (and if necessary acknowledged) from all its queues
Old question but oh well..
I publish all messages to the queue and consume messages one by one, then I stop the rabbitmq server during the consuming process, what I expect now is the left messages will be lost after the rabbitmq server is restarted, because the channel is not in confirm mode, but I still see all other messages in the queue after the server restart.
This is actually how it should work, IF the persistence is enabled. If the server crashes or something else goes wrong, the messages cannot be confirmed, and thus, won't be removed from the queue.
Messages will only be removed from the queue if they are confirmed to be handled, or the broker didn't yet write it to memory or disk before the server crashed.
Confirming and acknowledging can be set off if wanted, and the producer won't be waiting for the acks. I cannot find the exact command for it right now, but it does exist.
More on the acks and confirms: https://www.rabbitmq.com/reliability.html