I publish many messages to my queue in RabbitMQ when worker is off.
Then i start my worker and he takes all messages from queue and he is working fine but when i stop my worker proces before he ends, all messages are deleted from queue.
If i start second the same worker he has no messages to take.
$channel->queue_declare($action, false, false, false, false);
$channel->basic_qos(null, 1, null);
$channel->basic_consume($action, '', false, true, false, false, $callback);
How can i do to my worker takes only one message on start from queue and when worker stops and starts he continue takes messages from queue where he stops ?
If there are no messages left to take the messages have already been acknowledged.
"Auto acknowledgements" should be disabled so that an acknowledgement must be explicitly sent back manually (ie. after all work is done) to complete consuming the message.
$no_ack = false; # If no_ack is true the server does NOT expect a
# basic.ack response and will dequeue the message immediately!
$channel->basic_consume($action, '', false, $no_ack, false, false, $callback);
The PHP tutorial on the RabbitMQ site shows how to send an ack from within the callback.
As long as the worker is killed before the ack has been sent the message will be made available again when the connection is closed - and can be picked up by other consumers. See Message Acknowledgements.
Specifically, wrt the no-ack bit:
If this field is set (true) the server does not expect acknowledgements for messages. That is, when a message is delivered to the client the server assumes the delivery will succeed and immediately dequeues it. This functionality may increase performance but at the cost of reliability. Messages can get lost if a client dies before they are delivered to the application.
Related
Registered a async consumer of rabbitmq message.
Didn't get official suggestions how to handle the process exception in async consumer action?
Maybe need retry the queue message /republish the message to the queue with a retry times limitation.
When consuming from a queue in rabbitMq you can set an option called noAck that can be true or false.
true it will ack a message in the event of an error it cannot handle
false will automatically nack the message and will stay in the
queue to be pulled later.
(This will depend on the language you are using for your consumer. noAck = nodejs, autoAck = c#, etc.)
consumer.consume(q.queue, function (message) {
// your code
}, {noAck: false});
In regards to setting limited retries, I had to do this myself by passing the retry count in the header of the message I was passing and had to ack the message I was reading before sending the new version with the modified header back to the queue. I used multiple queues in order to maintain message integrity but this could be done with one queue.
I hope this has helped.
I am trying to understand the logic for message deletion in RabbitMQ.
My goal is to make messages persist even if there is not a client connected to read them, so that when clients reconnect the messages are waiting for them. I can use durable, lazy queues so that messages are persisted to disk, and I can use HA replication to ensure that multiple nodes get a copy of all queued messages.
I want to have messages go to two or more queues, using topic or header routing, and have one or more clients reading each queue.
I have two queues, A and B, fed by a header exchange. Queue A gets all messages. Queue B gets only messages with the "archive" header. Queue A has 3 consumers reading. Queue B has 1 consumer. If the consumer of B dies, but the consumers of A continue acknowledging messages, will RabbitMQ delete the messages or continue to store them? Queue B will not have anyone consuming it until B is restarted, and I want the messages to remain available for later consumption.
I have read a bunch of documentation so far, but still have not found a clear answer to this.
RabbitMQ will decide when to delete the messages upon acknowledgement.
Let's say you have a message sender:
var factory = new ConnectionFactory() { HostName = "localhost", Port = 5672, UserName = "guest", Password = "guest" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "hello",
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
string message = "Hello World!";
var body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish(exchange: "",
routingKey: "hello",
basicProperties: null,
body: body);
Console.WriteLine(" [x] Sent {0}", message);
}
This will create a durable queue "hello" and send the message "Hello World!" to it. This is what the queue would look like after sending one message to it.
Now let's set up two consumers, one that acknowledges the message was received and one that doesn't.
channel.BasicConsume(queue: "hello",
autoAck: false,
consumer: consumer);
and
channel.BasicConsume(queue: "hello",
autoAck: true,
consumer: consumer);
If you only run the first consumer, the message will never be deleted from the queue, because the consumer states that the messages will only disappear from the queue if the client manually acknowledges them: https://www.rabbitmq.com/confirms.html
The second consumer however will tell the queue that it can safely delete all the messages it received, automatically/immediately.
If you don't want to automatically delete these messages, you must disable autoAck and do some manual acknowledgement using the documentation:
http://codingvision.net/tips-and-tricks/c-send-data-between-processes-w-memory-mapped-file (Scroll down to "Manual Acknowledgement").
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);
The simple answer is that messages consumed from one queue have no bearing on messages in another. Once you publish a message, the broker distributes copies to as many queues as appropriate - but they are true copies of the message and are absolutely unrelated from that point forward so far as the broker is concerned.
Messages enqueued into a durable queue remain until they are pulled by a consumer on the queue, and optionally acknowledged.
Note that there are specific queue-level and message-level TTL settings that could affect this. For example, if the queue has a TTL, and the consumer does not reconnect before it expires, the queue will evaporate along with all its messages. Similarly, if a message has been enqueued with a specific TTL (which can also be set as a default for all messages on a particular queue), then once that TTL passes, the message will not be delivered to the consumer.
Secondary Note In the case where a message expires on the queue due to TTL, it will actually remain on the queue until it is next up to be delivered.
There are different ways where RabbitMQ deletes the messages.
Some of them are:
After Ack from consumer
Time-to-live(TTL) for that Queue reached.
Time-to-live(TTL) for messages on that Queue reached.
The last two points state that RabbitMQ allows you to set TTL(Time-to-live) for both messages and queues.
TTL can be set for a given queue by setting the x-message-ttl argument to queue.declare, or by setting the message-ttl policy.
Expiry time can be set for a given queue by setting the x-expires argument to queue.declare, or by setting the expires policy.
A message that has been in the queue for longer than the configured TTL is said to be dead.
Important point to note here is that a single message routed to different Queues can die at different times or sometimes never in each queue where it resides.
The death of a message in one Queue has no impact on the life of same message in some other Queue
The newly released ActiveMQ (5.12) now supports pausing and resuming message delivery to the consumers for queues which is very useful.
The problem is that, when you resume the message delivery,
If there is a message entering the queue: the broker will immediately send the pending messages to the consumer which is totally OK.
But if no message enters the queue: the pending messages in the queue will not be sent to the consumers until the expiration checking is performed on the queue (which by default is 30 seconds and can be controlled by the expireMessagesPeriod attribute) and non-expired messages will be sent to the consumers afterwards.
Obviously we can change the expireMessagesPeriod to limit this delay, but when you need a milisec precision, performing the expiration check every milisec will not make sense.
Is there anyway to force the queue to start sending messages immediately after resumption?
Bug fixed by ActiveMQ team (details here) and will appear in the 5.12.2 release.
I have a producer and broker on the same machine. The producer sends messages like so:
channel = connection.createChannel();
//Create a durable queue (if not already present)
channel.queueDeclare(merchantId, true, false, false, null);
//Publish message onto the queue
channel.basicPublish("", consumerId, true, false,
MessageProperties.MINIMAL_PERSISTENT_BASIC, "myMessage");
The consumer sits on another machine and listens to messages. It uses explicit acknowledgement like so:
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
//Handle message here
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
From what I understand, the ack is meant for the broker to dequeue the message.
But how can my producer come to know about the ack that the consumer sent?
Producers and consumers normally don't interact. This is by AMQP protocol design. For example, consuming a specific message may be done a long time after it was published, and there is no sense in leaving the producer up and running for a long time. Another example is when a publisher sends one message to a broker, and due to routing logic that message gets duplicated to more than one queue, leading to ambiguity (because multiple consumers can acknowledge the same message). AMQP protocol is asynchronous (mostly), and letting the publisher know about its message being consumed just doesn't fit the AMQP async model.
There are exceptions from that, notably, RPC calls. Then the producer becomes a producer-consumer. It sends a message and then immediately waits for a reply (there is a good RabbitMQ manual - Direct reply-to related to RPC with RabbtiMQ).
In general, you can ensure that a message is delivered to a broker with Confirms (aka Publisher Acknowledgements) alongside with Dead Letter Exchanges and Alternate Exchanges. Those cover most cases under which a message can be lost from its normal flow.
I am using RabbitMq really amazing message broker, but what i need is to have long message back up time in the queue. So that if a message is not reached to certain consumers who were inactive at that time and they are active after few mins eventually the message should be delivered to the respective consumer through fanout exchange.
What i tried is i had increased TTL in queue. Whether it is the correct way or some other tricks can be done?
Thanks in advance
You could try to use a queue with x-message-ttl, and x-dead-letter-exchange
args.put("x-message-ttl", 10000);
args.put("x-dead-letter-exchange",exchange_dead_letter);
channel.queueDeclare(queue, false, false, false, args);
x-dead-letter-exchange is an exchange and if the message is expired by the TTL time, the message is redirect to the x-dead-letter-exchange.
Then you can handle the message as you prefer.
You can find more detail here: http://www.rabbitmq.com/dlx.html.