I'm trying to understand how message acknowledgement works exactly under the AMQP protocol (specifically RabbitMQ) with a direct exchange with multiple consumers subscribed to the same routing key. It is essentially a fanout exchange, but I have it so it can fan out to different consumers based on the routing_key. My current mental model looks like this:
Publisher creates "reply_to" queue and publishes to routing key with a message telling consumers to send response to queue (RPC protocol), along with a correlation id which is passed back so that all future results are tied to that unique identifier
Exchange sends out message to all queues bound to that routing key. Here, there are two queues for two consumers, each bound to routing key "pumps"
After some time, the consumer replies back to the reply_to queue, and then acknowledges message so that THEIR EXCLUSIVE QUEUE deletes the message that was sent to its queue. Each consumer that received a message does this.
The broker sends the responses to the RPC queue. The publisher acknowledges each message it gets, acknowledging messages it receives for that
I know its confusing.. basically it comes down to this question - what is a message bound to? It is obvious in a round-robin scenario. Each message is being sent to one queue, and the consumer can ack it; however, if there are multiple consumers for the same message, it made sense to me that really, each queue (and each consumer bound to it) has its own message to the consumer, each of which must be acknowledged. Is that the case?
RabbitMQ has this to say:
https://www.rabbitmq.com/confirms.html#acknowledgement-modes
Depending on the acknowledgement mode used, RabbitMQ can consider a message to be successfully delivered either immediately after it is sent out (written to a TCP socket) or when an explicit ("manual") client acknowledgement is received.
Unfortunately, this mentions nothing about queues, and what happens when there are multiple of them with their own consumers.
With RabbitMQ, for true FanOut method, its best for each consumer to have their own queue bind to the exchange, with this each consumer will receive and consume the message without affecting other consumers
with a scenario like, Sale Rep sends orders to Cashiers, where there are multiple sale reps and multiple cashiers.
Sale rep sends order
Channel.ExchangeDeclare(exchange: "cashieradd", ExchangeType.Fanout);
var jsonResult = JsonConvert.SerializeObject(new CashierQueue()
{
transactionId = transactionId
});
var body = Encoding.UTF8.GetBytes(jsonResult);
Channel.BasicPublish(exchange: "cashieradd", routingKey: "", basicProperties: null, body: body);
Each cashier would subscribe to the exchange
{
var cashierAddQueue = Channel.QueueDeclare().QueueName;
Channel.QueueBind(queue: cashierAddQueue, exchange: "cashieradd", routingKey: "");
var consumer = new EventingBasicConsumer(Channel);
Channel.BasicConsume(queue: cashierAddQueue, autoAck: true, consumer: consumer);
return consumer;
}
this uses the RabbitMQ Dynamic Queue, however, any queue unique to the consumer would have the same effect.
A routingkey here is not necessarily required
Related
I'm a bit confused about RabbitMQ best practices regarding the use of Queues and Exchanges. Let's say I would like to deliver a GenerateInvoice message with some data for an invoice, and have multiple consumers processing the invoice data and generate a PDF. Each GenerateInvoice should be processed only by one consumer.
One approach is to declare an queue and publish the GenerateInvoice messages to this queue and let all consumers consume from this queue. That would distribute the message across the different consumers.
It's unclear to me, if the above is okay or best practice is delivering the messages to a Exchange instead of publishing them directly to a queue. Using an Exchange I have to ensure that an queue is declared after the producer has created the Exchange before it starts to publish messages. Otherwise no queue would receive the messages and the message would be lost.
Declaring a queue, publishing GenerateInvoice messages to the queue and having multiple consumers for the queue would work in this scenario.
The messages published to the queue will not be lost and they stay on RMQ if there are no consumers. Only thing is to make sure queue is declared before messages are published.
Java Example:
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
Then, Publish can be done as:
channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
and consume can be done as:
channel.basicConsume(QUEUE_NAME, true, deliverCallback, consumerTag -> { });
I publish the message, it does not stay on queue. Once subscribe, the message queue starts to stay. I want to keep the message in the queue even if the user is not subscribed at all. I am using qos = 1.
It is important to remmber that MQTT is a Pub/Sub system not a Message Queuing system.
With MQTT messages will only be queued for offline clients that already have a subscription (at QOS 1 or 2), a new client subscribing to a topic will only receive new messages.
You can use the retained flag to ensure the last message (with a retained flag set) is always delivered to a client when it subscribes to a topic before new messages, but this is a single message.
I am trying to understand the logic for message deletion in RabbitMQ.
My goal is to make messages persist even if there is not a client connected to read them, so that when clients reconnect the messages are waiting for them. I can use durable, lazy queues so that messages are persisted to disk, and I can use HA replication to ensure that multiple nodes get a copy of all queued messages.
I want to have messages go to two or more queues, using topic or header routing, and have one or more clients reading each queue.
I have two queues, A and B, fed by a header exchange. Queue A gets all messages. Queue B gets only messages with the "archive" header. Queue A has 3 consumers reading. Queue B has 1 consumer. If the consumer of B dies, but the consumers of A continue acknowledging messages, will RabbitMQ delete the messages or continue to store them? Queue B will not have anyone consuming it until B is restarted, and I want the messages to remain available for later consumption.
I have read a bunch of documentation so far, but still have not found a clear answer to this.
RabbitMQ will decide when to delete the messages upon acknowledgement.
Let's say you have a message sender:
var factory = new ConnectionFactory() { HostName = "localhost", Port = 5672, UserName = "guest", Password = "guest" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "hello",
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
string message = "Hello World!";
var body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish(exchange: "",
routingKey: "hello",
basicProperties: null,
body: body);
Console.WriteLine(" [x] Sent {0}", message);
}
This will create a durable queue "hello" and send the message "Hello World!" to it. This is what the queue would look like after sending one message to it.
Now let's set up two consumers, one that acknowledges the message was received and one that doesn't.
channel.BasicConsume(queue: "hello",
autoAck: false,
consumer: consumer);
and
channel.BasicConsume(queue: "hello",
autoAck: true,
consumer: consumer);
If you only run the first consumer, the message will never be deleted from the queue, because the consumer states that the messages will only disappear from the queue if the client manually acknowledges them: https://www.rabbitmq.com/confirms.html
The second consumer however will tell the queue that it can safely delete all the messages it received, automatically/immediately.
If you don't want to automatically delete these messages, you must disable autoAck and do some manual acknowledgement using the documentation:
http://codingvision.net/tips-and-tricks/c-send-data-between-processes-w-memory-mapped-file (Scroll down to "Manual Acknowledgement").
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);
The simple answer is that messages consumed from one queue have no bearing on messages in another. Once you publish a message, the broker distributes copies to as many queues as appropriate - but they are true copies of the message and are absolutely unrelated from that point forward so far as the broker is concerned.
Messages enqueued into a durable queue remain until they are pulled by a consumer on the queue, and optionally acknowledged.
Note that there are specific queue-level and message-level TTL settings that could affect this. For example, if the queue has a TTL, and the consumer does not reconnect before it expires, the queue will evaporate along with all its messages. Similarly, if a message has been enqueued with a specific TTL (which can also be set as a default for all messages on a particular queue), then once that TTL passes, the message will not be delivered to the consumer.
Secondary Note In the case where a message expires on the queue due to TTL, it will actually remain on the queue until it is next up to be delivered.
There are different ways where RabbitMQ deletes the messages.
Some of them are:
After Ack from consumer
Time-to-live(TTL) for that Queue reached.
Time-to-live(TTL) for messages on that Queue reached.
The last two points state that RabbitMQ allows you to set TTL(Time-to-live) for both messages and queues.
TTL can be set for a given queue by setting the x-message-ttl argument to queue.declare, or by setting the message-ttl policy.
Expiry time can be set for a given queue by setting the x-expires argument to queue.declare, or by setting the expires policy.
A message that has been in the queue for longer than the configured TTL is said to be dead.
Important point to note here is that a single message routed to different Queues can die at different times or sometimes never in each queue where it resides.
The death of a message in one Queue has no impact on the life of same message in some other Queue
I have a producer and broker on the same machine. The producer sends messages like so:
channel = connection.createChannel();
//Create a durable queue (if not already present)
channel.queueDeclare(merchantId, true, false, false, null);
//Publish message onto the queue
channel.basicPublish("", consumerId, true, false,
MessageProperties.MINIMAL_PERSISTENT_BASIC, "myMessage");
The consumer sits on another machine and listens to messages. It uses explicit acknowledgement like so:
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
//Handle message here
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
From what I understand, the ack is meant for the broker to dequeue the message.
But how can my producer come to know about the ack that the consumer sent?
Producers and consumers normally don't interact. This is by AMQP protocol design. For example, consuming a specific message may be done a long time after it was published, and there is no sense in leaving the producer up and running for a long time. Another example is when a publisher sends one message to a broker, and due to routing logic that message gets duplicated to more than one queue, leading to ambiguity (because multiple consumers can acknowledge the same message). AMQP protocol is asynchronous (mostly), and letting the publisher know about its message being consumed just doesn't fit the AMQP async model.
There are exceptions from that, notably, RPC calls. Then the producer becomes a producer-consumer. It sends a message and then immediately waits for a reply (there is a good RabbitMQ manual - Direct reply-to related to RPC with RabbtiMQ).
In general, you can ensure that a message is delivered to a broker with Confirms (aka Publisher Acknowledgements) alongside with Dead Letter Exchanges and Alternate Exchanges. Those cover most cases under which a message can be lost from its normal flow.
Producer sends messages rabbitmq, and the consumer receives messages from rabbitmq, then consumer send messages back to producer via rabbitmq.
Is it possible?
Are you trying to implement an RPC type of system, or do you just want to know how to send messages in both directions?
The basic pipe is unidirectional. You cannot send messages from the consumer to the producer through the same queue that the consumer received messages from the producer.
If you want send messages the other way, your consumer will need to be a producer as well, and your producer will need to be a consumer as well.
Yes its possible, but the producer will have to listen to a queue as well on which the client will publish/produce... , you can send the queueName/routingkey(of the producer) in the first msg sent from producer to client.. the client then after recieving the msg with the routingkey inside can send msg to original producer
Producer needs another queue to get the response back from Consumer, so called callback queue. Producer can send request with reply_to and correlation_id. Then, Consumer can utilize reply_to as the routing key to point to the correct callback queue and Producer can dequeue the callback queue and match the correlation_id.