I'm new in RabbitMQ and I'm trying to build a robust producer that keep trying to deliver the message
So I need to make sure that the message sent from my producer and placed in the queue.
I just need to confirm that it's in the queue 'may not received consumer yet becuse of the size of queue'.
My simple app publisher
var factory = new ConnectionFactory
{
Uri = new Uri($"amqp://{_config.MassTransit.Username}:{_config.MassTransit.Password}#{_config.MassTransit.Host}:{_config.MassTransit.Port}")
};
using IConnection connection = factory.CreateConnection();
using IModel channel = connection.CreateModel();
channel.BasicPublish(message.ExchangeName, message.RoutingKey, null,
Encoding.UTF8.GetBytes(JsonSerializer.Serialize(message)));
So, how can I confirm message in queue in order to avoid redelivery the message?
Thanks in advance.
You are looking for publisher confirms.
Related
I'm trying to create integration tests for a MessageListener that is processing messages from a SQS queue. One of the test cases I would like to verify is that an improperly formatted message sent to the MessageListener ends up in the Dead Letter Queue (also an SQS queue) that has been setup in the AWS console.
I'm having difficulty figuring out how to test this though, as you cannot search the Dead Letter Queue queue for a specific message id (i.e. see if it contains a message) without sifting through all the messages. My initial thought was to poll the dead letter queue and verify that the message id exists in there but it seems like that will have some problems. The integration test will send a message to the actual queue in our testing environment, so there may be an unknown number of other messages already present in the dead letter queue (and the messages in the dead letter queue could be continuously increasing). I also tried using some of the properties associated with a JMS message, but it seems like they aren't updated in my local instance of the message when the message goes from the test -> SQS queue -> MessageListener. Any ideas on how to test this scenario?
Here's an example of how the MessageListener is setup:
public class ExampleMessageListener implements MessageListener {
#Override
public void onMessage(final Message message) {
// message handling logic
// if something goes wrong in message handling logic message won't be acknowledged
// and should end up in the DLQ
message.acknowledge()
}
}
How I'm sending the message to the queue the MessageListener is consuming in my integration test:
SQSConnectionFactory connectionFactory = new SQSConnectionFactory(
new ProviderConfiguration(),
AmazonSQSClientBuilder.standard()
.withRegion(config.getRegion().getName())
.withCredentials(config.getCredentialsProvider())
);
SQSConnection connection = connectionFactory.createConnection();
session = connection.createSession(false, SQSSession.UNORDERED_ACKNOWLEDGE);
final Queue queue = session.createQueue("SQS-queue-name");
this.messageProducer = session.createProducer(queue);
final TextMessage textMessage = session.createTextMessage("improper message");
messageProducer.send(textMessage);
One additional note: There's a database that the MessageListener is modifying, and I'm able to use that to check if the properly formatted messages are changing data as expected. Improperly formatted messages don't end up interacting with this database though.
Keep in mind that Amazon SQS doesn't support message selectors via their JMS client implementation.
I'm trying to understand how message acknowledgement works exactly under the AMQP protocol (specifically RabbitMQ) with a direct exchange with multiple consumers subscribed to the same routing key. It is essentially a fanout exchange, but I have it so it can fan out to different consumers based on the routing_key. My current mental model looks like this:
Publisher creates "reply_to" queue and publishes to routing key with a message telling consumers to send response to queue (RPC protocol), along with a correlation id which is passed back so that all future results are tied to that unique identifier
Exchange sends out message to all queues bound to that routing key. Here, there are two queues for two consumers, each bound to routing key "pumps"
After some time, the consumer replies back to the reply_to queue, and then acknowledges message so that THEIR EXCLUSIVE QUEUE deletes the message that was sent to its queue. Each consumer that received a message does this.
The broker sends the responses to the RPC queue. The publisher acknowledges each message it gets, acknowledging messages it receives for that
I know its confusing.. basically it comes down to this question - what is a message bound to? It is obvious in a round-robin scenario. Each message is being sent to one queue, and the consumer can ack it; however, if there are multiple consumers for the same message, it made sense to me that really, each queue (and each consumer bound to it) has its own message to the consumer, each of which must be acknowledged. Is that the case?
RabbitMQ has this to say:
https://www.rabbitmq.com/confirms.html#acknowledgement-modes
Depending on the acknowledgement mode used, RabbitMQ can consider a message to be successfully delivered either immediately after it is sent out (written to a TCP socket) or when an explicit ("manual") client acknowledgement is received.
Unfortunately, this mentions nothing about queues, and what happens when there are multiple of them with their own consumers.
With RabbitMQ, for true FanOut method, its best for each consumer to have their own queue bind to the exchange, with this each consumer will receive and consume the message without affecting other consumers
with a scenario like, Sale Rep sends orders to Cashiers, where there are multiple sale reps and multiple cashiers.
Sale rep sends order
Channel.ExchangeDeclare(exchange: "cashieradd", ExchangeType.Fanout);
var jsonResult = JsonConvert.SerializeObject(new CashierQueue()
{
transactionId = transactionId
});
var body = Encoding.UTF8.GetBytes(jsonResult);
Channel.BasicPublish(exchange: "cashieradd", routingKey: "", basicProperties: null, body: body);
Each cashier would subscribe to the exchange
{
var cashierAddQueue = Channel.QueueDeclare().QueueName;
Channel.QueueBind(queue: cashierAddQueue, exchange: "cashieradd", routingKey: "");
var consumer = new EventingBasicConsumer(Channel);
Channel.BasicConsume(queue: cashierAddQueue, autoAck: true, consumer: consumer);
return consumer;
}
this uses the RabbitMQ Dynamic Queue, however, any queue unique to the consumer would have the same effect.
A routingkey here is not necessarily required
I am trying to understand the logic for message deletion in RabbitMQ.
My goal is to make messages persist even if there is not a client connected to read them, so that when clients reconnect the messages are waiting for them. I can use durable, lazy queues so that messages are persisted to disk, and I can use HA replication to ensure that multiple nodes get a copy of all queued messages.
I want to have messages go to two or more queues, using topic or header routing, and have one or more clients reading each queue.
I have two queues, A and B, fed by a header exchange. Queue A gets all messages. Queue B gets only messages with the "archive" header. Queue A has 3 consumers reading. Queue B has 1 consumer. If the consumer of B dies, but the consumers of A continue acknowledging messages, will RabbitMQ delete the messages or continue to store them? Queue B will not have anyone consuming it until B is restarted, and I want the messages to remain available for later consumption.
I have read a bunch of documentation so far, but still have not found a clear answer to this.
RabbitMQ will decide when to delete the messages upon acknowledgement.
Let's say you have a message sender:
var factory = new ConnectionFactory() { HostName = "localhost", Port = 5672, UserName = "guest", Password = "guest" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.QueueDeclare(queue: "hello",
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
string message = "Hello World!";
var body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish(exchange: "",
routingKey: "hello",
basicProperties: null,
body: body);
Console.WriteLine(" [x] Sent {0}", message);
}
This will create a durable queue "hello" and send the message "Hello World!" to it. This is what the queue would look like after sending one message to it.
Now let's set up two consumers, one that acknowledges the message was received and one that doesn't.
channel.BasicConsume(queue: "hello",
autoAck: false,
consumer: consumer);
and
channel.BasicConsume(queue: "hello",
autoAck: true,
consumer: consumer);
If you only run the first consumer, the message will never be deleted from the queue, because the consumer states that the messages will only disappear from the queue if the client manually acknowledges them: https://www.rabbitmq.com/confirms.html
The second consumer however will tell the queue that it can safely delete all the messages it received, automatically/immediately.
If you don't want to automatically delete these messages, you must disable autoAck and do some manual acknowledgement using the documentation:
http://codingvision.net/tips-and-tricks/c-send-data-between-processes-w-memory-mapped-file (Scroll down to "Manual Acknowledgement").
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);
The simple answer is that messages consumed from one queue have no bearing on messages in another. Once you publish a message, the broker distributes copies to as many queues as appropriate - but they are true copies of the message and are absolutely unrelated from that point forward so far as the broker is concerned.
Messages enqueued into a durable queue remain until they are pulled by a consumer on the queue, and optionally acknowledged.
Note that there are specific queue-level and message-level TTL settings that could affect this. For example, if the queue has a TTL, and the consumer does not reconnect before it expires, the queue will evaporate along with all its messages. Similarly, if a message has been enqueued with a specific TTL (which can also be set as a default for all messages on a particular queue), then once that TTL passes, the message will not be delivered to the consumer.
Secondary Note In the case where a message expires on the queue due to TTL, it will actually remain on the queue until it is next up to be delivered.
There are different ways where RabbitMQ deletes the messages.
Some of them are:
After Ack from consumer
Time-to-live(TTL) for that Queue reached.
Time-to-live(TTL) for messages on that Queue reached.
The last two points state that RabbitMQ allows you to set TTL(Time-to-live) for both messages and queues.
TTL can be set for a given queue by setting the x-message-ttl argument to queue.declare, or by setting the message-ttl policy.
Expiry time can be set for a given queue by setting the x-expires argument to queue.declare, or by setting the expires policy.
A message that has been in the queue for longer than the configured TTL is said to be dead.
Important point to note here is that a single message routed to different Queues can die at different times or sometimes never in each queue where it resides.
The death of a message in one Queue has no impact on the life of same message in some other Queue
I´m sending a message to a topic exchange which hasn´t any bindings to any queues. Just a blank exchange.
The channel is created as confirm channel and my confirm callback is called each time I send a message.
The strange thing is that for each message I get ack.
Am I doing something wrong or missunderstand the way how publisher confirmation works?
How can I know if a message is routed to a queue or dropped by the exchange?
I´m using amqplib for node.
Ok probably I didn´t explain my issue clear enough. So here is some code:
var amqp = require('amqplib/callback_api');
amqp.connect('amqp://host' , function(err, conn) {
conn.createConfirmChannel(function(err, ch) {
channel.assertExchange('my_awsome_exchange', 'topic', {durable: true});
channel.publish('my_awsome_exchange', 'routing_key', new Buffer('some data'),
{
mandatory: true
},
function(err){
// err is null no matter if a queue is bound to the exchange or not
console.log(err);
});
});
});
As you can see an exchange is created but no queue is bound to this exchange yet. So my expectation was that sending messages to this exchange would never be acked.
#Teddy: I know this section from the docs and this is the reason why I´m so confused.
As the message isn´t routed to any queue I would have expected the message to be nacked.
It is by design. Check this link. It says clearly:
When will messages be confirmed?
For unroutable messages, the broker
will issue a confirm once the exchange verifies a message won't route
to any queue (returns an empty list of queues). If the message is also
published as mandatory, the basic.return is sent to the client before
basic.ack. The same is true for negative acknowledgements
(basic.nack).
For routable messages, the basic.ack is sent when a message has been
accepted by all the queues. For persistent messages routed to durable
queues, this means persisting to disk. For mirrored queues, this means
that all mirrors have accepted the message.
I believe the misunderstanding lies in the fact, that an "Ack" means that the server successfully routed the message.
But the previous statement is false. Ack actually means that the server was able to handle the message successfully.
You can think of it like the RabbitMQ server saying: I take responsibility for your message
"basic.nack will only be delivered if an internal error occurs in the Erlang process responsible for a queue."
Quoted by the following Link
https://www.rabbitmq.com/confirms.html
I have tried the channel.confirmSelect() and its doesnt garuntee that message sent to queue rather than it give guarantee that message published to exchange. If no queue is bound at that moment then RabbitMQ server will simply discard the message.
I am trying to connect from my Android app to one queue called "messages".
The producer (one webservices under AMQP protocol) is already connected, it can be check through RabbitMQ admin panel.
To connect from my Android device I am coding like this.
private void connect() throws Exception {
this.sampleClient = new MqttClient(this.broker, this.clientId);
MqttConnectOptions connOpts = new MqttConnectOptions();
connOpts.setUserName("user");
connOpts.setPassword("user".toCharArray());
/*connOpts.setConnectionTimeout(60 * 10);
connOpts.setKeepAliveInterval(60 * 5);*/
connOpts.setCleanSession(true);
this.sampleClient.connect(connOpts);
this.sampleClient.setCallback(this);
this.sampleClient.subscribe("messages");
if(!this.sampleClient.isConnected()){
System.out.println("Not Connected");
return;
}
System.out.println("Connected");
}
I have tried with "amq.topic", "amq.topic.*", "amq.topic.messages", etc... But when I look in the RabbitMQ queue section "messages" is with 0 consumers, and have been set one new queue called "mqtt-subscription-Sampleqos1" automatically.
What's happening? How can I susbscribe to "messages" queue?
There are two important points about this question.
According with the RabbitMQ MQTT documentation: http://www.rabbitmq.com/mqtt.html
Firstly, every queues are bound automatically to amq.topic exchange by the mqtt-plugin.
Secondly, every subscriber has his own queue which look like this, mqtt-subscription-{cliend_id}{qosX} (where X is the qos level of the subscription)
Therefore, producer must to publish the message to "amq.topic" exchange, and "amq.topic.." routing-key, and receiver must to subscribe to "amq.topic.." routing-key.
First, make sure MQTT plugin is enabled: rabbitmq-plugins enable rabbitmq_mqtt
From the client side (here is you Android app), you need subscriber to a topic, lets say, topic my/android/app/messages
this.sampleClient.subscribe("my/android/app/messages");
Then, from the server side, because of RabbitMQ's implementation, you need send the message to a special exchange 'amq.topic' with appropriate route key my.android.app.messages (notice the mapping between '/' and '.', MQTT use / and AMQP use .). For example if you publish by pika AMQP Python lib, the code will looks like following:
channel.basic_publish(
exchange='amq.topic',
routing_key='my.android.app.messages',
body='hello world'
)
In your case, you want to receive message from queue "messages", basically there is no way to directly subscriber message from that AMQP queue on your MQTT client. The work around is create a service running on your server side, work as AMQP subscriber, receive message from "messages" queue, and transparent forward message to exchange amq.topic with proper routing key.
Hope my answer helpful.