Being a bit confused about these two terms, I'm thinking what is the purpose of having a persistent message but transient (non-durable) queue?
After all, if the broker restarts and the queues are not restored the recovered messages will be wasted.
You can have durable queue but "mortal" messages, so after broker restarts you can still have queue but it will be empty and vice versa, but as you sad, yes, you'll lose all messages in the queue.
In the combination you provided message persistence option is really useless but will cause no error.
But if you bind alternate exchange to exchange you are publishing messages to and it is durable, after restart, your can route messages to it if you don't have transient queue declared.
Example:
Assume we have such combination and properly bound queues, Q*1 receive messages M*1 and Q*2 - M*2.
[ Exchange-main/durable ] + [Exchange-alternate/durable]
[Qm1/transient][Qm2/transient] [Qax1/durable][Qax2/durable]
Let's publish messages [Mt1/transient] and `[Md1/durable], we'll get such situation:
[ Exchange-main/durable ] + [Exchange-alternate/durable]
[Qm1/transient][Qm2/transient] [Qax1/durable][Qax2/durable]
[Mt1/transient]
[Md1/durable]
After restart we'll get
[ Exchange-main/durable ] + [Exchange-alternate/durable]
[Qax1/durable][Qax2/durable]
Let's publish two messages again, [Mt1/transient] and `[Md1/durable]:
[ Exchange-main/durable ] + [Exchange-alternate/durable]
[Qax1/durable][Qax2/durable]
[Mt1/transient]
[Md1/durable]
So, restart broker again:
[ Exchange-main/durable ] + [Exchange-alternate/durable]
[Qax1/durable][Qax2/durable]
[Md1/durable]
Related
I'm working with SQS in my application. I have the following configuration.
justSaying
.WithSqsTopicSubscriber()
.IntoQueue(_busNamingConvention.QueueName())
.ConfigureSubscriptionWith(x =>
{
x.VisibilityTimeoutSeconds = 60;
x.RetryCountBeforeSendingToErrorQueue = 3;
})
.WithMessageHandler<MyMessage>(_handlerResolver)
.WithSqsMessagePublisher<MyMessage>(config => config.QueueName = _busNamingConvention.QueueName());
So, there will be 3 re-attempts before the messages gets to Dead Letter Queue. I want to consume this dead letter queue and process the message separately. In essence, I want to create a handler to deal with the messages in the DLQ.
I'm not sure if this is possible or SQS is not intended to be used this way. Please post if this is possible and if yes, is it okay to do this or is this an anti pattern.
I have a list of jobs in my application queue(RabbitMQ).
Some of these jobs are group together and must do in the order.(not continuous, but by order of dispatch time)
For example, consider this 4 jobs in the queue:
[
{ "group": "x", "dispatched_timestamp": 10001, "field1": "some data", "field2": "some other data"},
{ "group": "g", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"},
{ "group": "x", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"},
{ "group": "t", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"}
]
I must sure the first job in group "x" execute successfully before the thirth job(same group).
But i don't care if the fourth job execute sooner than the first(or whatever).
Because sometimes it may happen which all three job deliver to 3 consumer, but the first job fail for some reason(but the second and thirth job has done successful).
I know with this conditions there will be some situations which all jobs in the queue are belongs to same group, so multiple consumers can't work on them and they must deliver one by one. that's ok.
There's no such thing in AMQ protocol that can lead to this exact solution, there're some ways to solve this problem.
Define queue for each message group
Set concurrency as 1
Let me quote the message ordering from the doc
Section 4.7 of the AMQP 0-9-1 core specification explains the
conditions under which ordering is guaranteed: messages published in
one channel, passing through one exchange and one queue and one
outgoing channel will be received in the same order that they were
sent. RabbitMQ offers stronger guarantees since release 2.7.0.
Ref: https://www.rabbitmq.com/semantics.html
First of the foremost things for you is to preserve the message ordering, once we have ordered messages we can utilize the concurrency to handle the messages in order.
Let's say your queue has 5 messages as shown
Queue: Queue1
+--------------+
Head-->|m1|m2|m3|m4|m5| <---- Tail
+--------------+
There's the concept of competing consumers, competing consumers means there're more than consumers/subscribers for the same queue. If there is more than one consumer than each of them will run autonomously, which means ordering on the consumer side won't be preserved. To preserve the ordering on consumer side, we should not use competing consumers.
Even though now consumers are not competing, we can still lose message ordering, if we have more than one executor. More than one executor simply means we can poll the queue, send a polled message to any of the executors. Based on the CPU execution policy etc we will still lose the ordering, so now we need to restrict the number of executors to 1.
As we have only one executor each of the polled messages will be executed in orders, so it will become a serial execution.
For Queue1
The executor will consume the message in the following order
-> m1
-> m2
-> m3
-> m4
-> m5
Still, there's one missing piece, what happens if the execution of m1 is failing?
You can retry for N number of times before consuming the next message, to achieve this don't acknowledge unless you have successfully executed any polled message.
From design points of view, this does not look good, since you're processing messages in serial instead of parallel, though you don't have any other alternatives.
I have 3 nodes (A,B,C) in my cluster . Right now I want to configure the queue High Availability using the ha-nodes option with nodes A and C as the params.I am successfully configured the HA policy and its working. But after I use the DLX policy for all queues, the HA policy is not working anymore.
Is that normal or am I missing something here?
I want to use the HA policy and DLX policy together, but now it seems impossible. Thanks.
Only one policy is applied at a time for a given queue or exchange:
http://www.rabbitmq.com/parameters.html#policies
But you still can configure HA and dead-lettering together: you just need to do that in one policy. Here is an example:
{
"ha-mode": "nodes",
"ha-params": ["A", "C"],
"dead-letter-exchange": "my-dlx"
}
I want to consume multiple messages from specific queue or a specific exchange with a given key.
so the scenario is as follow:
Publisher publish message 1 over queue 1
Publisher publish message 2 over queue 1
Publisher publish message 3 over queue 1
Publisher publish message 4 over queue 2
Publisher publish message 5 over queue 2
..
Consumer consume messages from queue 1
get [message 1, message 2, message 3] all at once and handle them in one call back
listen_to(queue_name , num_of_msg_to_fetch or all, function(messages){
//do some stuff with the returned list
});
the messages are not coming at the same time, it is like events and i want to collect them in a queue, package them and send them to a third party.
I also read this post:
http://rabbitmq.1065348.n5.nabble.com/Consuming-multiple-messages-at-a-time-td27195.html
Thanks
Don't consume directly from the queue as queues follow round robin algorithm(an AMQP mandate)
Use shovel to transfer the queue contents to a fanout exchange and consume messages right from this exchange. You get all messages across all connected consumers. :)
If you want to consume multiple messages from specific queue, you can try as below.
channel.queueDeclare(QUEUE_NAME, false, false,false, null);
Consumer consumer = new DefaultConsumer(channel){
#Override
public void handleDelivery(String consumerTag,
Envelope envelope,
AMQP.BasicProperties properties,
byte[] body)
throws IOException {
String message = new String(body, "UTF-8");
logger.info("Recieved Message --> " + message);
}
};
You might need to conceptually separate domain-message from RMQ-message. As a producer you'd then bundle multiple domain messages into a single RMQ-message and .produce() it to RMQ. Remember this kind of design introduces timeouts and latencies due to the existence of a window (you might take some impression from Kafka that does bundling to optimize I/O at the cost of latency).
As a consumer then, you'd have a consumer, with typical .handleDelivery implementation that would transform the received body for the processing: byte[] -> Set[DomainMessage] -> your listener.
Want to know the behavior of rabbitmq multiple publisher and consumer.
Does rabbitmq server gives one message to any one of the consumer at a time and other consumers are ideal at that time?
OR
Consumers pick any unattended message from queue, so that at a time, more than one consumers are consuming the message from queue?
Basically I am designing a database queue and do not want more than one inserts at a time.
A message from the queue will be delivered to one consumer only. Ie: once the message makes its way to the queue - it won't be copied (broadcasted) to multiple consumers.
If you want to do broadcast - you have to use multiple queues.
See this tutorial for more details:
http://www.rabbitmq.com/tutorial-two-python.html
yes , RabitMQ supports multiple publisher and consumer.
Multiple Publisher
For publishing a messsge to rabbitmqyou need to declare a factory and do a connection to the rabbitmq server.
then decare a chennel to rabbitmq
ConnectionFactory FACTORY = new ConnectionFactory
FACTORY.setUsername ("guest")
FACTORY.setPassword ("guest")
FACTORY.setVirtualHost ("\")
FACTORY.setPort (5572)
FACTORY.setHost ("localhost")
Connection connection=FACTORY.newConnection
Channel channel = connection.createChannel
the basic key to route a message is a routing key
channel.basicPublish(EXCHANGE_NAME, "Queue1", MessageProperties.PERSISTENT_TEXT_PLAIN, "msg1".getBytes)
channel.basicPublish(EXCHANGE_NAME, "Queue2", MessageProperties.PERSISTENT_TEXT_PLAIN, "msg2".getBytes)
these two messages will be published to a seperate queue as per the routing key as mention queue1 and queue2
2.Multiple Consumer
for multiple consumer we declare a queue and bind to a particular routing key
the the message to that routing key will be publishe to respected queue.
channel.exchangeDeclare(EXCHANGE_NAME, "direct", durable)
channel.queueDeclare("q1", durable, false, false, null)
channel queueBind ("q1", EXCHANGE_NAME,"queue1")// routing key = "queue1"
val q1Consumer = new QueueingConsumer(channel)
channel basicConsume ("q1", false, q1Consumer)
like this u can consume messages from first queue
and same goes for second queue but specify the routing key as "queue2"
channel.exchangeDeclare(EXCHANGE_NAME, "direct", durable)
channel.queueDeclare("q2", durable, false, false, null)
channel queueBind ("q2", EXCHANGE_NAME,"queue2") // routing key = "queue2"
val q2Consumer = new QueueingConsumer(channel)
channel basicConsume ("q2", false, q2Consumer)