We have multiple consumer to handle create messages, but we want to ensure the FIFO order so if we create a product A, we must to reject the next creations of product A.
The problem is because we have several consumers to treate this type of messages it is possible to have consumer 1 that finish after consumer 2. For exemple the message 1 contains more data to be saved compared to mesage 2 for the same product.
Running RabbitMQ with multiple consumers violates FIFO principle of queue. Is there a way to avoid this with RabbitMQ or we must to orientate our architecture in a manner that the control is more Java threatement oriented ?
Thanks
Running RabbitMQ with multiple consumers violates FIFO principle of
queue.
No, it doesn't. The messages are delivered in FIFO order. Multiple consumers will result in messages being delivered round-robin among them, but they are still delivered in FIFO order.
If you want to retain this order, you must use only one consumer, or coordinate work between your consumer processes.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
Under the hood, how is a FIFO queue turned into a priority queue in a distributed fashion? Are they actually swapping the underlying datastructure, or is it a "hacked" fix
The underlying data structures are multiple queues, each assigned a priority. Each queue is an Erlang VM process. This is why having more than 10 or so priorities isn't recommended as performance suffers. If your load is light enough, this may be acceptable.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
When a message is published with a priority header, the message with the higher priority value gets placed on the head of the queue. This is done by actually swapping the messages in the queue. This is all done when the message is waiting to be consumed in the queue. In order to allow RabbitMQ to actually prioritise the messages, set the basic.qos of the consumer as low as possible. So if a consumer connects to an empty queue whose basic.qos is not set and to which messages are subsequently published, the messages may not spend any time at all waiting in the queue. In this case, the priority queue will not get any opportunity to prioritise them.
Reference: https://www.rabbitmq.com/priority.html
Is there is a way by which we can restrict RabbitMQ Queue to dispatch only a fixed number of messages from the Queue to the consumers?
I have 2 Queues Q1 and Q2 and 10 consumers.Every consumer can process the messages from Q1 and Q2.At any given time, only 2 consumers should process messages from Q2.All the 10 consumers can process message from Q1 simultaneously.
Is there any configuration in RabbitMQ which we can specify, so that RabbitMQ pushes only 2 messages from Q2 to any free consumer and push the next 2 only after they are acknowledged, even though other consumers are free and ready to consume.
More background on the issue:
Why only process 2 messages at a time ? :
Q2 messages are doing a web service call and the web service end point(third party) can only service 2 messages concurrently.
Cant we use concurrency ? :
If we use a ListenerContainer (Spring AMQP) the container is per consumer. We can restrict how many message one consumer can take at a time, but when we have 10 consumers, if there are messages in the Queue, each consumer will get its share.
Can we configure only 2 consumers listening to Q2 ? :
I understand we can achieve this by configuring only 2 consumers for Q2, but I am trying to avoid that. If for some reason these 2 consumers goes down, the processing of Q2 will be halted. If 10 consumers are configured, we can guarantee the processing will happen until the last consumer is down.
Looking to see if there is some config in RabbitMQ which we can make use of or any suggested solution.
Thanks in advance !
I'm pretty sure that consumer prefetch will accomplish what you want. But, Q2 can only have one consumer for this to work. There is no way to coordinate among multiple consumers - you would have to do that yourself, and could use RabbitMQ to do the coordination.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I think you're getting wrapped up in the problem definition. What you really need is trivial, so let's break this down a bit.
Given two queues, Q1 and Q2
10 consumers
Every consumer can process the messages from Q1 and Q2.
At any given time, only 2 consumers should process messages from Q2.
All the 10 consumers can process message from Q1 simultaneously.
Comments on problem statement
First, queues are assumed to be indepenedent. An independent process P will have queue Q, thus Q1 serves process P1. This is a strict mathematical requirement - you cannot define two queues for a single process P.
Thus, the second constraint is mathematically incorrect, for the same reason that you could not write a valid function that accepts a parameter of type string and bool interchangeably. It must accept one or the other, as they are not compatible types, or it must accept a single common ancestor of the types without regard to the subtypes. This is a variant of the Liskov Substitution Principle.
Redefining the problem
There are a total of 12 consumers in the system:
Q1 has 10 consumers
Q2 has 2 consumers
[Important] Consumers are not shared between queues
Is there any configuration in RabbitMQ which we can specify, so that RabbitMQ pushes only 2 messages from Q2 to any free consumer and push the next 2 only after they are acknowledged, even though other consumers are free and ready to consume.
Based on the new definition of the problem, you have two options:
Use a Basic.Get - pull the next message from the queue as soon as the consumer finishes processing the last message.
Use consumer prefetch with limit 1. This will deliver the first and second messages for each consumer immediately, then deliver additional messages one at a time as the next message for that consumer is acknowledged. This is a bit more complicated, but might make sense if your latency margins are less than 10 milliseconds.
Note that by properly defining the problem space, we have eliminated the fundamental problem of trying to figure out how to ensure only two consumers are processing Q2 messages at any time.
try the new feature Single Active Consumer from version 3.8+.
Single active consumer allows to have only one consumer at a time consuming from a queue and to fail over to another registered consumer in case the active one is cancelled or dies.
Consuming with only one consumer is useful when messages must be consumed and processed in the same order they arrive in the queue.
Single active consumer can be enabled when declaring a queue, with the x-single-active-consumer argument set to true
https://www.rabbitmq.com/consumers.html#single-active-consumer
e.g. with the Java client:
If I have two queues from which I want to consume messages, and I use a single SimpleMessageQueueListenerContainer for it, in which order would the listeners be invoked/messages consumed when both queues have messages?
I will try to be more specific of the problem I am working on:
I have a consumer application which needs to consume messages from 2 queues – say regular-jobs-queue and infrequent-jobs-queue. If there are any messages in ‘infrequent-jobs-queue’ I want to consume those before consuming messages from ‘regular-jobs-queue’. I might not be able to combine these and put all messages into a single rabbitmq level priority queue and assign higher priority to infrequent-job message because of some upcoming use-cases like purging regular-jobs without affecting infrequent-jobs and others.
I am aware that RabbitMQ has support for consumer priority but I am not very sure if it will be applicable here. I want all instances of my consumer application to first consume messages of infrequent-jobs-queue if any and not prioritize amongst these consumers.
Or should I like have 2 containers, with dedicated consumer thread(s) per queue and have an internal priority-queue data structure into which I can put messages as and when consumed from rabbitmq queue.
Any help would be really appreciated. Thanks.
~Rashida
You can't do what you want; messages will be delivered with equal priority.
Moving them to an internal in-memory queue will risk message loss.
You might want to consider using one of the RabbitTemplate.receive() or receiveAndConvert() methods instead of a message-driven container.
That way you have complete control.
We're seeing an issue where consumers of our message queues are picking up messages from queues at the top of the alphabetical range. We have two applications: a producer, and a subscriber. We're using RabbitMQ 3.6.1.
Let's say that the message queues are setup like so:
Our first application, the producer, puts say 100 messages/second onto each queue:
Our second application, the subscriber, has five unique consumer methods that can deal with messages on each respective queue. Each method binds to it's respective queue. A subscriber has a prefetch of 1 meaning it can only hold one message at a time, regardless of queue. We may run numerous instances of the subscriber like so:
So the situation is thus: each queue is receiving 100 msg/sec, and we have four instances of subscriber consuming these messages, so each queue has four consumers. Let's say that the consumer methods can deal with 25 msg/sec each.
What happens is that instead of all the queues being consumed equally, the alphabetically higher queues instead get priority. It's seems as though when the subscriber becomes ready, RabbitMQ looks down the list of queues that this particular ready channel is bound to, and picks the first queue with pending messages.
In our situation, A_QUEUE will have every message consumed. B_QUEUE may have some consumed in certain race conditions, but C_QUEUE/D_QUEUE and especially E_QUEUE will rarely get touched.
If we turn off the publisher, the queues will eventually drain, top to bottom.
Is it possible to configure either RabbitMQ itself or possibly even the channel to use some sort of round robin distribution policy or maybe even random policy so that when a channel has numerous bound queues, all with messages pending, the distribution is even?
to clarify: you have a single subscriber application with multiple consumers in it, right?
I'm guessing you're using a single RabbitMQ Connection within the subscriber app.
Are you also re-using a single RabbitMQ Channel for all of your consumers? If so, that would be a problem. Be sure to use a new Channel for each consumer you start.
Maybe the picture is wrong, but if it's not then your setup is wrong. You don't need 4 queues if you are going to have subscribers that listen to each and every queue. You'd just need one queue, that has multiple instances of the same subscriber consuming from it.
Now to answer, yes (but no need to configure, as long as prefetch is 1), actually rabbitmq does distribute messages evenly. You can find about about that here, and on the same place actually how your setup should look like. Here is a quote from the link.
RabbitMQ just dispatches a message when the message enters the queue.
It doesn't look at the number of unacknowledged messages for a
consumer. It just blindly dispatches every n-th message to the n-th
consumer.
I need to choose a new Queue broker for my new project.
This time I need a scalable queue that supports pub/sub, and keeping message ordering is a must.
I read Alexis comment: He writes:
"Indeed, we think RabbitMQ provides stronger ordering than Kafka"
I read the message ordering section in rabbitmq docs:
"Messages can be returned to the queue using AMQP methods that feature
a requeue
parameter (basic.recover, basic.reject and basic.nack), or due to a channel
closing while holding unacknowledged messages...With release 2.7.0 and later
it is still possible for individual consumers to observe messages out of
order if the queue has multiple subscribers. This is due to the actions of
other subscribers who may requeue messages. From the perspective of the queue
the messages are always held in the publication order."
If I need to handle messages by their order, I can only use rabbitMQ with an exclusive queue to each consumer?
Is RabbitMQ still considered a good solution for ordered message queuing?
Well, let's take a closer look at the scenario you are describing above. I think it's important to paste the documentation immediately prior to the snippet in your question to provide context:
Section 4.7 of the AMQP 0-9-1 core specification explains the
conditions under which ordering is guaranteed: messages published in
one channel, passing through one exchange and one queue and one
outgoing channel will be received in the same order that they were
sent. RabbitMQ offers stronger guarantees since release 2.7.0.
Messages can be returned to the queue using AMQP methods that feature
a requeue parameter (basic.recover, basic.reject and basic.nack), or
due to a channel closing while holding unacknowledged messages. Any of
these scenarios caused messages to be requeued at the back of the
queue for RabbitMQ releases earlier than 2.7.0. From RabbitMQ release
2.7.0, messages are always held in the queue in publication order, even in the presence of requeueing or channel closure. (emphasis added)
So, it is clear that RabbitMQ, from 2.7.0 onward, is making a rather drastic improvement over the original AMQP specification with regard to message ordering.
With multiple (parallel) consumers, order of processing cannot be guaranteed.
The third paragraph (pasted in the question) goes on to give a disclaimer, which I will paraphrase: "if you have multiple processors in the queue, there is no longer a guarantee that messages will be processed in order." All they are saying here is that RabbitMQ cannot defy the laws of mathematics.
Consider a line of customers at a bank. This particular bank prides itself on helping customers in the order they came into the bank. Customers line up in a queue, and are served by the next of 3 available tellers.
This morning, it so happened that all three tellers became available at the same time, and the next 3 customers approached. Suddenly, the first of the three tellers became violently ill, and could not finish serving the first customer in the line. By the time this happened, teller 2 had finished with customer 2 and teller 3 had already begun to serve customer 3.
Now, one of two things can happen. (1) The first customer in line can go back to the head of the line or (2) the first customer can pre-empt the third customer, causing that teller to stop working on the third customer and start working on the first. This type of pre-emption logic is not supported by RabbitMQ, nor any other message broker that I'm aware of. In either case, the first customer actually does not end up getting helped first - the second customer does, being lucky enough to get a good, fast teller off the bat. The only way to guarantee customers are helped in order is to have one teller helping customers one at a time, which will cause major customer service issues for the bank.
It is not possible to ensure that messages get handled in order in every possible case, given that you have multiple consumers. It doesn't matter if you have multiple queues, multiple exclusive consumers, different brokers, etc. - there is no way to guarantee a priori that messages are answered in order with multiple consumers. But RabbitMQ will make a best-effort.
Message ordering is preserved in Kafka, but only within partitions rather than globally. If your data need both global ordering and partitions, this does make things difficult. However, if you just need to make sure that all of the same events for the same user, etc... end up in the same partition so that they are properly ordered, you may do so. The producer is in charge of the partition that they write to, so if you are able to logically partition your data this may be preferable.
I think there are two things in this question which are not similar, consumption order and processing order.
Message Queues can -to a degree- give you a guarantee that messages will get consumed in order, they can't, however, give you any guarantees on the order of their processing.
The main difference here is that there are some aspects of message processing which cannot be determined at consumption time, for example:
As mentioned a consumer can fail while processing, here the message's consumption order was correct, however, the consumer failed to process it correctly, which will make it go back to the queue. At this point the consumption order is intact, but the processing order is not.
If by "processing" we mean that the message is now discarded and finished processing completely, then consider the case when your processing time is not linear, in other words processing one message takes longer than the other. For example, if message 3 takes longer to process than usual, then messages 4 and 5 might get consumed and finish processing before message 3 does.
So even if you managed to get the message back to the front of the queue (which by the way violates the consumption order) you still cannot guarantee they will also be processed in order.
If you want to process the messages in order:
Have only 1 consumer instance at all times, or a main consumer and several stand-by consumers.
Or don't use a messaging queue and do the processing in a synchronous blocking method, which might sound bad but in many cases and business requirements it is completely valid and sometimes even mission critical.
There are proper ways to guarantuee the order of messages within RabbitMQ subscriptions.
If you use multiple consumers, they will process the message using a shared ExecutorService. See also ConnectionFactory.setSharedExecutor(...). You could set a Executors.newSingleThreadExecutor().
If you use one Consumer with a single queue, you can bind this queue using multiple bindingKeys (they may have wildcards). The messages will be placed into the queue in the same order that they were received by the message broker.
For example you have a single publisher that publishes messages where the order is important:
try (Connection connection2 = factory.newConnection();
Channel channel2 = connection.createChannel()) {
// publish messages alternating to two different topics
for (int i = 0; i < messageCount; i++) {
final String routingKey = i % 2 == 0 ? routingEven : routingOdd;
channel2.basicPublish(exchange, routingKey, null, ("Hello" + i).getBytes(UTF_8));
}
}
You now might want to receive messages from both topics in a queue in the same order that they were published:
// declare a queue for the consumer
final String queueName = channel.queueDeclare().getQueue();
// we bind to queue with the two different routingKeys
final String routingEven = "even";
final String routingOdd = "odd";
channel.queueBind(queueName, exchange, routingEven);
channel.queueBind(queueName, exchange, routingOdd);
channel.basicConsume(queueName, true, new DefaultConsumer(channel) { ... });
The Consumer will now receive the messages in the order that they were published, regardless of the fact that you used different topics.
There are some good 5-Minute Tutorials in the RabbitMQ documentation that might be helpful:
https://www.rabbitmq.com/tutorials/tutorial-five-java.html