I have the following architecture:
Architecture
There are a fixed number of input sources. Each input source is equivalent.
The AMQP broker. I am using RabbitMQ in my case.
Currently, there are 2 consumers. Again, each consumer is equivalent.
The input sources are sending commands to be processed. These commands are forwarded by the broker and picked up by one of the two consumers.
I need the following behaviour:
If one input source sends multiple commands, all commands must be processed sequentially. That is, in the example of 2 commands, it is not allowed that consumer 1 is processing command 1 while consumer 2 is processing command 2 at the same time.
However, two commands originating from two different input sources can be processed simultaneously.
Is it possible to enforce this behaviour with AMQP/RabbitMQ?
You can cover your scenario using one consumer for each queue.
Each queue can process the message sequentially.
Another way is to use only one queue and use the envelope.getExchange() to understand the source, or tag your messages using the AMQP.BasicProperties properties
In this way, for example, you can consume the messages in multi-threading and assign one thread for each tag
To guarantee sequence you may need to aggregate the messages. You can batch the commands from one source into a message before publishing to the queue, so the message into the queue can contain one or more commands that will be executed by the consumer.
Related
If I make multiple StackExchange.Redis calls from a single thread using fire-and-forget, are they guaranteed to be delivered in order?
Use case: I am adding an item to a Redis stream and then using pub/sub to trigger another service to process that stream. To avoid a race condition, I need to make sure that the item is added to the stream before the pub/sub message is delivered.
While most StackExchange.Redis APIs are thread-safe, the order of delivery of commands sent through SE.Redis can't be guaranteed out-of-the-box in your scenario for several reasons:
your topology could have multiple nodes, where each message of your sequence is delivered to a different node for a change in the topology or according to your own preferences (CommandFlags.Prefer* / CommandFlags.Demand*);
your thread could host multiple tasks whose continuations do not respect the intended delivery order;
being fire-and-forget, a failure in the delivery of the first command would not stop sending the subsequent ones;
I need to make sure that the item is added to the stream before the pub/sub message is delivered.
I suggest using a Lua script to solve this, which would execute your commands within the same atomic unit and against the same node:
redis.call('XADD', 'foo', '*', 'bar', 'baz')
redis.call('PUBLISH', 'foo-added', '')
Similar to this question, we have FIFO queues and the messages must be processed in order. We want competing consumers from different machines for redundancy and performance reasons, but only one consumer on one machine should handle a message for a given queue at a time.
I tried setting the prefetch count to 1, but I believe this will only work if used with a single machine. Is this possible by default with RabbitMQ or do we need to implement our own lock?
Given a single queue with multiple consumers there is no way to block one of the consumers, all of them receive the messages in round-robin fashion.
EDIT
See https://www.rabbitmq.com/consumers.html#single-active-consumer
/EDIT
You could see this plugin, https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange to distribute the load using different queues.
I tried setting the prefetch count to 1
prefetch=1 means that the consumers take one message at a time.
do we need to implement our own lock
Yes, If you want one single consumer for queue avoiding other consumers.
EDIT
There are also the Exclusive Queues https://www.rabbitmq.com/queues.html#exclusive-queues but note:
Exclusive queues are deleted when their declaring connection is closed or gone (e.g. due to underlying TCP connection loss). They, therefore, are only suitable for client-specific transient state.
I have a camel route processing messages from a RabbitMQ endpoint. I am keeping the defaults for concurrentConsumers (1) and threadPoolSize(10).
I am relative new to RabbitMQ, and still do not quite understand the relationship between the concurrentConsumer and threadPoolSize properties. The messages in my queues need to be processed in sequence, which I think shall be achieved by using a single consumer. However, will using a threadPoolSize value greater than one cause messages to be processed in parallel?
The default value is 10 (source : https://camel.apache.org/components/latest/rabbitmq-component.html)
It won't affect your concurrency. That means the only one consumer will have 10 threads available to use for the process. You can check at exclusiveConsumer if you want only one consumer shared between all your apps (needed if you could have multiple apps targeting the queue)
I have a question about multi consumer concurrency.
I want to send works to rabbitmq that comes from web request to distributed queues.
I just want to be sure about order of works in multiple queues (FIFO).
Because this request comes from different users eech user requests/works must be ordered.
I have found this feature with different names on Azure ServiceBus and ActiveMQ message grouping.
Is there any way to do this in pretty RabbitMQ ?
I want to quaranty that customer's requests must be ordered each other.
Each customer may have multiple requests but those requests for that customer must be processed in order.
I desire to process quickly incoming requests with using multiple consumer on different nodes.
For example different customers 1 to 1000 send requests over 1 millions.
If I put this huge request in only one queue it takes a lot of time to consume. So I want to share this process load between n (5) node. For customer X 's requests must be in same sequence for processing
When working with event-based systems, and especially when using multiple producers and/or consumers, it is important to come to terms with the fact that there usually is no such thing as a guaranteed order of events. And to get a robust system, it is also wise to design the system so the message handlers are idempotent; they should tolerate to get the same message twice (or more).
There are way to many things that may (and actually should be allowed to) interfere with the order;
The producers may deliver the messages in a slightly different pace
One producer might miss an ack (due to a missed package) and will resend the message
One consumer may get and process a message, but the ack is lost on the way back, so the message is delivered twice (to another consumer).
Some other service that your handlers depend on might be down, so that you have to reject the message.
That being said, there is one pattern that servicebus-systems like NServicebus use to enforce the order messages are consumed. There are some requirements:
You will need a centralized storage (like a sql-server or document store) that allows for conditional updates; for instance you want to be able to store the sequence number of the last processed message (or how far you have come in the process), but only if the already stored sequence/progress is the right/expected one. Storing the user-id and the progress even for millions of customers should be a very easy operation for most databases.
You make sure the queue is configured with a dead-letter-queue/exchange for retries, and then set your original queue as a dead-letter-queue for that one again.
You set a TTL (for instance 30 seconds) on the retry/dead-letter-queue. This way the messages that appear on the dead-letter-queue will automatically be pushed back to your original queue after some timeout.
When processing your messages you check your storage/database if you are in the right state to handle the message (i.e. the needed previous steps are already done).
If you are ok to handle it you do and update the storage (conditionally!).
If not - you nack the message, so that it is thrown on the dead-letter queue. Basically you are saying "nah - I can't handle this message, there are probably some other message in the queue that should be handled first".
This way the happy-path is to process a great number of messages in the right order.
But if something happens and a you get a message out of band, you will throw it on the retry-queue (the dead-letter-queue) and Rabbit will make sure it will get back in the queue to be retried at a later stage. But only after a delay.
The beauty of this is that you are able to handle most of the situations that may interfere with processing the message (out of order messages, dependent services being down, your handler being shut down in the middle of handling the message) in exact the same way; by rejecting the message and letting your infrastructure (Rabbit) take care of it being retried after a while.
(Assuming the OP is asking about things like ActiveMQs "message grouping:)
This isn't currently built in to RabbitMQ AFAIK (it wasn't as of 2013 as per this answer) and I'm not aware of it now (though I haven't kept up lately).
However, RabbitMQ's model of exchanges and queues is very flexible - exchanges and queues can be easily created dynamically (this can be done in other messaging systems but, for example, if you read ActiveMQ documentation or Red Hat AMQ documentation you'll find all of the examples in the user guides are using pre-declared queues in configuration files loaded at system startup - except for RPC-like request/response communication).
Also it is very easy in RabbitMQ for a consumer (i.e., message consuming thread) to consume from multiple queues.
So you could build, on top of RabbitMQ, a system where you got your desired grouping semantics.
One way would be to create dynamic queues: The first time a customer order was seen or a new group of customer orders a queue would be created with a unique name for all messages for that group - that queue name would be communicated (via another queue) to a consumer who's sole purpose was to load-balance among other consumers that were responsible for handling customer order groups. I.e., the load-balancer would pull off of its queue a message saying "new group with queue name XYZ" and it would find in a pool of order group consumer a consumer which could take this load and pass it a message saying "start listening to XYZ".
Another way to do it is with pub/sub and topic routing - each customer order group would get a unique topic - and proceed as above.
RabbitMQ Consistent Hash Exchange Type
We are using RabbitMQ and we have found a plugin. It use Consistent Hashing algorithm to distribute messages in order to consistent keys.
For more information about Consistent Hashing ;
https://en.wikipedia.org/wiki/Consistent_hashing
https://www.youtube.com/watch?v=viaNG1zyx1g
You can find this plugin from rabbitmq web page
plugin : rabbitmq_consistent_hash_exchange
https://www.rabbitmq.com/plugins.html
I have NServiceBus running in a single process, but I would like to send Message A, but receive only Message B. However, I think because Message Endpoint Mappings are used for both sending and receiving, the process is trying to handle both messages - A and B. Any way around this issue? Both messages go onto the same queue, based on the fact that a single process can only listen to a single queue NSB limitation.
You could use two different AppDomains.
The statement
a single process can only listen to a single queue
Is not actually correct. It is more correct to say
a single appdomain can only listen to a single queue
Since you can have multiple appdomains per process you could have a since processes listening to multiple queues