Assume I have a mirrored queue deployed over multiple nodes (f.e. 1 master + 1 mirror). I can define the number of mirrors I want but is it possible to only accept a producer message when the message is stored at least on 2 queues (master + mirror). Otherwise it is still possible to loose a message when the master node fails before the message is mirrored.
So the mirroring activity should be part of the transaction.
You should use Publisher Confirms. When this is enabled, and your publisher has received confirmation, you can be certain that your message has been replicated to all queue mirrors.
Searching Google for site:rabbitmq.com high availability returns this document which mentions Publisher Confirms here.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
While looking at the Pub/Sub pattern, i came across the fellowing scenario:
Assume that you have a horizontally scaled app, that has X instances. All of them subscribe to a topic where messages like "Transfer $10 from account A to account B". When someone publish a message to that topic, all subscriber will get that message?
In the case above, clearly, the message should be taken by only 1 subscriber and handled only once.
How does one handle this scenario? Do you abandon the pub/sub and starts pooling?
Let me explain few things with example before you understand that completely. I have worked on Azure service bus so i will explain in that context.
In Pub/sub you have one topic and possible multiple subscription. Lets say we have topic "Shopping-Topic". We have 2 Subscriptions called "Payment-Subscription", "Cart-Subscription". Now we publish message "Payment-processed" on the topic. It's the discretion of subscription to pick that message and reason is that subscription have to mention that which messages it want pick.
In Azure service bus we have something called rule (message label). Default rule is that subscription is listening to all the messages but we can overwrite this behavior and say i am only interested in particular message. In the above case rule added against "Payment-Subscription" to listen the message "Payment-processed" so the message is added to "Payment-Subscription" subscription for it to process. Even though "Cart-Subscription" is also subscribed to the same topic but it is ignoring this message so it's not added to its subscription. This way any intended subscription can listen to particular message not necessarily all of them.
Now we discuss individual subscription. Let's say we have message added to "Payment-Subscription". This subscription has 2 instances/processes that are ready to process the message "Payment-processed". The first process to pick the message will process the message and remove it from subscription.
In RabbitMQ Normally, active consumers connected to the same queue receive messages from it in a round-robin fashion. So this insures that a message is processed exactly once.
So in your case you should design a queue where all the messages for
"Transfer $10 from account A to account B"
Are routed to and all the consumers register themselves on this queue itself , this insures that one message will go to only one subscriber.
Another point not related to your question but is important to know is that there is another concept called "Consumer Priorities" which allows you to ensure that high priority consumers receive messages while they are active, with messages only going to lower priority consumers when the high priority consumers block.
More info can be found here
If I have and exchange with a single queue bound to it. I have two processes each with their own connection. Each connection has its own consumer which is consuming messages from that single queue.
Is it possible that when a message is placed onto the queue, both consumers could be delivered the same message?
Pre-fetch seems to apply across consumers on a single channel, or connection if the global flag is set. What happens with consumers across multiple connections
No. Messages are always round-robin delivered to multiple consumers on the same queue.
One solution is to have two queues bound to the same exchange using the same routing key if you need messages to go to both consumers.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Two RabbitMQ clusters use two-way federation exchange to replicate messages, but how to achieve consume messages synchronously?
I use a consumer to consume messages in one cluster and messages in one cluster will be deleted, but in another cluster the messages are still there. How can I achieve that when I use a consumer to consume messages? In both two cluster the messages will be deleted.
Federation only copies messages, it does not copy actions on those messages. They will remain in the other cluster until they are consumed or deleted.
One option would be to set a message TTL for the other queue so that the messages will be automatically deleted after a certain period of time.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
We have multiple consumer to handle create messages, but we want to ensure the FIFO order so if we create a product A, we must to reject the next creations of product A.
The problem is because we have several consumers to treate this type of messages it is possible to have consumer 1 that finish after consumer 2. For exemple the message 1 contains more data to be saved compared to mesage 2 for the same product.
Running RabbitMQ with multiple consumers violates FIFO principle of queue. Is there a way to avoid this with RabbitMQ or we must to orientate our architecture in a manner that the control is more Java threatement oriented ?
Thanks
Running RabbitMQ with multiple consumers violates FIFO principle of
queue.
No, it doesn't. The messages are delivered in FIFO order. Multiple consumers will result in messages being delivered round-robin among them, but they are still delivered in FIFO order.
If you want to retain this order, you must use only one consumer, or coordinate work between your consumer processes.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
After reading documentation about what is Rabbit and what does, I have some common questions:
Case is: Producer sends one message to some consumers (subscribers).
I need a short explanation for all points of list below, what to use, and what to dig further.
How to clear queue and stop sending message to consumers after
specific time/date?
Can I include to confirmed message user's data like JSON?
Where is stored this data? In the same queue?
How to filter confirmed messages and then clear queue?
How to clear queue after specific time/date?
What happens if not one consumer no confirms message, how long they are stored?
Does consumer subscribe on queue or can subscribe on exchange too?
Using model one to many for sending message, how to set who have to
get message first/last or at the same time, here described that, but not clear is it on client or server side?
If no consumers, how to re-push message to another queue and close
current?
Each consumer has own queue?
Thank you in advance and any comment to this question!
If you can elaborate some of your questions and include what is your use case, I can edit the answer.
1 - As long as consumer is alive rabbitmq sends incoming messages to consumer. You can give TTL to messages/queues if you want them to expire after some time.
https://www.rabbitmq.com/ttl.html
2 - What you mean?
3 - Rabbitmq stores the data in mnesia database.
https://www.rabbitmq.com/persistence-conf.html
https://www.rabbitmq.com/relocate.html
4 - What you mean by filterig messages and clear queue? Successfully consumed messages removed from the queue immediatly.
5 - You can give ttl to queue or declare queue as auto delete
https://www.rabbitmq.com/ttl.html
https://www.rabbitmq.com/queues.html
6 - If consumers don't send ack to rabbit, messages stays unack as long as memory becomes full or rabbit becomes unavailable
7 - Both. A consumer can create its own queue and bind it to an exchange or it can consume from existing queue. It depends on the use case.
8 - It is hard to answer this without knowing details of what you mean by one-to-many. Direct exchange or fanout or whatelse, how many queues etc.
But, in rabbitmq, messages ordered by publish order by default.
According to link you shared, rabbitmq sends messages first to higher priority consumers until consumer prefetch count (unack messages on consumer) becomes its limits.
9 - You need to handle this case in the code. Or you can use management ui with Shovel plugin.
https://www.rabbitmq.com/management.html
https://www.rabbitmq.com/shovel.html
10 - Again, it depends on the design and use case.