A producer sends the message with routing key (x) to the exchange once in 5 secs. Based on the routing key, it is send to 2 queues A and B. The consumer which consumes from A wants it once in 5 secs , but the consumer that consumes from B needs the message only once in one min..
One way is to consume the messages from Queue B and discard the messages that we don't need(only considering once in one min) .
Is there any other better way for this to do in rabbitmq ?
A consumer can keep state about the last time he consumed a message. When a new message arrives, can check if the the desired time has passed since the last message was consumed. If it wasn't, he can just ignore the message.
This is logic that must be handled by the consumer, not the RabbitMQ broker.
The deduplication exchange of the RMQ deduplication plugin has been designed to fit this purpose.
You can specify the time in which you don't want to see a similar message more than once and the exchange will drop any further copy of the given message. Messages are identified via the x-deduplication-header.
The amount of time a message is guaranteed to be unique can be controlled via the x-cache-ttl exchange argument or message header.
Related
I wonder if the following scenario is possible:
Create an exchange of type direct
Publish a message to that exchange with routing key rk1
After that:
Create a queue which accepts messages with routing key rk1
Consume message published to exchange
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
So basically I want to be able to produce messages when there are no consumers present. And consume them some time later.
It seems like if there is no queue present, the message is dropped and there is no way to receive it.
Yes, this is correct, but it's only part of the story.
A message queue is the storage location for messages published to the server. A consumer is a designated connection set to receive messages put into a queue. The exchange is simply a location to push messages. It contains the routing semantics to determine which messages wind up in the queues on the server. When a message cannot be routed to a queue and/or consumer, there are various semantics that can apply, but the default is that the message is dropped.
Options for dealing with unroutable messages:
Alternate exchange - designates a different exchange where messages can be dumped if they cannot be routed to a queue on the current exchange. This can be thought of similar to how TCP/IP works when a destination host is not reachable on the current subnet, and traffic is forwarded to the gateway. Note that a queue must be bound to the alternate exchange for the message to be dumped into. A typical case might be to have it configured as fanout exchange with one queue to trap all messages sent into the alternate exchange.
Mandatory or Immediate - return a message back to the sender if it can't be delivered. The server does not store the message.
Mandatory designates that the message must be deliverable to a queue at the time it is published. If the message is not routable, the publisher will receive a basic.return.
Immediate designates that, in addition to being deliverable, must be immediately routed to a consumer on a particular queue (e.g. it's not good enough that it be dumped in a queue for pickup later - it has to be delivered to the end consumer right now.
In every case, if there is no queue, the server cannot store the message.
The entity queue is the one that is supposed to hold the messages , so without a queue the messages will be lost.
However in case you do not create any exchange with appropriate routing key you may leverage dead lettering feature in rabbitmq.
Another solution could be to declare the queue with the binding after the exchange and before publishing the message; this way the message will be routed and stored, but you may have to add some TTLs ( https://www.rabbitmq.com/ttl.html ).
After reading documentation about what is Rabbit and what does, I have some common questions:
Case is: Producer sends one message to some consumers (subscribers).
I need a short explanation for all points of list below, what to use, and what to dig further.
How to clear queue and stop sending message to consumers after
specific time/date?
Can I include to confirmed message user's data like JSON?
Where is stored this data? In the same queue?
How to filter confirmed messages and then clear queue?
How to clear queue after specific time/date?
What happens if not one consumer no confirms message, how long they are stored?
Does consumer subscribe on queue or can subscribe on exchange too?
Using model one to many for sending message, how to set who have to
get message first/last or at the same time, here described that, but not clear is it on client or server side?
If no consumers, how to re-push message to another queue and close
current?
Each consumer has own queue?
Thank you in advance and any comment to this question!
If you can elaborate some of your questions and include what is your use case, I can edit the answer.
1 - As long as consumer is alive rabbitmq sends incoming messages to consumer. You can give TTL to messages/queues if you want them to expire after some time.
https://www.rabbitmq.com/ttl.html
2 - What you mean?
3 - Rabbitmq stores the data in mnesia database.
https://www.rabbitmq.com/persistence-conf.html
https://www.rabbitmq.com/relocate.html
4 - What you mean by filterig messages and clear queue? Successfully consumed messages removed from the queue immediatly.
5 - You can give ttl to queue or declare queue as auto delete
https://www.rabbitmq.com/ttl.html
https://www.rabbitmq.com/queues.html
6 - If consumers don't send ack to rabbit, messages stays unack as long as memory becomes full or rabbit becomes unavailable
7 - Both. A consumer can create its own queue and bind it to an exchange or it can consume from existing queue. It depends on the use case.
8 - It is hard to answer this without knowing details of what you mean by one-to-many. Direct exchange or fanout or whatelse, how many queues etc.
But, in rabbitmq, messages ordered by publish order by default.
According to link you shared, rabbitmq sends messages first to higher priority consumers until consumer prefetch count (unack messages on consumer) becomes its limits.
9 - You need to handle this case in the code. Or you can use management ui with Shovel plugin.
https://www.rabbitmq.com/management.html
https://www.rabbitmq.com/shovel.html
10 - Again, it depends on the design and use case.
I have implemented the example from the RabbitMQ website:
RabbitMQ Example
I have expanded it to have an application with a button to send a message.
Now I started two consumer on two different computers.
When I send the message the first message is sent to computer1, then the second message is sent to computer2, the thrid to computer1 and so on.
Why is this, and how can I change the behavior to send each message to each consumer?
Why is this
As noted by Yazan, messages are consumed from a single queue in a round-robin manner. The behavior your are seeing is by design, making it easy to scale up the number of consumers for a given queue.
how can I change the behavior to send each message to each consumer?
To have each consumer receive the same message, you need to create a queue for each consumer and deliver the same message to each queue.
The easiest way to do this is to use a fanout exchange. This will send every message to every queue that is bound to the exchange, completely ignoring the routing key.
If you need more control over the routing, you can use a topic or direct exchange and manage the routing keys.
Whatever type of exchange you choose, though, you will need to have a queue per consumer and have each message routed to each queue.
you can't it's controlled by the server check Round-robin dispatching section
It decides which consumer turn is. i'm not sure if there is a set of algorithms you can pick from, but at the end server will control this (i think round robin algorithm is default)
unless you want to use routing keys and exchanges
I would see this more as a design question. Ideally, producers should create the exchanges and the consumers create the queues and each consumer can create its own queue and hook it up to an exchange. This makes sure every consumer gets its message with its private queue.
What youre doing is essentially 'worker queues' model which is used to distribute tasks among worker nodes. Since each task needs to be performed only once, the message is sent to only one node. If you want to send a message to all the nodes, you need a different model called 'pub-sub' where each message is broadcasted to all the subscribers. The following link shows a simple pub-sub tutorial
https://www.rabbitmq.com/tutorials/tutorial-three-python.html
I have a scenario in my RabbitMQ setup that I'm curious about how to solve. The diagram below illustrates it (exchanges and most queues removed for succinctness):
Scenario
Producer creates message A(1), it is received by the top consumer, which begins processing the message.
Producer creates message A(2), it is received by the bottom consumer (assuming both consumers are on a round-robin exchange).
The bottom consumer publishes message B(2), which is put into Message B consumer's queue
The poor slow top consumer finally finishes and emits its message B(1).
Problem
If we assume that B consumer cannot be made idempotent, how do we ensure the result of both B messages are applied in the correct order?
I had thought of using a timestamp that is applied to the initial publish of message A, and having the consumer maintain a timestamp of last change, rejecting any timestamps before that time, but that only works if each message causes the exact same kind of change and requires a lot of tracking.
Other ideas for how to approach this would be appreciated. Thanks!
I am not sure what is specific to RabbitMQ here, but the idea with timestamps sounds like a good start if you have a single producer.
The producer attaches a timestamp to the messages A, each message B take the same timestamp of its respective message A.
With your approach some messages would not be processed, eg, message B(1). If all messages should be processed by consumer B, but they should be processed in a deterministic order, then you can do a deterministic merge:
Consumer B is equipped with two queues, one queue for each consumer A. Consumer B always checks the top of both queues:
if both queues are non-empty, consumer B pops the message with the lowest timestamp.
if at least one queue is empty, the consumer B waits.
With this approach the order in which consumer B processes messages is given by the timestamps of the producer and no message is discarded. Assumptions are:
queues are FIFO
no process crashes
always the case that eventually each consumer A processes a message
consumer B can check the top of the queues in a non-blocking fashion
I have a rabbitmq cluster used as a working queue. There are 5 kinds of consumers who want to consume exactly the same data.
What I know for now is using fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
My question is, does rabbitmq support to push the same data to multi consumers? Just like a message need to be acked for a specified times to be deleted.
I got the following answer from rabbitmq email group. In short, the answer is no... and what I did above is the correct way.
http://rabbitmq.1065348.n5.nabble.com/Does-rabbitmq-support-to-push-the-same-data-to-multi-consumers-td36169.html#a36170
... fanout exchange to "copy" the data to 5 DIFFERENT queues. And the 5 consumers can consume different queue. This is kind of wasting resources because the data is the same in file queues.
You can consume with 5 consumers from one queue if you do not want to duplicate messages.
does rabbitmq support to push the same data to multiple consumers
In AMQP protocol terms you publish message to exchange and then broker (RabbitMQ) decide what to do with messages - assume it figured out the queue message intended for (one or more) and then put that message on top of that queue (queues in RabbitMQ are classic FIFO queues which is somehow break AMQP implementation in RabbitMQ). Only after that message may be delivered to consumer (or die due to queue length limit or per-queue or per-message ttl, if any).
message need to be acked for a specified times to be deleted
There are no way to change message body or attributes after message being published (actually, Dead Letter Exchanges extension and some other may change routing key, for example and add,remove and change some headers, but this is very specific case). So if you want to track ack's number you have to re-publish consumed message with changed body or header (depends on where do you plan to store ack's counter, but headers fits pretty nice for this.
Also note, that there are redeliverd message attribute which denotes whether message was already was consumed, but then redelivered. This flag doesn't count redelivers number so it usage is quite limited.