What will happen to messages posted to a virtual topic when there are no consumers listening ? Will the broker hold them for a certain while until a subscriber is available ?
More specifically :
At T0 and T1 messages M0 and M1 are posted. At T2, consumer C1 connects, will he receive M0 and M1 ? Obviously messages M2 and M3 posted at T3 and T4 will be received by C1, but what will a new Consumer, C2, that connects at T5 receice ? All messages, M2 and M3, or none ?
It depends on the nature of the topic:
if the topic is durable (has durable consumers subscribing to it), the broker will hold the messages in the topic until all the durable consumers consumes the messages.
if the topic is non-durable (no durable consumers), the message will not even be sent to the topic, as there will be no durable subscription.
For your example, I'll consider that you are using durable subscriptions / consumers:
Case 1:
T-2 C1 and C2 make durable subscription to the topic
T-1 C1 and C2 disconnect
T0: M0 is posted
T1: M1 is posted
T2: C1 connects. C1 receives M0 and M1
T3: M3 is posted. C1 receives M3
T4: M4 is posted. C1 receives M4
T5: C2 connects, C2 receives M0, M1, M2, M3, M4
That's because they are holding durable subscriptions
You need to be very careful when using durable topics / queues: if the consumer doesn't unsubscribe, the broker will hold the messages until the message store explodes. You will need to make sure it doesn't happen (by setting eviction policies and / or putting a Time to Live on the messages).
Of course the previous example will vary depending when the consumer does the durable subscription.
If you are using non-durable topics:
T-2 C1 and C2 make normal subscription to the topic
T-1 C1 and C2 disconnect
T0: M0 is posted
T1: M1 is posted
T2: C1 connects. C1 does not receive anything
T3: M3 is posted. C1 receives M3
T4: M4 is posted. C1 receives M4
T5: C2 connects, C2 does not receive anything
There are two ways to allow messages published to a virtual topic to suivive. The first one is through the durable subscriber and the other is that the publisher sends messages with delivery mode "PERSISTENT". When messages are published with the delivery mode of "PERSISTENT", the message will be saved on disk, otherwise, it will be save in-memory.
Why can't there be a observer/observable pattern - taking the example above:
When M0 is posted, C1 and C2 (consumers subscribed) are woken and can consume the event? I see this pattern better than the durable and non-durable - a hybrid approach.
Related
I am studying the locking method on Redis, and found this official documents
https://redis.io/commands/setnx#handling-deadlocks
My understanding is that SETNX is an atomic operation and ONLY ONE client can do it successful (get 1) and the others failed (get 0)
However in the section Handling deadlocks on this document, it said:
1) C1 and C2 read lock.foo to check the timestamp, because they both received 0 after executing SETNX, as the lock is still held by C3 that crashed after holding the lock.
2) C1 sends DEL lock.foo
3) C1 sends SETNX lock.foo and it succeeds
4) C2 sends DEL lock.foo
5) C2 sends SETNX lock.foo and it succeeds
ERROR: both C1 and C2 acquired the lock because of the race condition.
My question is that why 3) and 5) can be succeed? I think only C1 or C2 can be succeed but not both.
Please correct my understanding, thank you,.
SETNX only sets the key if it doesn't already exist. So, indeed, if multiple clients try to SETNX at the same time, only one will succeed.
But in this case (see steps 2 and 4), the clients are deleting the key before calling SETNX. Since the key no longer exists, there is nothing to prevent SETNX from succeeding.
i am trying to implement below scenario in my application
Exachange e1 -> Queue q1
DLX exchange e2 -> Queue q2
Also i have mentioned DLE and DLK in queue-q1 then message moving to queue-q2 on rejection/failure/timeout.
But how does i resend/retry message from queue-q2 to original queue-q1?
You can do that manually in your application after some analyze and filtering logic. Or you can make some TTL on that queue-q2 to let not consumed messages to be expired. And you also need to specify in this queue a x-dead-letter-exchange as a name for the Exachange e1 for desired recycling.
See more info yin this article:
Create the dead letter exchange, which is just a normal exchange with a special name
Create a retry_message queue and have all messages published to the dead letter exchange route here
When you setup the retry_message queue, be sure to default the following parameter values of the queue
x-message-ttl: 30000 – This will set a ttl on any message published to the queue. When the ttl expires, the message will be republished to the exchange specified in the x-dead-letter-exchange parameter.
x-dead-letter-exchange: original_exchange_name – This is where the message will get republished to once the message ttl expires. We normally want this be the name of the exchange where the message was originally published.
Suppose we have 3 nodes in a cluster.
node1,node2,node3
In node1 we have a
exchange e1 bounded to a queue q1 with binding key =key1
It is attached to a consumer1.
In node2 we have a
exchange e2 bounded to a queue q2 with binding key =key2
It is attached to a consumer2.
Can consumer2 read messages from q1 in cluster ? If not how can this be implemented ?
you can read rabbitMQ route totorial.Though it's using python,the concept would be the same.In the Putting it all together part the consumer 2 can receive info,error and warning from queue 2 while the consumer 1 get error from queue 1.
In your case,c2 can't read message from queue 1 now.To implement,the exchange setting don't need to change.Just bind queue 2 with exchange 1 key 1.
Say I have this pub/sub pattern implemented:
So basically I deliver a message to each C, who subscribed to exchange X.
I have instances of P, and a lot subscribers like C. Let's define C10, C11, C12, C13 as a group C1 and C20, C21, C22, C23 as a group C2.
How do I deliver a message so only one C will receive a message from each group? (I'm perfectly fine with round robin)
Just go to topics tutorial.
Routing key should look like C.C1 or C.C2.
Basically, send messages with routing key C.* ( so it they will go to C.C1 xor C.C2) , and subscribe each consumer to C.C1 xor C.C2. RMQ will distribute messages to all consumers subscribed to C.CN routing key in round-robin fashion.
So appears what I needed was a fanout exchange with named queues instead of exclusive ones.
Each C service declares a non-exclusive named queue and binds it to the exchange. And binds a consumer to that queue.
If two services would declare a same queue, and bind a consumer to it, they end up being round-robin'ed.
I have producer app with 2 separate instances (p1, p2), and consumer app with 2 separate instances (c1, c2).
Producer p1 connects to exchange with topic= t1, queueName =name1.
Consumer c1 connects to exchange with topic= t1, queueName =name1.
Producer p2 connects to exchange with topic= t2, queueName =name1.
Consumer c2 connects to exchange with topic= t2, queueName =name1.
I see in RabbitMQ GUI that I have 2 exchanges but only 1 queue.
Instead that c1 will receive messages from p1 only, and c2 will receive messages from p2 only, RabbitMQ is doing round robin on messages between c1 and c2. So the messages I send from p2 are being received both by c1 and c2.
I thought that in RabbitMQ the correlation is multiple queues per exchange, and the behavior here is unexpected. Why?
You can have multiple queues for every exchange, it's true; but the routing key is a queue matter, not a consumer matter.
The routing key will be used by rabbit to send the message to the right queue; once the message is received on a topic exchange, the message will be sent to all the queues binded to that specific topic. You have only one queue here, that's why both C1 and C2 get the message.
Check this link for a clear example.
If you need to separate C1 and C2, you need to bind them to 2 different queues, not to the same one.