I have tried to understand the RabbitMQ guarantees on publishing - so far without success - and I would like to know if someone can help me with this scenario:
Bindings
Exchange A -> Exchange B
Exchange A -> Exchange C
Exchange B -> Queue B1
Exchange C -> Queue C1
Properties
B1 and C1 is declared as persistent queues
B1 and C1 is both mirrored queues
All exchanges are declared as fanout
I'm running a RabbitMQ cluster with multiple nodes - so queues can be master on different nodes
If I publish to Exchange A - do i then either get my message published to all or no of the queues?
I'm currently running in Confirm mode - what would a confirm mean after a publish here?
Hope above makes sense - thanks!
If I publish to Exchange A - do i then either get my message published
to all or no of the queues?
Since you are using fanout exchanges, queue B1 and C1 will get any message published to exchange A. This is simple for you to test yourself - try it out!
I'm currently running in Confirm mode - what would a confirm mean after a publish here?
I'm not really sure what purpose your intermediate exchanges (B and C) are serving here. If you remove them, you can be certain that publisher confirms will work as you expect. With the intermediate exchanges ... I can't say off the top of my head. This would be an excellent question for the rabbitmq-users mailing list.
Related
In RabbitMQ, If I have an exchange and two queues bound to it with same routing key for example "rk1*", how many copies of the message will be physically stored in Rabbit?
More generally, does Rabbit store separate copy of message for each queue?
It really depends on what kind of exchange you have? Do you have a direct exchange, a topic exchange, or a fanout exchange.
Here is more on RabbitMq architecture.
Here is some working demo code for setting up a fanout exchange.
I inspected the file system and I found that message is copied, at least on disk, for every queue.
I've created Durable Topic exchange and 2 queues bound to it with same routing key.
This is how directory structure looks like:
When I publish message to the exchange, I can see it's text in the .idx file for each queue:
So I guess this answers my question. Hope it helps.
Assume I have a mirrored queue deployed over multiple nodes (f.e. 1 master + 1 mirror). I can define the number of mirrors I want but is it possible to only accept a producer message when the message is stored at least on 2 queues (master + mirror). Otherwise it is still possible to loose a message when the master node fails before the message is mirrored.
So the mirroring activity should be part of the transaction.
You should use Publisher Confirms. When this is enabled, and your publisher has received confirmation, you can be certain that your message has been replicated to all queue mirrors.
Searching Google for site:rabbitmq.com high availability returns this document which mentions Publisher Confirms here.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I've been trying to find a way to CC messages from a Qpid Exchange to another Queue for testing/monitoring purposes. I noticed that a RabbitMQ user out there was having a similar problem, and the solution seemed to be RabbitMQ's Firehose feature. Is there a similar solution in Qpid?
Here's some more details for the curious.
Let's call the Exchange "App.Ex", and through it are flowing messages for a single other intended recipient (let's call him "Bob")
I connect to App.Ex, initiate a session, start a receiver, and start fetching (using code adapted from the QPID Book's "A Simple Messaging Program in Python")
I start seeing the messages I want to see. HOWEVER, in doing so I've robbed Bob of the messages he needs!
So there's the rub, how can I get the messages CC'd to me, but in a way that Bob still receives his messages?
I have permission to modify the messaging configuration, so I can create my own Queues and Exchanges if need be. Thoughts appreciated!
A direct exchange is probably most appropriate because you can have some queues with CC like behavior and some without, and you can change it anytime on a live exchange.
You can have two queues bound to the same subject/routing key. When a message is sent to the exchange with that particular subject/routing key, both bound queues will receives copies of the same message.
Both queues bar1 and bar2 are bound to routing_key foo. When producer B posts messages to the exchange with routing_key = foo, both bar1 and bar1 receive copies of all messages.
Ask if you need commands for creating the exchange and appropriate bindings.
However there are more ways to do the same thing:
You can also achieve the similar behavior using a topic queue, with exact matches on topic name
Lastly, you can also use a fanout exchange, where any message you send to the queue, a copy is routed to all the queues bound to the exchange.
Note that all of these exchange types are from the AMQP spec, so they are not qpid specific, you could do the same thing or something very similar in any AMQP implementation.
I setup two nodes, A and B. Both have RabbitMQ with the federation plugin installed.
From the Web UI, I can see the "Federation Status" > "State" is "running" on A and B.
On A, I created a queue called "test1".
On B, I can see the "test1" queue (replicated from A).
On A, I added a message.
However, the message does not appear in the replicated queue on B - the message stays on A.
This is the policy I used on A and B:
rabbitmqctl set_policy --apply-to exchanges my-queue "test1" \
'{"federation-upstream-set":"all"}'
So, it's like this: A (upstream) -> B (downstream) and B (upstream) -> A (downstream)
Am I supposed to see messages replicated to both A and B? Did I misconfigure the directions?
However, the message does not appear in the replicated queue on B - the message stays on A.
TL;DR: federated exchange != federated queue.
References:
https://www.rabbitmq.com/federated-exchanges.html
https://www.rabbitmq.com/federated-queues.html
The "How it works" section on federated queues explains:
" The federated queue will only retrieve messages when it has run out of messages locally, it has consumers that need messages, and the upstream queue has "spare" messages that are not being consumed ... "
Whereas the "What does a federated exchange do?" explains:
" ... messages published to the upstream exchanges are copied to the federated exchange, as though they were published directly to it ... "
recap:
if you use a federated queue,
you would need a consumer on the B side that needs messages (pull model?).
if you use a federated exchange,
messages a copied directly (push model?).
Use cases
Redundancy / Backups
Federated exchanges copy messages (max-hops copies) so they can be used for redundancy.
E.g.
here is my data, back it up.
Content distribution network
Federated exchanges copy messages (max-hops copies) so they can be used to distribute content across regions (that's also redundancy btw) provided you configure the topology correctly.
E.g.
hey everybody, please apply this security patch, which you can find at your nearest broker.
Load balancing
Federated queues can be used for load balancing: if a message is available upstream and there is no consumer there to process it, a free consumer downstream is able to receive the message and work on it. Rock on.
E.g.
I'm a computer, and I feel bored, can I help you? Any job you need me to do?
double-whammy
Federated exchange + federated queues = you can distribute the same set of tasks to multiple regions (cluster), and one worker in each cluster can perform the job.
E.g.
It's end of the quarter, I need performance metrics for each region (cluster), each store manager (one node in cluster) will aggregate metrics (inside cluster), and we'll give gift cards to the top 3.
I have trouble understanding the routing in RabbitMQ. Consider I have several producers (let call them clients) that produce messages to the queue. E.g., clients A, B, and C send messages to queue X1.
Let the consumer respond to all messages sending responses back to the queue. E.g., consumer gets message from queue X1, does something, and sends responses to the queue X1.
How can, client A determine where are in the queue X1 messages sent to it and where are messages sent to clients B or C?
I can't declare one queue per connection because of large number of connections expected (~10^6). So I'm in trouble here. Any suggestions? Thanks.
I think you need to look at the RPC tutorial. From your description it sounds like that is what you want to do. However that would probably require you to declare more queues than you want.
Approaching this a different way. I cannot understand why you would send a reply back to the producer not only by the same exchange but the same queue that the consumers are consuming from.
Would it not make sense to have producers P1,P2 and P3 send to exchange X1 with routing key "abc.aaa.xyz" / "abc.bbb.xyz" / "abc.ccc.xyz". Then have queues Q1, Q2 and Q3 bound to X1 with binding keys ".aaa." / ".bbb." / ".ccc." or just Q1 with binding key "abc.*.xyz" (I am unclear on exactly what you want so just making some suggestions). Which are consumed by Consumers C1, C2 and C3
When the Consumer has finished processing the message then it will send a message to X2, with routing key that identifies itself. The producers will consume from queues bound to X2.
The point I am trying to make is that you do not want more than one consumer reading from a queue. There is only one case in which you want that and that is a task queue. I am not clear on your use case so you may want a task queue. If you do then you should still not have your producers reading from the same task queue as your consumers. Aside from task queues you should have one consumer read from one queue. You may have many queues to one exchange and even many bindings from one queue to one exchange.
I hope this helps