Suppose we have 3 nodes in a cluster.
node1,node2,node3
In node1 we have a
exchange e1 bounded to a queue q1 with binding key =key1
It is attached to a consumer1.
In node2 we have a
exchange e2 bounded to a queue q2 with binding key =key2
It is attached to a consumer2.
Can consumer2 read messages from q1 in cluster ? If not how can this be implemented ?
you can read rabbitMQ route totorial.Though it's using python,the concept would be the same.In the Putting it all together part the consumer 2 can receive info,error and warning from queue 2 while the consumer 1 get error from queue 1.
In your case,c2 can't read message from queue 1 now.To implement,the exchange setting don't need to change.Just bind queue 2 with exchange 1 key 1.
Related
i am trying to implement below scenario in my application
Exachange e1 -> Queue q1
DLX exchange e2 -> Queue q2
Also i have mentioned DLE and DLK in queue-q1 then message moving to queue-q2 on rejection/failure/timeout.
But how does i resend/retry message from queue-q2 to original queue-q1?
You can do that manually in your application after some analyze and filtering logic. Or you can make some TTL on that queue-q2 to let not consumed messages to be expired. And you also need to specify in this queue a x-dead-letter-exchange as a name for the Exachange e1 for desired recycling.
See more info yin this article:
Create the dead letter exchange, which is just a normal exchange with a special name
Create a retry_message queue and have all messages published to the dead letter exchange route here
When you setup the retry_message queue, be sure to default the following parameter values of the queue
x-message-ttl: 30000 – This will set a ttl on any message published to the queue. When the ttl expires, the message will be republished to the exchange specified in the x-dead-letter-exchange parameter.
x-dead-letter-exchange: original_exchange_name – This is where the message will get republished to once the message ttl expires. We normally want this be the name of the exchange where the message was originally published.
I have producer app with 2 separate instances (p1, p2), and consumer app with 2 separate instances (c1, c2).
Producer p1 connects to exchange with topic= t1, queueName =name1.
Consumer c1 connects to exchange with topic= t1, queueName =name1.
Producer p2 connects to exchange with topic= t2, queueName =name1.
Consumer c2 connects to exchange with topic= t2, queueName =name1.
I see in RabbitMQ GUI that I have 2 exchanges but only 1 queue.
Instead that c1 will receive messages from p1 only, and c2 will receive messages from p2 only, RabbitMQ is doing round robin on messages between c1 and c2. So the messages I send from p2 are being received both by c1 and c2.
I thought that in RabbitMQ the correlation is multiple queues per exchange, and the behavior here is unexpected. Why?
You can have multiple queues for every exchange, it's true; but the routing key is a queue matter, not a consumer matter.
The routing key will be used by rabbit to send the message to the right queue; once the message is received on a topic exchange, the message will be sent to all the queues binded to that specific topic. You have only one queue here, that's why both C1 and C2 get the message.
Check this link for a clear example.
If you need to separate C1 and C2, you need to bind them to 2 different queues, not to the same one.
Two queues are bound to a topic exchange with the following routing keys:
Queue A, bound with routing key pattern match *.foo
Queue B, bound with routing key pattern match *.bar
I'd like to add a third queue to this exchange that receives messages that are neither foo messages nor bar messages. If I bind this queue with a # routing key, I naturally get all messages I need, but including foo's and bar's which I don't want.
Any way to route messages patching a pattern NOT *.foo AND NOT *.bar ?
If you want to catch all messages that doesn't match any bindings, that can be done with Alternate Exchange.
Add alternate exchange for existent one and collect all messages from that alternate exchanges:
standard workflow --> [main exchange (topic)]
| --> via binding *.foo --> [foo queue]
| --> via binding *.bar --> [bar queue]
v
[alternate exchange (let it be topic too)]
--> via binding * --> []
For more specific cases when you have N bindings but you want to catch all messages that doesn't match M bindings (where M < N) it is more problematic, but technically can be done via Dead Letter Exchange and then publish it to custom exchange where you have only M bindings, and then apply case with Alternate Exchange. But it even sounds rusty, not even think about performance degradation (applied only if you have really high messages flow).
I have the following scenario:
There are 3 rabbitmq queues to which producers push their messages based on the priority of the message.(myqueue_high, myqueue_medium, myqueue_low)
I want to have a single consumer which can pull from these queues in order or priority i.e. it keeps pulling from high queue as long as messages are there. o/w it pulls from medium. If medium is also empty it pulls from low.
How do i achieve this? Do i need to write a custom component?
It would be easier to put all the messages to one queue but with different priorities. That way, the priority sorting would be done in the broker and the Camel consumer would get the messages already sorted by priority. However, RabbitMQ implements the FIFO principle and does not support priority handling (yet).
Solution 1
Camel allows you to reorganise messages based on some comparator using a Resequencer: https://camel.apache.org/resequencer.html:
from("rabbitmq://hostname[:port]/myqueue_high")
.setHeader("priority", constant(9))
.to("direct:messageProcessing");
from("rabbitmq://hostname[:port]/myqueue_medium")
.setHeader("priority", constant(5))
.to("direct:messageProcessing");
from("rabbitmq://hostname[:port]/myqueue_low")
.setHeader("priority", constant(1))
.to("direct:messageProcessing");
// sort by priority by allowing duplicates (message can have same priority)
// and use reverse ordering so 9 is first output (most important), and 0 is last
// (of course we could have set the priority the other way around, but this way
// we keep align with the JMS specification...)
// use batch mode and fire every 3th second
from("direct:messageProcessing")
.resequence(header("priority")).batch().timeout(3000).allowDuplicates().reverse()
.to("mock:result");
That way, all incoming messages are routed to the same sub route (direct:messageProcessing) where the messages are reordered according the priority header set by the incoming routes.
Solution 2
Use SEDA with a prioritization queue:
final PriorityBlockingQueueFactory<Exchange> priorityQueueFactory = new PriorityBlockingQueueFactory<Exchange>();
priorityQueueFactory.setComparator(new Comparator<Exchange>() {
#Override
public int compare(final Exchange exchange1, final Exchange exchange2) {
final Integer prio1 = (Integer) exchange1.getIn().getHeader("priority");
final Integer prio2 = (Integer) exchange2.getIn().getHeader("priority");
return -prio1.compareTo(prio2); // 9 has higher priority then 0
}
});
final SimpleRegistry registry = new SimpleRegistry();
registry.put("priorityQueueFactory", priorityQueueFactory);
final ModelCamelContext context = new DefaultCamelContext(registry);
// configure and start your context here...
The route definition:
from("rabbitmq://hostname[:port]/myqueue_high")
.setHeader("priority", constant(9))
.to("seda:priority?queueFactory=#priorityQueueFactory"); // reference queue in registry
from("rabbitmq://hostname[:port]/myqueue_medium")
.setHeader("priority", constant(5))
.to("seda:priority?queueFactory=#priorityQueueFactory");
from("rabbitmq://hostname[:port]/myqueue_low")
.setHeader("priority", constant(1))
.to("seda:priority?queueFactory=#priorityQueueFactory");
from("seda:priority")
.to("direct:messageProcessing");
Solution 3
Use JMS such as Camel's ActiveMQ component instead of SEDA if you need persistence in case of failures. Just forward the incoming messages from RabbitMQ to a JMS destination with setting the JMSPriority header.
Solution 4
Skip the RabbitMQ entirely and just use a JMS broker such as ActiveMQ that supports prioritization.
Up until now, my RabbitMQ consumer clients have used a prefetch value of 1. I'm looking to increase the value in order to gain performance. If I set the value to 2, will the RabbitMQ server send each consumer 2 messages at once such that I will need to parse the two messages and store the second one in a List until the first is processed and acknowledged? Or will the API handle this behind the scenes?
I'm using the Java AMQP client library:
ConnectionFactory factory = new ConnectionFactory();
...
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.basicQos(2);
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume(CONSUME_QUEUE_NAME, false, consumer);
while (!Thread.currentThread().isInterrupted()) {
try {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String m = new String(delivery.getBody(), "UTF-8");
// Will m contain two messages? Will I have to each message and keep track of them within a List?
...
}
The api handles this behind the scenes, so there are no worries there for you.
Regarding which message gets where, RMQ will just deliver by using round robin, that is if you have the queue: 1 2 3 4 5 6 and consumer1 and consumer2.
consumer1 will have 1 3 5
consumer2 will have 2 4 6
Should the connection die to any of your consumers the prefetched messages will be redelivered to the active consumers using the same delivery method.
This should be interesting reading and a good starting point to figure more exactly what happens:
Tutorial no.2 which I'm sure you've read
Reliability
The api internally queue messages in a blocking queue.
Setting the prefetch count more than 1 is actually a good idea since your worker need not wait for each and every message to arrive. It can read up to N messages (where N is the prefetch count). It can start working on a message as soon as it has finished the previous one.
Also, you have the option to acknowledge multiple messages at once instead of acknowledging individually.
channel.basicAck(lastDeliveryTag, true);
boolean true indicates to acknowledge all the messages upto and including the supplied lastDeliveryTag