I have a WCF service with the following settings:
Binding = WebHttpBinding
InstanceContextMode = Single
ConcurrencyMode = Multiple
MaxConcurrentSessions = a high value
The documentation states about MaxConcurrentCalls: the MaxConcurrentCalls property specifies the maximum number of messages actively processing across a ServiceHost object. Each channel can have one pending message that does not count against the value of MaxConcurrentCalls until begins to process it.
Several questions:
What does the sentence "Each channel can have one pending message that does not count against the value of MaxConcurrentCalls until begins to process it" exactly mean?
If the MaxConcurrentCalls tresshold is reached, are new TCP connections queued?
If the MaxConcurrentCalls tresshold is reached, are new requests on existing TCP connections queued (during pipe-lining)?
How to specify the length of those queues?
Thanks!
Rene
Related
I'm trying to implement exponential backoff with a RabbitMQ headers exchange, and I had each queue be bound with x-match: "all" and x-retry-count: [RETRY COUNT FOR THIS LEVEL]. However, what I found was that if I try to retry a task and I have backoff queues for 100, 200, 400, and 800 millisecond wait time, each task I send to the retry exchange somehow matches every queue.
As you can see in the picture below, for the 200ms backoff queue, I'm binding the header x-retry-count: 2, but a task with the header x-retry-count: 1 is matching it (and the x-retry-count values for all other queues in the backoff exchange too). Why would that be?
Found what was going on. x-retry-count doesn't count as a header that can be matched on because it starts with x-; naming the header retry-count does work
I have the following scenario:
There are 3 rabbitmq queues to which producers push their messages based on the priority of the message.(myqueue_high, myqueue_medium, myqueue_low)
I want to have a single consumer which can pull from these queues in order or priority i.e. it keeps pulling from high queue as long as messages are there. o/w it pulls from medium. If medium is also empty it pulls from low.
How do i achieve this? Do i need to write a custom component?
It would be easier to put all the messages to one queue but with different priorities. That way, the priority sorting would be done in the broker and the Camel consumer would get the messages already sorted by priority. However, RabbitMQ implements the FIFO principle and does not support priority handling (yet).
Solution 1
Camel allows you to reorganise messages based on some comparator using a Resequencer: https://camel.apache.org/resequencer.html:
from("rabbitmq://hostname[:port]/myqueue_high")
.setHeader("priority", constant(9))
.to("direct:messageProcessing");
from("rabbitmq://hostname[:port]/myqueue_medium")
.setHeader("priority", constant(5))
.to("direct:messageProcessing");
from("rabbitmq://hostname[:port]/myqueue_low")
.setHeader("priority", constant(1))
.to("direct:messageProcessing");
// sort by priority by allowing duplicates (message can have same priority)
// and use reverse ordering so 9 is first output (most important), and 0 is last
// (of course we could have set the priority the other way around, but this way
// we keep align with the JMS specification...)
// use batch mode and fire every 3th second
from("direct:messageProcessing")
.resequence(header("priority")).batch().timeout(3000).allowDuplicates().reverse()
.to("mock:result");
That way, all incoming messages are routed to the same sub route (direct:messageProcessing) where the messages are reordered according the priority header set by the incoming routes.
Solution 2
Use SEDA with a prioritization queue:
final PriorityBlockingQueueFactory<Exchange> priorityQueueFactory = new PriorityBlockingQueueFactory<Exchange>();
priorityQueueFactory.setComparator(new Comparator<Exchange>() {
#Override
public int compare(final Exchange exchange1, final Exchange exchange2) {
final Integer prio1 = (Integer) exchange1.getIn().getHeader("priority");
final Integer prio2 = (Integer) exchange2.getIn().getHeader("priority");
return -prio1.compareTo(prio2); // 9 has higher priority then 0
}
});
final SimpleRegistry registry = new SimpleRegistry();
registry.put("priorityQueueFactory", priorityQueueFactory);
final ModelCamelContext context = new DefaultCamelContext(registry);
// configure and start your context here...
The route definition:
from("rabbitmq://hostname[:port]/myqueue_high")
.setHeader("priority", constant(9))
.to("seda:priority?queueFactory=#priorityQueueFactory"); // reference queue in registry
from("rabbitmq://hostname[:port]/myqueue_medium")
.setHeader("priority", constant(5))
.to("seda:priority?queueFactory=#priorityQueueFactory");
from("rabbitmq://hostname[:port]/myqueue_low")
.setHeader("priority", constant(1))
.to("seda:priority?queueFactory=#priorityQueueFactory");
from("seda:priority")
.to("direct:messageProcessing");
Solution 3
Use JMS such as Camel's ActiveMQ component instead of SEDA if you need persistence in case of failures. Just forward the incoming messages from RabbitMQ to a JMS destination with setting the JMSPriority header.
Solution 4
Skip the RabbitMQ entirely and just use a JMS broker such as ActiveMQ that supports prioritization.
Up until now, my RabbitMQ consumer clients have used a prefetch value of 1. I'm looking to increase the value in order to gain performance. If I set the value to 2, will the RabbitMQ server send each consumer 2 messages at once such that I will need to parse the two messages and store the second one in a List until the first is processed and acknowledged? Or will the API handle this behind the scenes?
I'm using the Java AMQP client library:
ConnectionFactory factory = new ConnectionFactory();
...
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
channel.basicQos(2);
QueueingConsumer consumer = new QueueingConsumer(channel);
channel.basicConsume(CONSUME_QUEUE_NAME, false, consumer);
while (!Thread.currentThread().isInterrupted()) {
try {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String m = new String(delivery.getBody(), "UTF-8");
// Will m contain two messages? Will I have to each message and keep track of them within a List?
...
}
The api handles this behind the scenes, so there are no worries there for you.
Regarding which message gets where, RMQ will just deliver by using round robin, that is if you have the queue: 1 2 3 4 5 6 and consumer1 and consumer2.
consumer1 will have 1 3 5
consumer2 will have 2 4 6
Should the connection die to any of your consumers the prefetched messages will be redelivered to the active consumers using the same delivery method.
This should be interesting reading and a good starting point to figure more exactly what happens:
Tutorial no.2 which I'm sure you've read
Reliability
The api internally queue messages in a blocking queue.
Setting the prefetch count more than 1 is actually a good idea since your worker need not wait for each and every message to arrive. It can read up to N messages (where N is the prefetch count). It can start working on a message as soon as it has finished the previous one.
Also, you have the option to acknowledge multiple messages at once instead of acknowledging individually.
channel.basicAck(lastDeliveryTag, true);
boolean true indicates to acknowledge all the messages upto and including the supplied lastDeliveryTag
I need to set message priority so that High priority messages are consumed before Low priority messages by Receivers.
First I tried with message.setJMSPriority() method to set the priority but it was not working in HornetQ and ActiveMQ so finally I set the priority of the Message Producer using setPriority() method and it works fine now.
Why isn't Messsge.setJMSPriority() working in any of the JMS vendor implementations and why do we need to set the priority of the Producer not the message itself to set the priority of the message? What is the use of Messsge.setJMSPriority() method then?
Any suggestion or comment is appreciated.
To answer this question all you need to do is read the API docs for the setJMSPriority method and it tells you why. Here's the relevant text.
Sets the priority level for this message.
JMS providers set this field when a message is sent. This method can be used to change the value for a message that has been received.
The JMS Provider (ActiveMQ, HornetMQ, etc) set the priority in the producer on send to either the default value of 4, or to whatever value you've set the producer to use, so setting the value before send on the message itself won't have any effect.
The following will not work:
msg.setJMSPriority(9); // Not working!
In this code, the message priority is set to 9, indicating this is a high-priority message.
However, when the message is sent, the message will have a priority of 4 (normal
priority). The reason? Like the message expiration, the JMS provider will look at the
message priority property on the sender, or on the send(..) invocation, and then invoke the setJMSPriority on the message method prior
to placing the message on the queue. Since the default message priority is 4 (normal
priority), the message priority will not be set to a high priority message, as the developer had originally intended.
Like the message expiration, there are two ways of setting the message priority: you
can invoke the setPriority() method on the MessageProducer (QueueSender or Topic
Publisher) or set the message priority when sending the message:
//set the default message priority for all messages to 9 (high)
QueueSender qSender = qSession.createSender(requestQ);
qSender.setPriority(9);
qSender.send(msg1);
//this message is low priority
qSender.send(msg2, DeliveryMode.PERSISTENT, 1, 30000);
In this example, msg1 will be sent with a priority of 9 (high priority), whereas msg2 will
be sent with a priority of 1 (low priority).
This is a JMS Specification requirement.
You should change the priority on the Message Producer.
You can read JmsTemplate http://static.springsource.org/spring/docs/3.0.6.RELEASE/spring-framework-reference/html/jms.html
Some JMS providers allow the setting of default QOS values administratively through the configuration of the ConnectionFactory.
Check isExplicitQosEnabled property.
Want to know the behavior of rabbitmq multiple publisher and consumer.
Does rabbitmq server gives one message to any one of the consumer at a time and other consumers are ideal at that time?
OR
Consumers pick any unattended message from queue, so that at a time, more than one consumers are consuming the message from queue?
Basically I am designing a database queue and do not want more than one inserts at a time.
A message from the queue will be delivered to one consumer only. Ie: once the message makes its way to the queue - it won't be copied (broadcasted) to multiple consumers.
If you want to do broadcast - you have to use multiple queues.
See this tutorial for more details:
http://www.rabbitmq.com/tutorial-two-python.html
yes , RabitMQ supports multiple publisher and consumer.
Multiple Publisher
For publishing a messsge to rabbitmqyou need to declare a factory and do a connection to the rabbitmq server.
then decare a chennel to rabbitmq
ConnectionFactory FACTORY = new ConnectionFactory
FACTORY.setUsername ("guest")
FACTORY.setPassword ("guest")
FACTORY.setVirtualHost ("\")
FACTORY.setPort (5572)
FACTORY.setHost ("localhost")
Connection connection=FACTORY.newConnection
Channel channel = connection.createChannel
the basic key to route a message is a routing key
channel.basicPublish(EXCHANGE_NAME, "Queue1", MessageProperties.PERSISTENT_TEXT_PLAIN, "msg1".getBytes)
channel.basicPublish(EXCHANGE_NAME, "Queue2", MessageProperties.PERSISTENT_TEXT_PLAIN, "msg2".getBytes)
these two messages will be published to a seperate queue as per the routing key as mention queue1 and queue2
2.Multiple Consumer
for multiple consumer we declare a queue and bind to a particular routing key
the the message to that routing key will be publishe to respected queue.
channel.exchangeDeclare(EXCHANGE_NAME, "direct", durable)
channel.queueDeclare("q1", durable, false, false, null)
channel queueBind ("q1", EXCHANGE_NAME,"queue1")// routing key = "queue1"
val q1Consumer = new QueueingConsumer(channel)
channel basicConsume ("q1", false, q1Consumer)
like this u can consume messages from first queue
and same goes for second queue but specify the routing key as "queue2"
channel.exchangeDeclare(EXCHANGE_NAME, "direct", durable)
channel.queueDeclare("q2", durable, false, false, null)
channel queueBind ("q2", EXCHANGE_NAME,"queue2") // routing key = "queue2"
val q2Consumer = new QueueingConsumer(channel)
channel basicConsume ("q2", false, q2Consumer)