Mule jms consumers are picking message slowly - activemq

Question

I m implementing mule ESB 3.9 to consume messages from active mq.
I have around 15 domain components to which it actually orchestrates to reach the final destination
I have configured a connection pool for jms and am using numberofconsumers as 32 and maximum thread in reciever as 500.
But what I see is until 4 messages per second the requests sent by jmeter is fine. But if I increase to 5 msgs per second then I find the request sent by jms averages to around 3 msgs per second slowly..
Any thought on how to configure the number of consumers and reciever thread ?
My target is 10 messages per second and also all my 15 components takes only 30 milli seconds each at max to process .so processing time by those components is good...yet I see the total throughput is around 3 seconds at times and mostly its because messages are not being picked immediately.
Please suggest
Activemq xml
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see:
http://activemq.apache.org/slow-consumer-handling.html
-->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="10000"/>
</pendingMessageLimitStrategy>
<slowConsumerStrategy>
<abortSlowConsumerStrategy/>
</slowConsumerStrategy>
</policyEntry>
<!--Mahesh added start-->
<policyEntry queue=">" >
<slowConsumerStrategy>
<abortSlowConsumerStrategy/>
</slowConsumerStrategy>
</policyEntry>
<!--Mahesh added end-->
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="90" />
</memoryUsage>
<storeUsage>
<storeUsage limit="5 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="5 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
Other information on Mule ESB
I am using a prefetch of 1 message
No of consumers are 32 and receiver threads are 500
mule-config ---->
<spring:beans>
<context:component-scan base-package="com.test.esb.orm.db" />
<spring:import resource="classpath*:applicationContext.xml" />
<spring:import resource="classpath*:test-orm-beans.xml" />
<spring:import resource="mule-domain-spring.xml"/>
</spring:beans>
<db:generic-config name="Generic_Database_Configuration"
dataSource-ref="testDataSource"
doc:name="Generic Database Configuration"/>
<db:generic-config name="Generic_TrackTraceDatabase_Configuration"
dataSource-ref="testTrackTraceDataSource"
doc:name="Generic Database Configuration for Track Trace"/>
<!-- ********************* ESB Gateway Configurations ******************************** -->
<http:listener-config name="test-Shared-http-listener" host="${domain.gateway.host}" port="${domain.gateway.port}" doc:name="HTTP Listener connector for ESB Gateway (From ui)">
<http:worker-threading-profile maxThreadsActive="64"
poolExhaustedAction="WAIT"
threadWaitTimeout="30000" />
</http:listener-config>
<jms:activemq-connector name="com.test.esb.trans.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
numberOfConsumers="16" acknowledgementMode="AUTO_ACKNOWLEDGE"
validateConnections="true"
persistentDelivery="true"
doc:name="AMQ Connector For Domain Routing Services"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<dispatcher-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
<jms:activemq-connector name="com.test.esb.domain.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
numberOfConsumers="16" acknowledgementMode="AUTO_ACKNOWLEDGE"
validateConnections="true"
persistentDelivery="false"
doc:name="AMQ Connector For Domain Routing Services"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<dispatcher-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
<jms:activemq-connector name="com.test.esb.domain.orchestrator.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
numberOfConsumers="16" acknowledgementMode="AUTO_ACKNOWLEDGE"
validateConnections="true"
persistentDelivery="true"
doc:name="AMQ Connector For Domain Routing Services"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<dispatcher-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
<jms:activemq-connector name="com.test.esb.interface2.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
validateConnections="true"
numberOfConcurrentTransactedReceivers="16"
persistentDelivery="false"
doc:name="AMQ Connector for interface2 Interface"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT" threadWaitTimeout="30000"/>
<dispatcher-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
<jms:activemq-connector name="com.test.esb.destination.router.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
validateConnections="true"
numberOfConcurrentTransactedReceivers="8"
persistentDelivery="false"
doc:name="AMQ Connector for DestinationRouter"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<dispatcher-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
<jms:activemq-connector name="com.test.esb.dlq.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
validateConnections="true"
numberOfConcurrentTransactedReceivers="16"
persistentDelivery="false"
doc:name="AMQ Connector for Dead Letter Queue"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT" threadWaitTimeout="30000"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
mule-spring config---->
<spring:bean id="domainRedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<spring:property name="maximumRedeliveries" value="5" />
<spring:property name="initialRedeliveryDelay" value="500" />
<spring:property name="maximumRedeliveryDelay" value="10000" />
<spring:property name="useExponentialBackOff" value="false" />
<spring:property name="backOffMultiplier" value="3" />
</spring:bean>
<!-- ActiveMQ Connection factory -->
<spring:bean id="domainConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="tcp://192.0.0.0:61616?jms.prefetchPolicy.all=1" />
<spring:property name="redeliveryPolicy" ref="domainRedeliveryPolicy" />
</spring:bean>
<!-- amqExceptionConnectionFactory Connection factory -->
<spring:bean id="amqExceptionConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="failover:(${domain.amq.failover.url}?jms.prefetchPolicy.all=50)" />
<spring:property name="redeliveryPolicy" ref="domainRedeliveryPolicy" />
</spring:bean>
<spring:bean id="pooledConnectionFactory" class="org.apache.activemq.jms.pool.PooledConnectionFactory" >
<spring:property name="connectionFactory" ref="domainConnectionFactory"/>
<spring:property name="maxConnections" value="10000" />
<spring:property name="maximumActiveSessionPerConnection" value="10000" />
</spring:bean>
<spring:bean id="trackTraceRedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<spring:property name="maximumRedeliveries" value="5" />
<spring:property name="initialRedeliveryDelay" value="500" />
<spring:property name="maximumRedeliveryDelay" value="10000" />
<spring:property name="useExponentialBackOff" value="false" />
<spring:property name="backOffMultiplier" value="3" />
</spring:bean>
<spring:bean id="trackTraceConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="failover:(${domain.amq.failover.url})" />
<!-- value="tcp://${domain.amq.host}:${domain.amq.port}" /> -->
<spring:property name="redeliveryPolicy" ref="trackTraceRedeliveryPolicy" />
</spring:bean>
<spring:bean id="interface2ConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="tcp://192.0.0.0:61616?jms.prefetchPolicy.all=1" />
<!-- value="tcp://${domain.amq.host}:${domain.amq.port}" /> -->
<spring:property name="redeliveryPolicy" ref="interface2RedeliveryPolicy" />
</spring:bean>
<spring:bean id="interface2RedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<spring:property name="maximumRedeliveries" value="5" />
<spring:property name="initialRedeliveryDelay" value="500" />
<spring:property name="maximumRedeliveryDelay" value="10000" />
<spring:property name="useExponentialBackOff" value="false" />
<spring:property name="backOffMultiplier" value="3" />
</spring:bean>
<!-- ********************* Destination Router Configurations ******************************** -->
<!-- ActiveMQ Connection factory for destination Router -->
<spring:bean id="destinationRouterConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="tcp://192.0.0.0:61616?jms.prefetchPolicy.all=1" />
<!-- value="tcp://${domain.amq.host}:${domain.amq.port}" /> -->
<spring:property name="redeliveryPolicy"
ref="destinationRouterRedeliveryPolicy" />
</spring:bean>
<spring:bean id="destinationRouterRedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<spring:property name="maximumRedeliveries" value="5" />
<spring:property name="initialRedeliveryDelay" value="500" />
<spring:property name="maximumRedeliveryDelay" value="10000" />
<spring:property name="useExponentialBackOff" value="false" />
<spring:property name="backOffMultiplier" value="3" />
</spring:bean>
<!-- ********************* Dead Letter Queue (DLQ) Configurations ******************************** -->
<!-- ActiveMQ Connection factory for Dead Letter Queue -->
<spring:bean id="dlqConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="failover:(${domain.amq.failover.url})" />
<spring:property name="redeliveryPolicy" ref="dlqRedeliveryPolicy" />
</spring:bean>
<spring:bean id="dlqRedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<spring:property name="maximumRedeliveries" value="5" />
<spring:property name="initialRedeliveryDelay" value="500" />
<spring:property name="maximumRedeliveryDelay" value="10000" />
<spring:property name="useExponentialBackOff" value="false" />
<spring:property name="backOffMultiplier" value="3" />
</spring:bean>

Answer 1

numberOfConsumers is the atribute that determines the number of client threads if the endpoing is using transactions. If the endpoint is in a transaction then you need to use numberOfConcurrentTransactedReceivers instead.
You should really remove the <dispatcher-threading-profile> and <receiver-threading-profile> configurations. 500 threads per JMS configuration is a lot for your system, it is a lot of resources if they are created, and will not help for this use case at all. You should set these values only because of some real reason.
Try to capture a thread dump when the processing is slow to analyze what are the threads doing. That should point to the root cause.
Reference: https://help.mulesoft.com/s/article/JMS-Usage-of-numberOfConsumers-and-numberOfConcurrentTransactedReceivers