ActiveMQ : KahaDB and LevelDB slow disk space expansion - activemq

Has anyone observed a creeping disk space growth of their ActiveMQ KahaDB or LevelDB stores ?
We are using the following :
Windows Server 2008 SP1 (for deployment) and Windows 7 SP1 (for development)
Java 1.7
ActiveMQ 5.11.1 and 5.13.2 (tried both KahaDB and LevelDB)
ServiceMix 5.4.0, with its Karaf server hosting Camel-based Java application beans and its ActiveMQ server unused
Message rates = 200 XML messages per second coming in and about the same going out (translated into JSON by the Java application beans)
Number of queues = about 50 queues across which are shared the above messages; 2 queues in particular are handling up to half the traffic
All messages set to:
delivery persistence = false
time to live (TTL) = either 1 or 2 minutes (maximum)
KahaDB
When using ActiveMQ with a KahaDB, I have watched the db*.log files initially "leap-frog" from db-1.log to db-2.log then to db-3.log and so on, with the older db*.log files being cleaned up (deleted) as expected. Then at some point may be 10 hours later, the older db*.log files stop getting cleaned up; new db*.log files appear and the KahaDB begins to expand at about 4 GB per day. And when it eventually reaches our configured 50 GB cap, the server of course stops working.
It seems as if a small minority messages are failing to expire (in spite of being given a TTL) and thus block normal clean up. I've tried to manually delete the old db*.log files but I am blocked from doing so with a warning that those files are still in use.
The KahaDB db*.log files are "semi-readable" (!) in a text editor; i.e. I can recognise application messages but this does not tell me much about what has really happened.
LevelDB
I've recently tried using LevelDB with ActiveMQ 5.13.2, and the store still appears to expand albeit much more slowly (about 200 MB per day).
I believe the LevelDB data store uses compression; most files appear to be binary and thus quite opaque without a suitable reader/browser.
Has anyone found or implemented a tool to read/browse the non-persistent messages of a KahaDB or LevelDB message store ?
Our Configuration
Below is what our activemq.xml typically looks like :
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean>
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="XXXX" dataDirectory="${activemq.data}">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
<deadLetterStrategy>
<discarding />
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
-->
<managementContext>
<managementContext createConnector="true" connectorPort="1099"/>
</managementContext>
<!--
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
-->
<persistenceAdapter>
<levelDB directory="${activemq.data}/leveldb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<networkConnectors>
<networkConnector
uri="XXX"
networkTTL="XXX"
duplex="XXX"
prefetchSize="XXX"
userName="${networkconnector.username}"
password="${networkconnector.password}"
name="XXX" >
<excludedDestinations>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="XXX" />
<queue physicalName="XXX" />
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
<transportConnectors>
<transportConnector name="amqp" uri="amqp://0.0.0.0:XXXX?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="openwire" uri="tcp://0.0.0.0:XXXX?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="openwire-outbound" uri="tcp://0.0.0.0:XXXX?maximumConnections=1000&wireformat.maxFrameSize=104857600"/>
</transportConnectors>
<plugins>
<simpleAuthenticationPlugin>
<users>
<authenticationUser username="XXX" password="XXX" groups="XXX" />
</users>
</simpleAuthenticationPlugin>
<authorizationPlugin>
<map>
<authorizationMap>
<authorizationEntries>
<authorizationEntry topic=">" read="XXX" write="XXX" admin="XXX" />
<authorizationEntry queue=">" read="XXX" write="XXX" admin="XXX" />
<authorizationEntry queue="XXX" read="XXX" write="XXX" admin="XXX" />
<authorizationEntry queue="ActiveMQ.Advisory.>" read="XXX" write="XXX" admin="XXX" />
<authorizationEntry topic="ActiveMQ.Advisory.>" read="XXX" write="XXX" admin="XXX" />
</authorizationEntries>
<tempDestinationAuthorizationEntry>
<tempDestinationAuthorizationEntry read="tempDestinationAdmins" write="tempDestinationAdmins" admin="tempDestinationAdmins" />
</tempDestinationAuthorizationEntry>
</authorizationMap>
</map>
</authorizationPlugin>
<runtimeConfigurationPlugin checkPeriod="1000" />
</plugins>
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
<import resource="jetty.xml"/>
</beans>

Related

Can PatternLayout be used with HTTP appender in log4j2.xml?

We have a requirement to push our application logs to ELK from the Mule applications in CloudHub. For this, I am using ELK configs in the log4j HTTP appender.
Below is the log4j Config in my mule application app1:
<?xml version="1.0" encoding="utf-8"?>
<Configuration>
<!--These are some of the loggers you can enable.
There are several more you can find in the documentation.
Besides this log4j configuration, you can also use Java VM environment variables
to enable other logs like network (-Djavax.net.debug=ssl or all) and
Garbage Collector (-XX:+PrintGC). These will be append to the console, so you will
see them in the mule_ee.log file. -->
<Appenders>
<RollingFile name="file" fileName="/logs/mule/application/app1.log"
filePattern="/logs/mule/application/app1-%d{dd-MMM-yyyy}-%i.log">
<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n" />
<SizeBasedTriggeringPolicy size="10 MB" />
<DefaultRolloverStrategy max="20"/>
</RollingFile>
<Http name="ELK" url="https://<url to elk>/mule-cloudhub-logs/_doc">
<JsonLayout compact="true" eventEol="true" properties="true" />
<Property name="kbn-xsrf" value="true" />
<Property name="Content-Type" value="application/json" />
<Property name="Authorization" value="ApiKey apikey" />
<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n" />
</Http>
</Appenders>
<Loggers>
<!-- Http Logger shows wire traffic on DEBUG. -->
<!--AsyncLogger name="org.mule.service.http.impl.service.HttpMessageLogger" level="DEBUG" /-->
<AsyncLogger name="org.mule.service.http" level="WARN"/>
<AsyncLogger name="org.mule.extension.http" level="WARN"/>
<!-- Mule logger -->
<AsyncLogger name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO"/>
<AsyncRoot level="INFO">
<AppenderRef ref="file" />
<AppenderRef ref="ELK" />
</AsyncRoot>
</Loggers>
</Configuration>
Now, the output in ELK for this, comes out to be :
[MuleRuntime].uber.08: [app1].get:\v1\dummy\uri:app1-config.CPU_INTENSIVE #232321
No matter what kind of pattern I put in the PatternLayout in the HTTP appender, the output is still the same in elk.
Additionally, when am deploying the application, the deployment logs show a one line error as below :
2022-09-15 12:54:09,270 WrapperListener_start_runner ERROR appender Http has no parameter that matches element PatternLayout
Is there is an issue in my config or some other bug?
You can define only 1 layout per Appender. Your Http appender has two, the JsonLayout and PatternLayout

Mule jms consumers are picking message slowly

I m implementing mule ESB 3.9 to consume messages from active mq.
I have around 15 domain components to which it actually orchestrates to reach the final destination
I have configured a connection pool for jms and am using numberofconsumers as 32 and maximum thread in reciever as 500.
But what I see is until 4 messages per second the requests sent by jmeter is fine. But if I increase to 5 msgs per second then I find the request sent by jms averages to around 3 msgs per second slowly..
Any thought on how to configure the number of consumers and reciever thread ?
My target is 10 messages per second and also all my 15 components takes only 30 milli seconds each at max to process .so processing time by those components is good...yet I see the total throughput is around 3 seconds at times and mostly its because messages are not being picked immediately.
Please suggest
Activemq xml
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see:
http://activemq.apache.org/slow-consumer-handling.html
-->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="10000"/>
</pendingMessageLimitStrategy>
<slowConsumerStrategy>
<abortSlowConsumerStrategy/>
</slowConsumerStrategy>
</policyEntry>
<!--Mahesh added start-->
<policyEntry queue=">" >
<slowConsumerStrategy>
<abortSlowConsumerStrategy/>
</slowConsumerStrategy>
</policyEntry>
<!--Mahesh added end-->
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="90" />
</memoryUsage>
<storeUsage>
<storeUsage limit="5 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="5 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
Other information on Mule ESB
I am using a prefetch of 1 message
No of consumers are 32 and receiver threads are 500
mule-config ---->
<spring:beans>
<context:component-scan base-package="com.test.esb.orm.db" />
<spring:import resource="classpath*:applicationContext.xml" />
<spring:import resource="classpath*:test-orm-beans.xml" />
<spring:import resource="mule-domain-spring.xml"/>
</spring:beans>
<db:generic-config name="Generic_Database_Configuration"
dataSource-ref="testDataSource"
doc:name="Generic Database Configuration"/>
<db:generic-config name="Generic_TrackTraceDatabase_Configuration"
dataSource-ref="testTrackTraceDataSource"
doc:name="Generic Database Configuration for Track Trace"/>
<!-- ********************* ESB Gateway Configurations ******************************** -->
<http:listener-config name="test-Shared-http-listener" host="${domain.gateway.host}" port="${domain.gateway.port}" doc:name="HTTP Listener connector for ESB Gateway (From ui)">
<http:worker-threading-profile maxThreadsActive="64"
poolExhaustedAction="WAIT"
threadWaitTimeout="30000" />
</http:listener-config>
<jms:activemq-connector name="com.test.esb.trans.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
numberOfConsumers="16" acknowledgementMode="AUTO_ACKNOWLEDGE"
validateConnections="true"
persistentDelivery="true"
doc:name="AMQ Connector For Domain Routing Services"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<dispatcher-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
<jms:activemq-connector name="com.test.esb.domain.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
numberOfConsumers="16" acknowledgementMode="AUTO_ACKNOWLEDGE"
validateConnections="true"
persistentDelivery="false"
doc:name="AMQ Connector For Domain Routing Services"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<dispatcher-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
<jms:activemq-connector name="com.test.esb.domain.orchestrator.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
numberOfConsumers="16" acknowledgementMode="AUTO_ACKNOWLEDGE"
validateConnections="true"
persistentDelivery="true"
doc:name="AMQ Connector For Domain Routing Services"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<dispatcher-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
<jms:activemq-connector name="com.test.esb.interface2.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
validateConnections="true"
numberOfConcurrentTransactedReceivers="16"
persistentDelivery="false"
doc:name="AMQ Connector for interface2 Interface"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT" threadWaitTimeout="30000"/>
<dispatcher-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
<jms:activemq-connector name="com.test.esb.destination.router.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
validateConnections="true"
numberOfConcurrentTransactedReceivers="8"
persistentDelivery="false"
doc:name="AMQ Connector for DestinationRouter"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<dispatcher-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
<jms:activemq-connector name="com.test.esb.dlq.jmsConnector"
username="${domain.amq.user.id}" password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
validateConnections="true"
numberOfConcurrentTransactedReceivers="16"
persistentDelivery="false"
doc:name="AMQ Connector for Dead Letter Queue"
specification="1.1">
<receiver-threading-profile maxThreadsActive="500" poolExhaustedAction="WAIT" threadWaitTimeout="30000"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
mule-spring config---->
<spring:bean id="domainRedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<spring:property name="maximumRedeliveries" value="5" />
<spring:property name="initialRedeliveryDelay" value="500" />
<spring:property name="maximumRedeliveryDelay" value="10000" />
<spring:property name="useExponentialBackOff" value="false" />
<spring:property name="backOffMultiplier" value="3" />
</spring:bean>
<!-- ActiveMQ Connection factory -->
<spring:bean id="domainConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="tcp://192.0.0.0:61616?jms.prefetchPolicy.all=1" />
<spring:property name="redeliveryPolicy" ref="domainRedeliveryPolicy" />
</spring:bean>
<!-- amqExceptionConnectionFactory Connection factory -->
<spring:bean id="amqExceptionConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="failover:(${domain.amq.failover.url}?jms.prefetchPolicy.all=50)" />
<spring:property name="redeliveryPolicy" ref="domainRedeliveryPolicy" />
</spring:bean>
<spring:bean id="pooledConnectionFactory" class="org.apache.activemq.jms.pool.PooledConnectionFactory" >
<spring:property name="connectionFactory" ref="domainConnectionFactory"/>
<spring:property name="maxConnections" value="10000" />
<spring:property name="maximumActiveSessionPerConnection" value="10000" />
</spring:bean>
<spring:bean id="trackTraceRedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<spring:property name="maximumRedeliveries" value="5" />
<spring:property name="initialRedeliveryDelay" value="500" />
<spring:property name="maximumRedeliveryDelay" value="10000" />
<spring:property name="useExponentialBackOff" value="false" />
<spring:property name="backOffMultiplier" value="3" />
</spring:bean>
<spring:bean id="trackTraceConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="failover:(${domain.amq.failover.url})" />
<!-- value="tcp://${domain.amq.host}:${domain.amq.port}" /> -->
<spring:property name="redeliveryPolicy" ref="trackTraceRedeliveryPolicy" />
</spring:bean>
<spring:bean id="interface2ConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="tcp://192.0.0.0:61616?jms.prefetchPolicy.all=1" />
<!-- value="tcp://${domain.amq.host}:${domain.amq.port}" /> -->
<spring:property name="redeliveryPolicy" ref="interface2RedeliveryPolicy" />
</spring:bean>
<spring:bean id="interface2RedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<spring:property name="maximumRedeliveries" value="5" />
<spring:property name="initialRedeliveryDelay" value="500" />
<spring:property name="maximumRedeliveryDelay" value="10000" />
<spring:property name="useExponentialBackOff" value="false" />
<spring:property name="backOffMultiplier" value="3" />
</spring:bean>
<!-- ********************* Destination Router Configurations ******************************** -->
<!-- ActiveMQ Connection factory for destination Router -->
<spring:bean id="destinationRouterConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="tcp://192.0.0.0:61616?jms.prefetchPolicy.all=1" />
<!-- value="tcp://${domain.amq.host}:${domain.amq.port}" /> -->
<spring:property name="redeliveryPolicy"
ref="destinationRouterRedeliveryPolicy" />
</spring:bean>
<spring:bean id="destinationRouterRedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<spring:property name="maximumRedeliveries" value="5" />
<spring:property name="initialRedeliveryDelay" value="500" />
<spring:property name="maximumRedeliveryDelay" value="10000" />
<spring:property name="useExponentialBackOff" value="false" />
<spring:property name="backOffMultiplier" value="3" />
</spring:bean>
<!-- ********************* Dead Letter Queue (DLQ) Configurations ******************************** -->
<!-- ActiveMQ Connection factory for Dead Letter Queue -->
<spring:bean id="dlqConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL"
value="failover:(${domain.amq.failover.url})" />
<spring:property name="redeliveryPolicy" ref="dlqRedeliveryPolicy" />
</spring:bean>
<spring:bean id="dlqRedeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<spring:property name="maximumRedeliveries" value="5" />
<spring:property name="initialRedeliveryDelay" value="500" />
<spring:property name="maximumRedeliveryDelay" value="10000" />
<spring:property name="useExponentialBackOff" value="false" />
<spring:property name="backOffMultiplier" value="3" />
</spring:bean>
numberOfConsumers is the atribute that determines the number of client threads if the endpoing is using transactions. If the endpoint is in a transaction then you need to use numberOfConcurrentTransactedReceivers instead.
You should really remove the <dispatcher-threading-profile> and <receiver-threading-profile> configurations. 500 threads per JMS configuration is a lot for your system, it is a lot of resources if they are created, and will not help for this use case at all. You should set these values only because of some real reason.
Try to capture a thread dump when the processing is slow to analyze what are the threads doing. That should point to the root cause.
Reference: https://help.mulesoft.com/s/article/JMS-Usage-of-numberOfConsumers-and-numberOfConcurrentTransactedReceivers

Exception:Error loading metadata for index file: segments_2j|M|field... while running infinispan in 2 machines

I tried running infinispan in two machines and persisted the index data in one machine. When i try to run simultaneously in 2 machines with indexing and persisting(cache-store) into the database, i am getting the following exception,
Caused by: java.io.FileNotFoundException: Error loading metadata for index file: segments_2j|M|Course
at org.infinispan.lucene.impl.DirectoryImplementor.openInput(DirectoryImplementor.java:134)
at org.infinispan.lucene.impl.DirectoryLuceneV4.openInput(DirectoryLuceneV4.java:101)
at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:113)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:341)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:57)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:923)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:53)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:67)
at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.readerFactory(SharingBufferReaderProvider.java:131)
at org.hibernate.search.indexes.impl.SharingBufferReaderProvider$PerDirectoryLatestReader.<init>(SharingBufferReaderProvider.java:206)
at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:108)
... 24 more
My infinispan config file is:
infinispan-config.xml
<?xml version="1.0" encoding="UTF-8"?> <infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:7.2 http://www.infinispan.org/schemas/infinispan-config-7.2.xsd
urn:infinispan:config:store:jdbc:7.2 http://www.infinispan.org/schemas/infinispan-cachestore-jdbc-config-7.2.xsd"
xmlns="urn:infinispan:config:7.2"
xmlns:jdbc="urn:infinispan:config:store:jdbc:7.2">
<!-- *************************** -->
<!-- System-wide global settings -->
<!-- *************************** -->
<jgroups>
<!-- Note that the JGroups transport uses sensible defaults if no configuration
property is defined. See the JGroupsTransport javadocs for more flags.
jgroups-udp.xml is the default stack bundled in the Infinispan core jar: integration
and tuning are tested by Infinispan. -->
<stack-file name="default-jgroups-tcp" path="my-jgroupstcp.xml"/>
</jgroups>
<cache-container name="HibernateSearch" default-cache="default" statistics="false" shutdown-hook="DONT_REGISTER">
<transport stack="default-jgroups-tcp"/>
<!-- Duplicate domains are allowed so that multiple deployments with default configuration
of Hibernate Search applications work - if possible it would be better to use JNDI to share
the CacheManager across applications -->
<jmx duplicate-domains="true"/>
<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<replicated-cache name="LuceneIndexesMetadata" mode="ASYNC" async-marshalling="true">
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<transaction mode="NONE" />
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence passivation="false">
<jdbc:string-keyed-jdbc-store preload="true" fetch-state="true" read-only="false" purge="false">
<write-behind />
<property name="key2StringMapper">org.infinispan.lucene.LuceneKey2StringMapper</property>
<jdbc:connection-pool connection-url="jdbc:mysql://localhost:3306/hsearch"
driver="com.mysql.jdbc.Driver" username="my-username"
password="my-password"></jdbc:connection-pool>
<jdbc:string-keyed-table drop-on-exit="false" create-on-start="true" prefix="ISPN_STRING_TABLE">
<jdbc:id-column name="ID" type="VARCHAR(255)"/>
<jdbc:data-column name="DATA" type="MEDIUMBLOB"/>
<jdbc:timestamp-column name="TIMESTAMP" type="BIGINT"/>
</jdbc:string-keyed-table>
</jdbc:string-keyed-jdbc-store>
</persistence>
<indexing index="ALL"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
</replicated-cache>
<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<distributed-cache name="LuceneIndexesData" mode="ASYNC" async-marshalling="true">
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<transaction mode="NONE"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence passivation="false">
<jdbc:string-keyed-jdbc-store preload="true" fetch-state="true" read-only="false" purge="false">
<write-behind />
<property name="key2StringMapper">org.infinispan.lucene.LuceneKey2StringMapper</property>
<jdbc:connection-pool connection-url="jdbc:mysql://localhost:3306/hsearch"
driver="com.mysql.jdbc.Driver" username="my-username"
password="my-password"></jdbc:connection-pool>
<jdbc:string-keyed-table drop-on-exit="false" create-on-start="true" prefix="ISPN_STRING_TABLE">
<jdbc:id-column name="ID" type="VARCHAR(255)"/>
<jdbc:data-column name="DATA" type="MEDIUMBLOB"/>
<jdbc:timestamp-column name="TIMESTAMP" type="BIGINT"/>
</jdbc:string-keyed-table>
</jdbc:string-keyed-jdbc-store>
</persistence>
<indexing index="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
</distributed-cache>
<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<replicated-cache name="LuceneIndexesLocking" mode="ASYNC" async-marshalling="true">
<locking striping="false" acquire-timeout="10000" concurrency-level="500" write-skew="false"/>
<transaction mode="NONE"/>
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<indexing index="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true"/>
</replicated-cache>
</cache-container>
</infinispan>
My jgroup configuration file is,
my-jgroup.xml
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-3.6.xsd">
<TCP bind_addr="${jgroups.tcp.address:192.168.1.48}"
bind_port="${jgroups.tcp.port:7800}"
enable_diagnostics="false"
thread_naming_pattern="pl"
send_buf_size="640k"
sock_conn_timeout="300"
thread_pool.min_threads="${jgroups.thread_pool.min_threads:2}"
thread_pool.max_threads="${jgroups.thread_pool.max_threads:30}"
thread_pool.keep_alive_time="60000"
thread_pool.queue_enabled="false"
internal_thread_pool.min_threads="${jgroups.internal_thread_pool.min_threads:5}"
internal_thread_pool.max_threads="${jgroups.internal_thread_pool.max_threads:20}"
internal_thread_pool.keep_alive_time="60000"
internal_thread_pool.queue_enabled="true"
internal_thread_pool.queue_max_size="500"
oob_thread_pool.min_threads="${jgroups.oob_thread_pool.min_threads:20}"
oob_thread_pool.max_threads="${jgroups.oob_thread_pool.max_threads:200}"
oob_thread_pool.keep_alive_time="60000"
oob_thread_pool.queue_enabled="false"
/>
<MPING bind_addr="${jgroups.tcp.address:192.168.1.48}"
mcast_addr="${jgroups.mping.mcast_addr:228.2.4.6}"
mcast_port="${jgroups.mping.mcast_port:43366}"
ip_ttl="${jgroups.udp.ip_ttl:2}"
/>
<MERGE3 min_interval="10000"
max_interval="30000"
/>
<FD_SOCK />
<FD_ALL timeout="60000"
interval="15000"
timeout_check_interval="5000"
/>
<VERIFY_SUSPECT timeout="5000" />
<pbcast.NAKACK2 use_mcast_xmit="false"
xmit_interval="1000"
xmit_table_num_rows="50"
xmit_table_msgs_per_row="1024"
xmit_table_max_compaction_time="30000"
max_msg_batch_size="100"
resend_last_seqno="true"
/>
<UNICAST3 xmit_interval="500"
xmit_table_num_rows="50"
xmit_table_msgs_per_row="1024"
xmit_table_max_compaction_time="30000"
max_msg_batch_size="100"
conn_expiry_timeout="0"
/>
<pbcast.STABLE stability_delay="500"
desired_avg_gossip="5000"
max_bytes="1M"
/>
<pbcast.GMS print_local_addr="false"
join_timeout="15000"
/>
<MFC max_credits="2m"
min_threshold="0.40"
/>
<FRAG2 />
My persistence.xml is,
<?xml version="1.0" encoding="UTF-8" ?>
<persistence-unit name="IC" transaction-type="RESOURCE_LOCAL">
<class>com.csgsol.model.Course</class>
<properties>
<property name="javax.persistence.jdbc.driver" value="com.mysql.jdbc.Driver"/>
<property name="javax.persistence.jdbc.url" value="jdbc:mysql://192.168.1.99:3306/sampleDb"/>
<property name="javax.persistence.jdbc.user" value="my-username"/>
<property name="javax.persistence.jdbc.password" value="my-password"/>
<property name="hibernate.dialect" value="org.hibernate.dialect.MySQL5InnoDBDialect"/>
<property name="hibernate.hbm2ddl.auto" value="update"/>
<property name="hibernate.search.default.directory_provider" value="infinispan"/>
<property name="hibernate.search.default.indexmanager" value="org.infinispan.query.indexmanager.InfinispanIndexManager"/>
<property name="hibernate.search.default.indexmanager" value="near-real-time"/>
<property name="hibernate.search.default.exclusive_index_use" value="false"/>
<property name="hibernate.search.default.worker.execution" value = "async"/>
<property name="hibernate.search.lucene_version" value="LUCENE_4_10_4"/>
<property name="hibernate.search.infinispan.configuration_resourcename" value="infinispan-config.xml"/>
</properties>
</persistence-unit>
</persistence>
The versions i am using are
Infinispan - 7.2.0.Final
Lucene - 4.10.4
You should not use mode="ASYNC" for index caches. Since Infinispan 8.2.x (https://issues.jboss.org/browse/ISPN-4065) this configuration is forbidden

ActiveMQ dlqDeliveryFailureCause java.lang.Throwable: duplicate from store for queue

I have a 3 broker setup of ActiveMQ v5.14.1. The setup involves set of composite-destinations that pulls a copy of message from another queue. Following is the configuration of one of the broker -
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="brokerC"
dataDirectory="${activemq.data}"
schedulePeriodForDestinationPurge="10000"
schedulerSupport="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" gcInactiveDestinations="true" inactiveTimoutBeforeGC="30000" >
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>
<networkBridgeFilterFactory>
<conditionalNetworkBridgeFilterFactory replayWhenNoConsumers="true"/>
</networkBridgeFilterFactory>
</policyEntry>
<policyEntry topic=">" >
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!-- Added entry for network of brokers -->
<networkConnectors>
<networkConnector name="linkFromCToA"
uri="static:(tcp://xx.xxx.xx.xxx:61616)"
useVirtualDestSubs="true"/>
<networkConnector name="linkFromCToB"
uri="static:(tcp://xx.xxx.xx.xxx:61616)"
useVirtualDestSubs="true"/>
</networkConnectors>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="20 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<compositeQueue name="Q.1" forwardOnly="false">
<forwardTo>
<queue physicalName="Q.2" />
</forwardTo>
</compositeQueue>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
</broker>
These configurations are similar on all 3 of the brokers (of course except the broker url's). On dlq of all 3 brokers after some time i see the following exception -
java.lang.Throwable: duplicate from store for queue://Q.2
This exception is in the header 'dlqDeliveryFailureCause' of the message sitting in dlq. I observed that in 1 broker setup this issue never comes up. It is only when I have 2 or more network of brokers setup.
For anyone getting stuck on this issue, have a look at the following link
ActiveMQ User discussion.
Two suggestions 1. disable audit at queue level and 2. change messageTTL=2 (coz i have 3 brokers).

ActiveMQ set properties with startup parameters

Is there a way to set up some properties defined in activemq.xml (such as the shared storage) as startup parameters ?
<kahaDB directory="${data}/kahadb"/>
Running amq with -Dparam=value does not seem to work
I find it weird that this doesn't work for you, for example we are currently using the following Broker configuration
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
<property name="ignoreResourceNotFound" value="true" />
<property name="ignoreUnresolvablePlaceholders" value="true" />
<property name="locations">
<list>
<value>classpath:activemq.properties</value>
<value>file:///${user.home}/activemq.properties</value>
<value>file:///#{systemProperties['activemq.properties']}</value>
</list>
</property>
</bean>
<broker useJmx="${activemq.expose.jmx}" persistent="false"
brokerName="${activemq.brokerName}"
xmlns="http://activemq.apache.org/schema/core">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" enableAudit="false">
<networkBridgeFilterFactory>
<conditionalNetworkBridgeFilterFactory replayWhenNoConsumers="true"/>
</networkBridgeFilterFactory>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="${activemq.memoryUsage}" />
</memoryUsage>
<tempUsage>
<tempUsage limit="${activemq.tempUsage}" />
</tempUsage>
</systemUsage>
</systemUsage>
<networkConnectors>
<networkConnector
name="queues"
uri="static:(${activemq.otherBrokers})"
networkTTL="3"
decreaseNetworkConsumerPriority="true"
conduitSubscriptions="false" >
<excludedDestinations>
<topic physicalName=">" />
</excludedDestinations>
</networkConnector>
<networkConnector
name="topics"
uri="static:(${activemq.otherBrokers})"
networkTTL="3"
decreaseNetworkConsumerPriority="true"
conduitSubscriptions="true" >
<excludedDestinations>
<queue physicalName=">" />
</excludedDestinations>
</networkConnector>
</networkConnectors>
<transportConnectors>
<!-- expose a TCP transport for clients to use -->
<transportConnector
uri="${activemq.protocol}${activemq.host}:${activemq.tcp.port}"
updateClusterClients="true"
rebalanceClusterClients="true" />
<transportConnector
uri="${activemq.websocket.protocol}${activemq.websocket.host}:${activemq.websocket.port}"
updateClusterClients="true"
rebalanceClusterClients="true" />
</transportConnectors>
</broker>
</beans>
Here normally the properties are read from the given properties file, however I have often used direct -D JMV system properties to overrule those from the properties file for some tests and never encountered any issues. I guess you could try the slighly more complex syntax of #{systemProperties['property.name']} to specifically set that a given property is a system property and not from a placeholder.
One last Point: which version of ActiveMQ are you using and which version of spring is this version using? I think the whole placeholder syntax was added somewhere around spring 3.1, so really quite some time ago, but this is the only reason I could imagine why this isn't working for you.