ActiveMQ set properties with startup parameters - activemq

Is there a way to set up some properties defined in activemq.xml (such as the shared storage) as startup parameters ?
<kahaDB directory="${data}/kahadb"/>
Running amq with -Dparam=value does not seem to work

I find it weird that this doesn't work for you, for example we are currently using the following Broker configuration
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
<property name="ignoreResourceNotFound" value="true" />
<property name="ignoreUnresolvablePlaceholders" value="true" />
<property name="locations">
<list>
<value>classpath:activemq.properties</value>
<value>file:///${user.home}/activemq.properties</value>
<value>file:///#{systemProperties['activemq.properties']}</value>
</list>
</property>
</bean>
<broker useJmx="${activemq.expose.jmx}" persistent="false"
brokerName="${activemq.brokerName}"
xmlns="http://activemq.apache.org/schema/core">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" enableAudit="false">
<networkBridgeFilterFactory>
<conditionalNetworkBridgeFilterFactory replayWhenNoConsumers="true"/>
</networkBridgeFilterFactory>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="${activemq.memoryUsage}" />
</memoryUsage>
<tempUsage>
<tempUsage limit="${activemq.tempUsage}" />
</tempUsage>
</systemUsage>
</systemUsage>
<networkConnectors>
<networkConnector
name="queues"
uri="static:(${activemq.otherBrokers})"
networkTTL="3"
decreaseNetworkConsumerPriority="true"
conduitSubscriptions="false" >
<excludedDestinations>
<topic physicalName=">" />
</excludedDestinations>
</networkConnector>
<networkConnector
name="topics"
uri="static:(${activemq.otherBrokers})"
networkTTL="3"
decreaseNetworkConsumerPriority="true"
conduitSubscriptions="true" >
<excludedDestinations>
<queue physicalName=">" />
</excludedDestinations>
</networkConnector>
</networkConnectors>
<transportConnectors>
<!-- expose a TCP transport for clients to use -->
<transportConnector
uri="${activemq.protocol}${activemq.host}:${activemq.tcp.port}"
updateClusterClients="true"
rebalanceClusterClients="true" />
<transportConnector
uri="${activemq.websocket.protocol}${activemq.websocket.host}:${activemq.websocket.port}"
updateClusterClients="true"
rebalanceClusterClients="true" />
</transportConnectors>
</broker>
</beans>
Here normally the properties are read from the given properties file, however I have often used direct -D JMV system properties to overrule those from the properties file for some tests and never encountered any issues. I guess you could try the slighly more complex syntax of #{systemProperties['property.name']} to specifically set that a given property is a system property and not from a placeholder.
One last Point: which version of ActiveMQ are you using and which version of spring is this version using? I think the whole placeholder syntax was added somewhere around spring 3.1, so really quite some time ago, but this is the only reason I could imagine why this isn't working for you.

Related

ActiveMQ dlqDeliveryFailureCause java.lang.Throwable: duplicate from store for queue

I have a 3 broker setup of ActiveMQ v5.14.1. The setup involves set of composite-destinations that pulls a copy of message from another queue. Following is the configuration of one of the broker -
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="brokerC"
dataDirectory="${activemq.data}"
schedulePeriodForDestinationPurge="10000"
schedulerSupport="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" gcInactiveDestinations="true" inactiveTimoutBeforeGC="30000" >
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>
<networkBridgeFilterFactory>
<conditionalNetworkBridgeFilterFactory replayWhenNoConsumers="true"/>
</networkBridgeFilterFactory>
</policyEntry>
<policyEntry topic=">" >
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!-- Added entry for network of brokers -->
<networkConnectors>
<networkConnector name="linkFromCToA"
uri="static:(tcp://xx.xxx.xx.xxx:61616)"
useVirtualDestSubs="true"/>
<networkConnector name="linkFromCToB"
uri="static:(tcp://xx.xxx.xx.xxx:61616)"
useVirtualDestSubs="true"/>
</networkConnectors>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="20 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<compositeQueue name="Q.1" forwardOnly="false">
<forwardTo>
<queue physicalName="Q.2" />
</forwardTo>
</compositeQueue>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
</broker>
These configurations are similar on all 3 of the brokers (of course except the broker url's). On dlq of all 3 brokers after some time i see the following exception -
java.lang.Throwable: duplicate from store for queue://Q.2
This exception is in the header 'dlqDeliveryFailureCause' of the message sitting in dlq. I observed that in 1 broker setup this issue never comes up. It is only when I have 2 or more network of brokers setup.
For anyone getting stuck on this issue, have a look at the following link
ActiveMQ User discussion.
Two suggestions 1. disable audit at queue level and 2. change messageTTL=2 (coz i have 3 brokers).

ActiveMQ : KahaDB and LevelDB slow disk space expansion

Has anyone observed a creeping disk space growth of their ActiveMQ KahaDB or LevelDB stores ?
We are using the following :
Windows Server 2008 SP1 (for deployment) and Windows 7 SP1 (for development)
Java 1.7
ActiveMQ 5.11.1 and 5.13.2 (tried both KahaDB and LevelDB)
ServiceMix 5.4.0, with its Karaf server hosting Camel-based Java application beans and its ActiveMQ server unused
Message rates = 200 XML messages per second coming in and about the same going out (translated into JSON by the Java application beans)
Number of queues = about 50 queues across which are shared the above messages; 2 queues in particular are handling up to half the traffic
All messages set to:
delivery persistence = false
time to live (TTL) = either 1 or 2 minutes (maximum)
KahaDB
When using ActiveMQ with a KahaDB, I have watched the db*.log files initially "leap-frog" from db-1.log to db-2.log then to db-3.log and so on, with the older db*.log files being cleaned up (deleted) as expected. Then at some point may be 10 hours later, the older db*.log files stop getting cleaned up; new db*.log files appear and the KahaDB begins to expand at about 4 GB per day. And when it eventually reaches our configured 50 GB cap, the server of course stops working.
It seems as if a small minority messages are failing to expire (in spite of being given a TTL) and thus block normal clean up. I've tried to manually delete the old db*.log files but I am blocked from doing so with a warning that those files are still in use.
The KahaDB db*.log files are "semi-readable" (!) in a text editor; i.e. I can recognise application messages but this does not tell me much about what has really happened.
LevelDB
I've recently tried using LevelDB with ActiveMQ 5.13.2, and the store still appears to expand albeit much more slowly (about 200 MB per day).
I believe the LevelDB data store uses compression; most files appear to be binary and thus quite opaque without a suitable reader/browser.
Has anyone found or implemented a tool to read/browse the non-persistent messages of a KahaDB or LevelDB message store ?
Our Configuration
Below is what our activemq.xml typically looks like :
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean>
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="XXXX" dataDirectory="${activemq.data}">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
<deadLetterStrategy>
<discarding />
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
-->
<managementContext>
<managementContext createConnector="true" connectorPort="1099"/>
</managementContext>
<!--
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
-->
<persistenceAdapter>
<levelDB directory="${activemq.data}/leveldb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<networkConnectors>
<networkConnector
uri="XXX"
networkTTL="XXX"
duplex="XXX"
prefetchSize="XXX"
userName="${networkconnector.username}"
password="${networkconnector.password}"
name="XXX" >
<excludedDestinations>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="XXX" />
<queue physicalName="XXX" />
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
<transportConnectors>
<transportConnector name="amqp" uri="amqp://0.0.0.0:XXXX?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="openwire" uri="tcp://0.0.0.0:XXXX?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="openwire-outbound" uri="tcp://0.0.0.0:XXXX?maximumConnections=1000&wireformat.maxFrameSize=104857600"/>
</transportConnectors>
<plugins>
<simpleAuthenticationPlugin>
<users>
<authenticationUser username="XXX" password="XXX" groups="XXX" />
</users>
</simpleAuthenticationPlugin>
<authorizationPlugin>
<map>
<authorizationMap>
<authorizationEntries>
<authorizationEntry topic=">" read="XXX" write="XXX" admin="XXX" />
<authorizationEntry queue=">" read="XXX" write="XXX" admin="XXX" />
<authorizationEntry queue="XXX" read="XXX" write="XXX" admin="XXX" />
<authorizationEntry queue="ActiveMQ.Advisory.>" read="XXX" write="XXX" admin="XXX" />
<authorizationEntry topic="ActiveMQ.Advisory.>" read="XXX" write="XXX" admin="XXX" />
</authorizationEntries>
<tempDestinationAuthorizationEntry>
<tempDestinationAuthorizationEntry read="tempDestinationAdmins" write="tempDestinationAdmins" admin="tempDestinationAdmins" />
</tempDestinationAuthorizationEntry>
</authorizationMap>
</map>
</authorizationPlugin>
<runtimeConfigurationPlugin checkPeriod="1000" />
</plugins>
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
<import resource="jetty.xml"/>
</beans>

How to call many databases without hard-coding?

I have following scenario:
There are many sites/departments and every site has it's own ServiceMix and different number of databases. Those databases are grouped, for example there are two groups: production_one and production_two.
On every site I deployed bundles as a datasource, one datasource for every database.
Datasource: blueprint.xml
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:cxf="http://camel.apache.org/schema/blueprint/cxf"
xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
<!-- Properties -->
<!-- blueprint property placeholders, that will use etc/datasources.cfg as the properties file -->
<cm:property-placeholder id="db.placeholder" persistent-id="datasources">
</cm:property-placeholder>
<!-- Datasource -->
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver" />
<property name="url"
value="jdbc:sqlserver://${bundle_ds_1_db_host}:${bundle_ds_1_db_port};databaseName=${bundle_ds_1_db_database};" />
<property name="username" value="${bundle_ds_1_db_username}" />
<property name="password" value="${bundle_ds_1_db_password}" />
</bean>
<!-- Init database-->
<bean id="initDatabase"
class="com.xyz.mssql.ds.DatabaseInitBean"
init-method="create" destroy-method="destroy">
<property name="ds" ref="dataSource" />
</bean>
<service ref="dataSource" interface="javax.sql.DataSource">
<service-properties>
<entry key="datasource.name" value="${bundle_ds_1_ds_name}" />
<entry key="datasource.type" value="${bundle_ds_1_ds_type}" />
<entry key="datasource.id" value="${bundle_ds_1_ds_id}" />
<entry key="datasource.group_id" value="${bundle_ds_1_ds_group_id}" />
</service-properties>
</service>
There is also bundle, which receives a SOAP message via cxf and process it by calling database:
CXF: blueprint.xml
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:cxf="http://camel.apache.org/schema/blueprint/cxf"
xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint/cxf http://camel.apache.org/schema/blueprint/cxf/camel-cxf.xsd
http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
<cxf:cxfEndpoint id="processXEndpoint" address="/mssql-soap1/"
serviceClass="com.xyz.mssql.soap1.ProcessXService" />
<bean id="saveDataProcessor"
class="com.xyz.mssql.soap1.SaveDataProcessor">
<property name="ds" ref="dataSource" />
</bean>
<bean id="getDataProcessor"
class="com.xyz.mssql.soap1.GetDataProcessor">
<property name="ds" ref="dataSource" />
</bean>
<!-- DataSource -->
<reference id="dataSource" interface="javax.sql.DataSource"
filter="(datasource.name=mssql-ds-1)">
<reference-listener bind-method="onBind"
unbind-method="onUnbind">
<bean class="com.xyz.mssql.soap1.ListenerBean" />
</reference-listener>
</reference>
<!-- Camel Context -->
<camelContext xmlns="http://camel.apache.org/schema/blueprint">
<route id="cxf">
<from uri="cxf:bean:processXEndpoint" />
<recipientList>
<simple>direct:${header.operationName}</simple>
</recipientList>
</route>
<route id="saveRoute">
<from uri="direct:saveData" />
<log message="Calling saveData" />
<process ref="saveDataProcessor" />
</route>
<route id="getRoute">
<from uri="direct:getData" />
<log message="Call getData" />
<process ref="getDataProcessor" />
</route>
</camelContext>
</blueprint>
The problem is, that following code inject to bean (ex: SaveDataProcessor) only one datasource resolved by it's name "datasource.name=mssql-ds-1", but I need to call many databases and on every servicemix installation there will be different number of them.
I need to call all databases (datasources) which have same type, for example "datasource.type=production_one".
Is it possible? How to do it?
How to pass to bean an array of datasources, or call this bean against every datasource which meet same conditions (name, type and so on).

How to Setup SSL for Camel Route using the embedded Active MQ Broker in a service mix?

I am trying to use blueprint XML for the setup
My Blueprint XML looks like this
<?xml version="1.0" encoding="UTF-8"?>
<blueprint
xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0
http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">
<camelContext xmlns="http://camel.apache.org/schema/blueprint">
<route>
<from uri="file:activemq/input"/>
<to uri="file:activemq/output"/>
<setBody>
<simple>
FileMovedEvent(file: ${file:name}, timestamp: ${date:now:hh:MM:ss.SSS})
</simple>
</setBody>
<to uri="activemqs:queue:Main.Queue.Vibhav"/>
</route>
</camelContext>
<bean id="activemqConnectionFactory" class="org.apache.activemq.ActiveMQSslConnectionFactory">
<property name="brokerURL" value="ssl://localhost:61616" />
<property name="keyStore" value="file:C:/Users/xxx/Desktop/apache-servicemix-4.5.3/etc/myclient.ks"/>
<property name="keyStorePassword" value="test123"/>
<property name="trustStore" value = "file:C:/Users/xxx/Desktop/apache-servicemix-4.5.3/etc/myclient.ts"/>
</bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
<property name="maxConnections" value="8" />
<property name="connectionFactory" ref="activemqConnectionFactory" />
</bean>
<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory"/>
<property name="transacted" value="false"/>
<property name="concurrentConsumers" value="10"/>
</bean>
<bean id="activemqs" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig"/>
</bean>
</blueprint>
Note: In the broker I have already added the ssl context and the ssl connectors, also the plugin which has ssl certificate with the jaas realm.
I fixed this issue by setting the trust store password in the connection factory properties.

How to configure heartbeat for spring-rabbitmq

How can I configure spring-rabbitmq connection-factory
<rabbit:connection-factory id="connectionFactory" host="${rabbitmq.host}" port="${rabbitmq.port:5672}"
username="guest" password="guest"/>
With a requested heartbeat ?
You can provide the underlying connection factory as a bean, properties set on the rabbit:connection-factory will be overridden.
<rabbit:connection-factory id="connectionFactory" host="${rabbitmq.host}" port="${rabbitmq.port:5672}"
username="guest" password="guest" connection-factory="rcf" />
<bean id="rcf" class="com.rabbitmq.client.ConnectionFactory">
<property name="host" value="${rabbitmq.host}"/>
<property name="requestedHeartbeat" value="10" />
</bean>
You should use requestedHeartbeat property for connectionfacory bean
<bean id="rabbitconnectionFactory" class="com.rabbitmq.client.ConnectionFactory">
<property name="host" value="localhost"/>
<property name="requestedHeartbeat" value="100" />
</bean>