ActiveMQ consume/forward messages from another ActiveMQ instance - activemq

I have two brokers A and B. If I want to forward message from A to B everything is simple. I just need network connector in A broker configured like this:
<networkConnectors>
<networkConnector staticBridge="true" userName="user" password="pass" uri="static://(tcp://B:61616)">
<staticallyIncludedDestinations>
<queue physicalName="QUEUE.TO.FORWARD.MESSAGE" />
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
I tought if I want to consume messageges from broker B from some other queue (let's name it QUEUE.TO.CONSUME) i just need do the same thing but with duplex set to true and just listen on QUEUE.TO.CONSUME on broker A like this:
<networkConnectors>
<networkConnector name="from-B-to-A" staticBridge="true" duplex="true" userName="user" password="pass" uri="static://(tcp://B:61616)">
<staticallyIncludedDestinations>
<queue physicalName="QUEUE.TO.CONSUME" />
</staticallyIncludedDestinations>
</networkConnector>
<networkConnector staticBridge="true" userName="user" password="pass" uri="static://(tcp://B:61616)">
<staticallyIncludedDestinations>
<queue physicalName="QUEUE.TO.FORWARD.MESSAGE" />
</staticallyIncludedDestinations>
</networkConnector>
</networkConnectors>
But it does not work as I expected. It seem that only every second message is forwared and the remaining are just lost. Suprisingly that creates two consumers on broker B QUEUE.TO.CONSUME and I assume that one of them consumes message without forwarding to broker A. How to create bridge on broker A that allows me consume messages from broker B without loosing messages. Creating network connector in broker B is not an option for now.
I've also tried create inbound queue bridge like this:
<jmsBridgeConnectors>
<jmsQueueConnector outboundQueueConnectionFactory="#remoteBroker" localUsername="user" localPassword="password">
<inboundQueueBridges>
<inboundQueueBridge inboundQueueName="QUEUE.TO.CONSUME" localQueueName="QUEUE.TO.CONSUME" />
</inboundQueueBridges>
</jmsQueueConnector>
</jmsBridgeConnectors>
...
</broker>
<bean id="remoteBroker" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="failover://(nio:B:61616)" />
<property name="userName" value="user" />
<property name="password" value="password" />
</bean>
This configuration creates consumer on remote broker B but it doesn't consume any messages which just hanging as enqueued and nothing happens. Broker A still doesn't receive any messages to its local queue.

Ok, I figure it out. I've just used embedded Apache Camel to define routing to remote host and it looks like this (camel.xml in conf directory):
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:camel="http://camel.apache.org/schema/spring"
xsi:schemaLocation="
http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<camelContext id="context" xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="remoteBroker:queue:QUEUE.TO.CONSUME"/>
<to uri="localBroker:queue:QUEUE.TO.CONSUME"/>
</route>
</camelContext>
<bean id="remoteBroker" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://B:61616"/>
<property name="userName" value="user"/>
<property name="password" value="password"/>
</bean>
</property>
</bean>
<bean id="localBroker" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="vm://localhost"/>
</bean>
</property>
</bean>
</beans>
where localhost i broker A. And in activemq.xml:
<import resource="camel.xml"/>

Related

Apache Ignite high data availability - partitional & backup setup

I run Apache Ignite to store large data set for computation & retrieval. For now I am trying to see if in-memory itself can address the caching problem.
I have partitioned the cache & set the backup count to 1. I believe, the data will be copied to another node to address any failure, this means, any one of the node goes down, the respective data should be available from the backup node. So, querying the cache should not be affected.
In my setup, when I shutdown one of the node, the data becomes unavailable & query to the cache returns null. Below is my ignite setup (run locally). What's the right way to configure the partition with right backup to address any node failures?
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util" xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean abstract="true" id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Set to true to enable distributed class loading for examples, default is false. -->
<property name="peerClassLoadingEnabled" value="true" />
<property name="rebalanceThreadPoolSize" value="4" />
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="mycache" />
<property name="cacheMode" value="PARTITIONED" />
<property name="backups" value="1" />
<property name="rebalanceMode" value="SYNC" />
<property name="writeSynchronizationMode" value="FULL_SYNC" />
<property name="partitionLossPolicy" value="READ_ONLY_SAFE" />
</bean>
</list>
</property>
<!-- Enable task execution events for examples. -->
<property name="includeEventTypes">
<list>
<!--Task execution events-->
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_TIMEDOUT" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_SESSION_ATTR_SET" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_REDUCED" />
<!--Cache events-->
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_READ" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED" />
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
Check that your nodes are a part of the same cluster. You should see info in the logs that the topology includes multiple servers.
Your understanding of the cache configuration is correct - the data should be available after one node goes down. Moreover, with the READ_ONLY_SAFE policy you cannot silently lose data even if the cluster doesn't have enough copies - your cache reads would start throwing errors.
Since you're getting null I'm guessing that your client just reconnected to the second server, which is not connected to the first one (and is therefore empty).

Ignite C++ client mode, Near cache

I have an ignite server running in replicated mode and many clients on same node which has near cache enabled. Now I don't find a significant performance difference when I run client with near cache and without near cache.
My understanding of near cache is that frequently used key and value would be stored on client itself, so there won't be an actual Get() call made to server. please correct me if I am wrong.
Can someone share a working near cache configuration xml.
SERVER CONFIG:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="cacheMode" value="LOCAL" />
<!-- Enable near cache to cache recently accessed data. -->
<!-- <property name="nearConfiguration">
<bean class="org.apache.ignite.configuration.NearCacheConfiguration"/>
</property> -->
<property name="nearConfiguration">
<bean class="org.apache.ignite.configuration.NearCacheConfiguration">
</bean>
</property>
</bean>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery.
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!-- <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> -->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<!-- <value>127.0.0.1:48550..48551</value> -->
<value>XXX.ZZZ.yyy.36:47500..47501</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
CLIENT CONFIG:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="cacheMode" value="LOCAL" />
<!-- Enable near cache to cache recently accessed data. -->
<!-- <property name="nearConfiguration">
<bean class="org.apache.ignite.configuration.NearCacheConfiguration"/>
</property> -->
<property name="nearConfiguration">
<bean class="org.apache.ignite.configuration.NearCacheConfiguration">
</bean>
</property>
</bean>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery.
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!-- <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> -->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<!-- <value>127.0.0.1:48550..48551</value> -->
<value>XXX.ZZZ.yyy.38:47500..47501</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
Yes, near cache improves performance by caching often used entries on node locally, but it doesn't make sense if you run all tests on single machine or JVM. Near cache allows not to go on remote node for data, but in your test everything already works locally.
Also Near cache have no sense for server nodes on REPLICATED or PARTITIONED cache, where number of backups equals or bigger than number of data nodes, because all data set already available for each node locally.
So to get performance boost you need to configure client node to use Near cache, when server nodes work on remote machines. Do not forget to warm up near cache before measuring.
Here is XML snippet for setting Near cache:
...
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<!-- Your other cache config -->
<property name="nearConfiguration">
<bean class="org.apache.ignite.configuration.NearCacheConfiguration"/>
</property>
</bean>
...

Multiple Persistence Store for Apache Ignite

I have one use case where I have to support multiple persistence store for my ignite cluster,For example Cache A1 should be primed from Database db1 and Cache B1 should be primed from database db2. can this be done?.In ignite Configuration XML I can only provide one persistence store details,
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<!-- Datasource for Persistence. -->
<bean name="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver" />
<property name="url" value="jdbc:oracle:thin:#localhost:1521:roc12c" />
<property name="username" value="test" />
<property name="password" value="test" />
</bean>
In my CacheStore implementation I can only access this Database right?.
I've not tried this, but if its similar to other bean-configured systems. You should be able to create another bean with a different name and configuration. Then in your cache configuration for A1 and B1 specify the different data sources. That being said, I'm guessing that theoretically.
It may be that you are already doing so, but I can't tell from your question. If you instead choose to implement your caches in this manner https://apacheignite.readme.io/docs/persistent-store you can definitely configure two caches to have different data sources. This is how I'm currently implementing multiple caches. In the cache store I use I specifically call out which database to go to.
Here is a cache configuration I use for mine.
<property name="cacheConfiguration">
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<!-- Set a cache name. -->
<property name="name" value="recordData"/>
<property name="rebalanceMode" value="ASYNC"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="backups" value="1"/>
<!-- Enable Off-Heap memory with max size of 10 Gigabytes (0 for unlimited). -->
<property name="memoryMode" value="OFFHEAP_TIERED"/>
<property name="offHeapMaxMemory" value="0"/>
<property name="swapEnabled" value="false"/>
<property name="cacheStoreFactory">
<bean class="javax.cache.configuration.FactoryBuilder" factory-method="factoryOf">
<constructor-arg value="com.company.util.MyDataStore"/>
</bean>
</property>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
</bean>
</property>
Cache store is configured per cache, so you just need to inject different data sources to different stores. What you showed is just a standalone data source bean, it's not even a part of IgniteConfiguration. You can have multiple data source beans with different IDs.

log4j2.xml configuration from file for mule applications in mule 3.6.2

How can I get the configuration from a specific file like '/opt/applications/app1/log/config/log4j2.xml' for a mule application in mule 3.6.2. The common way is to get the configuration from a config file log4j2.xml which is stored in the resource folder, we need to read this configuration file from other external path.
By default, Mule use its own log4j2 file for logging. To read log4j2.xml configuration file from external path, add the next beans in your file Application Context, For that specify the external file to be used in the context General of Mule.
Application Context:
<bean id="loggerContext" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetClass">
<value> org.apache.logging.log4j.LogManager</value>
</property>
<property name="targetMethod">
<value>getContext</value>
</property>
<property name="arguments">
<value>false</value>
</property>
</bean>
<bean id="loggerContext1" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetObject" ref="loggerContext" />
<property name="targetMethod">
<value>setConfigLocation</value>
</property>
<property name="arguments">
<value>${log4j.external.path}</value>
</property>
</bean>
<bean id="loggerContext2" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetObject" ref="loggerContext" />
<property name="targetMethod">
<value>reconfigure</value>
</property>
</bean>
Then you need to import this context from your file Mule flow, with:
Mule Config
<mule xmlns:https="http://www.mulesoft.org/schema/mule/https"
xmlns="http://www.mulesoft.org/schema/mule/core"
{..}
<!-- Add this: -->
<spring:beans>
<spring:import resource="classpath*:application-context.xml" />
</spring:beans>
{..}
<flow name="http-name" >
{..}
</flow>
</mule>

How to use simple-spring-memcached with AWS Auto Discovery

How to use simple-spring-memcached library (SSM) with AWS Elasti Cache Auto Discovery feature? We are using spymemcached as client.
So currently you are using spymemcached and want to add cache layer using simple spring memcached (SSM), right? If yes then please provide your current configuration for spymemcached. It should be easy to use the same configuration with SSM.
UPDATE
I've added new dedicated memcached provider in SSM that uses AWS ElastiCache Cluster Client. It is available on master branch and not released yet. If you build SSM from master or use snapshot available in this repository then you can use Auto Discovery feature.
Remove dependencies to spymemcached-provider and spymemcached instead add a new dependency:
<dependency>
<groupId>com.google.code.simple-spring-memcached</groupId>
<artifactId>aws-elasticache-provider</artifactId>
<version>3.4.1-SNAPSHOT</version>
</dependency>
Use below configuration:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context" xmlns:cache="http://www.springframework.org/schema/cache"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-3.1.xsd
http://www.springframework.org/schema/cache
http://www.springframework.org/schema/cache/spring-cache-3.1.xsd">
<cache:annotation-driven />
<bean name="cacheManager" class="com.google.code.ssm.spring.SSMCacheManager">
<property name="caches">
<set>
<bean class="com.google.code.ssm.spring.SSMCache">
<constructor-arg name="cache" index="0" ref="defaultCache" />
<!-- 5 minutes -->
<constructor-arg name="expiration" index="1" value="300" />
<!-- #CacheEvict(..., "allEntries" = true) won't work because allowClear is false,
so we won't flush accidentally all entries from memcached instance -->
<constructor-arg name="allowClear" index="2" value="false" />
</bean>
</set>
</property>
</bean>
<bean name="defaultCache" class="com.google.code.ssm.CacheFactory">
<property name="cacheName" value="defaultCache" />
<property name="cacheClientFactory">
<bean name="cacheClientFactory" class="com.google.code.ssm.providers.elasticache.MemcacheClientFactoryImpl" />
</property>
<property name="addressProvider">
<bean class="com.google.code.ssm.config.DefaultAddressProvider">
<!-- set only single address to configuration endpoint -->
<property name="address" value="mycluster.fnjyzo.cfg.use1.cache.amazonaws.com:11211" />
</bean>
</property>
<property name="configuration">
<bean class="com.google.code.ssm.providers.elasticache.ElastiCacheConfiguration">
<!-- set client mode to dynamic to enable Auto Discovery feature -->
<property name="clientMode" value="#{T(net.spy.memcached.ClientMode).Dynamic}" />
</bean>
</property>
</bean>
</beans>
And let me know if it works for you.
UPDATE 2
New Simple Spring Memcached version 3.5.0 with AWS Auto Discovery feature is available on github and central maven repository.