PropertySourcesPlaceholderConfigurer with MethodInvokingFactoryBean - unresolved value - properties

I am using PropertySourcesPlaceholderConfigurer to access a file that contains 2 values :
key1=value1
key2=value2.
<bean id="mainProperties"
class="org.springframework.context.support.PropertySourcesPlaceholderConfigurer" id="">
<property name="locations">
<list>
<value>file:datafile.properties</value>
</list>
</property>
</bean>
values in datafile.properties are then set into system properties using MethodInvokingFactoryBean.
<bean class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetObject">
<!-- System.getProperties() -->
<bean class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetClass" value="java.lang.System" />
<property name="targetMethod" value="getProperties" />
</bean>
</property>
<property name="targetMethod"
value="putAll" />
<property name="arguments">
<!-- The new Properties -->
<util:properties>
<prop key="my.key1">${key1}</prop>
<prop key="my.key2">${key2}</prop>
</util:properties>
</property>
</bean>
ISSUE - ${key1} & ${key2} are not resolved. I was expecting that these values will be resolved since i am loading datafile.properties using PropertySourcesPlaceholderConfigurer. Can someone please help ?

I am beginner in Spring. My solution is:
File datafile.properties:
key1=Hellow
key2=world.
Configuration in root-content.xml
<bean id="propertySources"
class="org.springframework.context.support.PropertySourcesPlaceholderConfigurer">
<property name="locations">
<array>
<value>/WEB-INF/jdbc.properties</value>
<value>/WEB-INF/datafile.properties</value>
</array>
</property>
</bean>
<bean id="sysProps"
class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetClass" value="java.lang.System"/>
<property name="targetMethod" value="getProperties"/>
</bean>
<bean id="myNewProps"
class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetObject" ref="sysProps"/>
<property name="targetMethod" value="putAll"/>
<property name="arguments">
<map>
<entry key="my.key1" value="${key1}"/>
<entry key="my.key2" value="${key2}"/>
</map>
</property
</bean>
In my central controller MainController.java:
#RequestMapping(value = "/", method = RequestMethod.GET)
public String home() {
//...
Properties prop = System.getProperties();
String myKey1 = prop.getProperty("my.key1");
String myKey2 = prop.getProperty("my.key2");
System.out.println("\nProperties:\n my.key1 = " + myKey1 +
"\n my.key2 = " + myKey2);
//...
}
Console output:
Properties:
my.key1 = Hellow
my.key2 = world.
Note:
When using two placeholderConfigurers, may cause exception: Could not resolve placeholder 'key1' in string value "${key1}". Each of used placeholderConfigurer must have a different placeholder.
For example:
<bean id="propertyConfigurer"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="/WEB-INF/jdbc.properties"/>
</bean>
<!-- default placeholder ${ } -->
<bean id="propertySources"
class="org.springframework.context.support.PropertySourcesPlaceholderConfigurer">
<property name="locations">
<array>
<value>file:datafile.properties</value>
</array>
</property>
<property name="placeholderPrefix" value="#["></property>
<property name="placeholderSuffix" value="]"></property>
</bean>
<!-- placeholder #[ ] for datafile.properties -->

Related

How to add a new node into Ignite cluster?

We’ve an ignite instance(Gridgain) running in the server and would like to add one more node into the same cluster. I used the same config as the current instance but nothing happening after a long wait.I tested the connection with the server by creating a thick client and works well, So I am not sure what is going on?. Please see the screen below
Running server config
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="rho-cache"/>
<property name="consistentId" value="rho-cache"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<!-- Set the size of wal segments to 128MB -->
<property name="walSegmentSize" value="#{128 * 1024 * 1024}"/>
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<!-- Enable perisistence -->
<property name="persistenceEnabled" value="true"/>
</bean>
</property>
<property name="storagePath" value="work"/>
</bean>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>10.1.8.186</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
New Node Config
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="rho-1-cache"/>
<property name="consistentId" value="rho-1-cache"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<!-- Set the size of wal segments to 128MB -->
<property name="walSegmentSize" value="#{128 * 1024 * 1024}"/>
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<!-- Enable perisistence -->
<property name="persistenceEnabled" value="true"/>
</bean>
</property>
<property name="storagePath" value="work"/>
</bean>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>10.1.8.186</value>
<value>195.168.44.88</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
Screen when starting the second node

Apache Ignite Join Query Returns Incorrect Results Due to Data not Being Co-located

I'm new to Ignite and following various blogs to understand the configuration required.
As a starter, I have used "movie lens" data (Movies & Ratings) and used Ignite Web Console to generate the required configuration.
Please find the required details about my code below:
1) MySQL Table Structure
CREATE TABLE movies( movie_id INTEGER,
movie_name VARCHAR(250), genre VARCHAR(250),
CONSTRAINT pk_movie PRIMARY KEY (movie_id) );
CREATE TABLE ratings(
user_id INTEGER, movie_id INTEGER, rating FLOAT, timestamp TIMESTAMP,
CONSTRAINT pk_rating PRIMARY KEY (user_id, movie_id) );
2) Ignite Config File in Spring Boot Project (Auto Generated by Ignite Web Console)
<?xml version="1.0" encoding="UTF-8"?>
<!-- This file was generated by Ignite Web Console (05/15/2019, 18:37) -->
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<!-- Load external properties file. -->
<bean id="placeholderConfig" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:secret.properties"/>
</bean>
<!-- Data source beans will be initialized from external properties file. -->
<bean id="dsGeneric_Movielens" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="jdbcUrl" value="${dsGeneric_Movielens.jdbc.url}"/>
<property name="user" value="${dsGeneric_Movielens.jdbc.username}"/>
<property name="password" value="${dsGeneric_Movielens.jdbc.password}"/>
</bean>
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="ImportedCluster"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47510</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="MoviesCache"/>
<property name="cacheMode" value="REPLICATED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="cacheStoreFactory">
<bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
<property name="dataSourceBean" value="dsGeneric_Movielens"/>
<property name="dialect">
<bean class="org.apache.ignite.cache.store.jdbc.dialect.BasicJdbcDialect">
</bean>
</property>
<property name="types">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
<property name="cacheName" value="MoviesCache"/>
<property name="keyType" value="java.lang.Integer"/>
<property name="valueType" value="com.learnwithmanoj.ignite.model.Movies"/>
<property name="databaseSchema" value="movielens"/>
<property name="databaseTable" value="movies"/>
<property name="keyFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="movie_id"/>
<constructor-arg value="int"/>
<constructor-arg value="movieId"/>
</bean>
</list>
</property>
<property name="valueFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.VARCHAR"/>
</constructor-arg>
<constructor-arg value="movie_name"/>
<constructor-arg value="java.lang.String"/>
<constructor-arg value="movieName"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.VARCHAR"/>
</constructor-arg>
<constructor-arg value="genre"/>
<constructor-arg value="java.lang.String"/>
<constructor-arg value="genre"/>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</property>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="java.lang.Integer"/>
<property name="valueType" value="com.learnwithmanoj.ignite.model.Movies"/>
<property name="keyFieldName" value="movieId"/>
<property name="keyFields">
<list>
<value>movieId</value>
</list>
</property>
<property name="fields">
<map>
<entry key="movieName" value="java.lang.String"/>
<entry key="genre" value="java.lang.String"/>
<entry key="movieId" value="java.lang.Integer"/>
</map>
</property>
<property name="aliases">
<map>
<entry key="movieId" value="movie_id"/>
<entry key="movieName" value="movie_name"/>
</map>
</property>
</bean>
</list>
</property>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="RatingsCache"/>
<property name="cacheMode" value="REPLICATED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="cacheStoreFactory">
<bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
<property name="dataSourceBean" value="dsGeneric_Movielens"/>
<property name="dialect">
<bean class="org.apache.ignite.cache.store.jdbc.dialect.BasicJdbcDialect">
</bean>
</property>
<property name="types">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
<property name="cacheName" value="RatingsCache"/>
<property name="keyType" value="com.learnwithmanoj.ignite.model.RatingsKey"/>
<property name="valueType" value="com.learnwithmanoj.ignite.model.Ratings"/>
<property name="databaseSchema" value="movielens"/>
<property name="databaseTable" value="ratings"/>
<property name="keyFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="user_id"/>
<constructor-arg value="int"/>
<constructor-arg value="userId"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="movie_id"/>
<constructor-arg value="int"/>
<constructor-arg value="movieId"/>
</bean>
</list>
</property>
<property name="valueFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.REAL"/>
</constructor-arg>
<constructor-arg value="rating"/>
<constructor-arg value="java.lang.Double"/>
<constructor-arg value="rating"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.TIMESTAMP"/>
</constructor-arg>
<constructor-arg value="timestamp"/>
<constructor-arg value="java.sql.Timestamp"/>
<constructor-arg value="timestamp"/>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</property>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="com.learnwithmanoj.ignite.model.RatingsKey"/>
<property name="valueType" value="com.learnwithmanoj.ignite.model.Ratings"/>
<property name="keyFields">
<list>
<value>userId</value>
<value>movieId</value>
</list>
</property>
<property name="fields">
<map>
<entry key="userId" value="java.lang.Integer"/>
<entry key="movieId" value="java.lang.Integer"/>
<entry key="rating" value="java.lang.Double"/>
<entry key="timestamp" value="java.sql.Timestamp"/>
</map>
</property>
<property name="aliases">
<map>
<entry key="userId" value="user_id"/>
<entry key="movieId" value="movie_id"/>
</map>
</property>
<property name="indexes">
<list>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="fk_movie_id"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="movieId" value="false"/>
</map>
</property>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</beans>
3) Ignite Config File in Ignite Installation Folder
<?xml version="1.0" encoding="UTF-8"?>
<!-- This file was generated by Ignite Web Console (05/15/2019, 08:52) -->
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<!-- Load external properties file. -->
<bean id="placeholderConfig" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:secret.properties"/>
</bean>
<!-- Data source beans will be initialized from external properties file. -->
<bean id="datasource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="jdbcUrl" value="${datasource.jdbc.url}"/>
<property name="user" value="${datasource.jdbc.username}"/>
<property name="password" value="${datasource.jdbc.password}"/>
</bean>
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="MovieRatingsCluster"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47510</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
Problem Statement:
When I start my Spring Boot Application as a stand-alone Ignite node, then all the data from the MySQL database is getting synced into Ignite node as expected. Also, if I try to join between the Ratings and Movies table, it's joining properly and giving the correct result.
Join Query Used:
select r.movie_id, m.movie_name, count(r.rating) as count from
"RatingsCache".ratings r inner join "MoviesCache".movies m where
r.movie_id = m.movie_id group by r.movie_id order by count desc limit
5;
Now coming to the problem. If I do the same testing using more than one node, then the Join starts to give incorrect results due to the data not being colocated.
Can anyone guide me with the right set of configuration which needs to be added to fix the Join problem in Partition Mode?
I have even tried to add the below config.
<property name="cacheKeyConfiguration">
<list>
<bean class="org.apache.ignite.cache.CacheKeyConfiguration">
<property name="typeName" value="com.learnwithmanoj.ignite.model.RatingsKey" />
<property name="affinityKeyFieldName" value="movie_id" />
</bean>
</list>
</property>
But got the below error.
Caused by: org.apache.ignite.binary.BinaryObjectException: Binary type
has different affinity key fields
[typeName=com.learnwithmanoj.ignite.model.RatingsKey,
affKeyFieldName1=null, affKeyFieldName2=movie_id]
Your solution should work. First you need to clean marshaller/ directory under Ignite home dir on all nodes, then restart those nodes.
You might also need to specify key field name in caps: MOVIE_ID.

Apache Ignite configuration error

We have build a 4 node Apache Ignite Cluster and able to do connect and perform the basic operation like creating a Cache from a Java program.
But it fails to connect to the Ignite cluster when I did the MySQL integration.
Following is the error message.
Exception in thread "main" class org.apache.ignite.IgniteException: Resource field is not assignable from the resource: class org.springframework.jdbc.datasource.DriverManagerDataSource
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:906)
at org.apache.ignite.Ignition.start(Ignition.java:350)
at PersonExample.PersonStoreExample.main(PersonStoreExample.java:16)
Caused by: class org.apache.ignite.IgniteCheckedException: Resource field is not assignable from the resource: class org.springframework.jdbc.datasource.DriverManagerDataSource
at org.apache.ignite.internal.processors.resource.GridResourceUtils.inject(GridResourceUtils.java:50)
at org.apache.ignite.internal.processors.resource.GridResourceSpringBeanInjector.inject(GridResourceSpringBeanInjector.java:67)
at org.apache.ignite.internal.processors.resource.GridResourceIoc.injectInternal(GridResourceIoc.java:172)
at org.apache.ignite.internal.processors.resource.GridResourceIoc.inject(GridResourceIoc.java:97)
at org.apache.ignite.internal.processors.resource.GridResourceProcessor.injectGeneric(GridResourceProcessor.java:257)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepare(GridCacheProcessor.java:539)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepare(GridCacheProcessor.java:528)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCache(GridCacheProcessor.java:1270)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:784)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:926)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1736)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1589)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1042)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:964)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:850)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:749)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:619)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:589)
at org.apache.ignite.Ignition.start(Ignition.java:347)
... 1 more
Following is the Ignite Config file:
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<!--
Alter configuration below as needed.
-->
<bean class="org.springframework.jdbc.datasource.DriverManagerDataSource" name="dataSource">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://<<mysqk_host>>:3306/sample_db" />
<property name="username" value="root" />
<property name="password" value="hadoop" />
</bean>
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="personCache" />
<property name="readThrough" value="true" />
<property name="writeThrough" value="true" />
<property name="cacheStoreFactory">
<bean class="javax.cache.configuration.FactoryBuilder" factory-method="factoryOf">
<constructor-arg value="PersonExample.PersonStore" />
</bean>
</property>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="java.lang.Long" />
<property name="valueType" value="PersonExample.Person" />
<property name="fields">
<map>
<entry key="id" value="java.lang.Long" />
<entry key="name" value="java.lang.String" />
<entry key="orgId" value="java.lang.Long" />
<entry key="salary" value="java.lang.Integer" />
</map>
</property>
</bean>
</list>
</property>
</bean>
</list>
</property>
<!-- <property name="peerClassLoadingEnabled" value="true"></property> -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
<property name="zkConnectionString" value=“<<zk_host>>:2181" />
</bean>
</property>
</bean>
</property>
</bean>
How to resolve this error?
Error has been resolved with the following changes:
Added the latest MySQL JDBC jar in ignite Libs directory
Ensured Cluster config file and Client config files are same (Earlier client config file contained few additional configurations in cache configuration).
Added the Person class (Which is a MySQL table POJO, that will be stored in Ignite cache).

Ignite DataStreamer not loading data

I'm using ignite version 2.0.0
I am trying to load about 40M data from db to my ignite instances(2 in this case). I read about the Ignite DataStreamers from https://apacheignite.readme.io/v1.2/docs/data-streamers
Because of the very low data loading speed of loadCache() i changed loadCache() with IgniteDataStreamer.addData().
Upon execution, i notice in the web console that the metrics for on-heap entries gets incrementing(i.e., it shows that data is being loaded). But when i query the ignite cache, i get the result as empty.
Also i notice that the server logs are showcasing this exception:
[01:46:19,541][ERROR][flusher-0-#66%RemoteIgniteCluster%][GridCacheWriteBehindStore] Unable to update underlying store: CacheJdbcPojoStore []
javax.cache.CacheException: Failed to read property value from non binary object [class=class java.lang.Integer, property=class_no]
at org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStore.extractBinaryParameter(CacheJdbcPojoStore.java:122)
at org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStore.extractParameter(CacheJdbcPojoStore.java:69)
at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.fillValueParameters(CacheAbstractJdbcStore.java:1414)
at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.writeAll(CacheAbstractJdbcStore.java:1081)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.updateStore(GridCacheWriteBehindStore.java:804)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.applyBatch(GridCacheWriteBehindStore.java:720)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.access$2400(GridCacheWriteBehindStore.java:75)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.flushCacheCoalescing(GridCacheWriteBehindStore.java:1108)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.body(GridCacheWriteBehindStore.java:1006)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
[01:50:00,945][ERROR][flusher-0-#66%RemoteIgniteCluster%][GridCacheWriteBehindStore] Unable to update underlying store: CacheJdbcPojoStore []
javax.cache.integration.CacheWriterException: Failed to write entries in database
at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.writeAll(CacheAbstractJdbcStore.java:1151)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.updateStore(GridCacheWriteBehindStore.java:804)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.applyBatch(GridCacheWriteBehindStore.java:720)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.access$2400(GridCacheWriteBehindStore.java:75)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.flushCacheCoalescing(GridCacheWriteBehindStore.java:1108)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.body(GridCacheWriteBehindStore.java:1006)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
My LoadCache.java code is:
package load;
import javax.naming.event.ObjectChangeListener;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteDataStreamer;
import org.apache.ignite.Ignition;
public class LoadCaches {
public static void main(String[] args) throws Exception {
try (Ignite ignite = Ignition.start("RemoteIgniteCluster-client.xml");IgniteDataStreamer<Integer, Object> stmr = ignite.dataStreamer("PersonsCache")) {
System.out.println(">>> Loading caches...");
stmr.allowOverwrite(true);
stmr.autoFlushFrequency(1000);
System.out.println(">>> Loading cache: PersonsCache");
for (int i = 0; i < 5000000; i++)
stmr.addData(i, new Integer(i));
/*ignite.cache("PersonsCache").loadCache(null);*/
System.out.println(">>> All caches loaded!");
}
}
}
I tried changing
IgniteDataStreamer<Integer, Object> stmr = ignite.dataStreamer("PersonsCache"))
and
stmr.addData(i, new Integer(i));
with
IgniteDataStreamer<Integer, String> stmr = ignite.dataStreamer("PersonsCache"))
and
stmr.addData(i, Integer.toString(i));
and yet i get the same exception in the server logs, and when i try to query the cache i get empty results.
My spring xmls look like this:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<!-- Load external properties file. -->
<bean id="placeholderConfig" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:secret.properties"/>
</bean>
<!-- Data source beans will be initialized from external properties file. -->
<bean id="dsMySQL_DB" class="com.mysql.jdbc.jdbc2.optional.MysqlDataSource">
<property name="URL" value="${dsMySQL_DB.jdbc.url}"/>
<property name="user" value="${dsMySQL_DB.jdbc.username}"/>
<property name="password" value="${dsMySQL_DB.jdbc.password}"/>
</bean>
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="testcluster"/>
<property name="peerClassLoadingEnabled" value="false" />
<property name="includeEventTypes">
<list>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED" />
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED" />
</list>
</property>
<!-- Configure internal thread pool. -->
<property name="publicThreadPoolSize" value="64" />
<!-- Configure system thread pool. -->
<property name="systemThreadPoolSize" value="32" />
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<!-- <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> -->
<property name="addresses">
<list>
<value>127.0.0.1:47500..47510</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="atomicConfiguration">
<bean class="org.apache.ignite.configuration.AtomicConfiguration">
<property name="backups" value="0"/>
</bean>
</property>
<property name="cacheKeyConfiguration">
<list>
<bean class="org.apache.ignite.cache.CacheKeyConfiguration">
<constructor-arg value="com.gmail.testcluster.model.Persons"/>
<constructor-arg value="age"/>
</bean>
</list>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<!-- Set rebalance batch size to 1 MB. -->
<property name="rebalanceBatchSize" value="#{1024 * 1024}" />
<!-- Explicitly disable rebalance throttling. -->
<property name="rebalanceThrottle" value="0" />
<!-- Set 4 threads for rebalancing. -->
<property name="rebalanceThreadPoolSize" value="4" />
<property name="name" value="PersonsCache"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="0"/>
<property name="affinity">
<bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
<property name="partitions" value="2048"/>
</bean>
</property>
<!-- <property name="queryDetailMetricsSize" value="50"/> -->
<property name="cacheStoreFactory">
<bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
<property name="dataSourceBean" value="dsMySQL_DB"/>
<property name="dialect">
<bean class="org.apache.ignite.cache.store.jdbc.dialect.MySQLDialect">
</bean>
</property>
<property name="types">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
<property name="cacheName" value="PersonsCache"/>
<property name="keyType" value="java.lang.Integer"/>
<property name="valueType" value="com.gmail.testcluster.model.Persons"/>
<property name="databaseSchema" value="MY_DB"/>
<property name="databaseTable" value="PERSONS"/>
<property name="keyFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="ID"/>
<constructor-arg value="int"/>
<constructor-arg value="id"/>
</bean>
</list>
</property>
<property name="valueFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="CLASS_NO"/>
<constructor-arg value="java.lang.Integer"/>
<constructor-arg value="class_no"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.VARCHAR"/>
</constructor-arg>
<constructor-arg value="NAME"/>
<constructor-arg value="java.lang.String"/>
<constructor-arg value="name"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="AGE"/>
<constructor-arg value="java.lang.Integer"/>
<constructor-arg value="age"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.DOUBLE"/>
</constructor-arg>
<constructor-arg value="R_AMT"/>
<constructor-arg value="java.lang.Double"/>
<constructor-arg value="rAmt"/>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</property>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="writeBehindEnabled" value="true"/>
<property name="writeBehindBatchSize" value="500"/>
<property name="writeBehindFlushSize" value="1000"/>
<property name="writeBehindFlushFrequency" value="60000"/>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="java.lang.Integer"/>
<property name="valueType" value="com.gmail.testcluster.model.Persons"/>
<property name="fields">
<map>
<entry key="class_no" value="java.lang.Integer"/>
<entry key="name" value="java.lang.String"/>
<entry key="age" value="java.lang.Integer"/>
<entry key="rAmt" value="java.lang.Double"/>
</map>
</property>
<property name="aliases">
<map>
<entry key="rAmt" value="R_AMT"/>
</map>
</property>
<!-- <property name="indexes">
<list>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="queryindexing"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="class_no" value="true"/>
</map>
</property>
</bean>
</list>
</property> -->
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
I am not sure where the problem is, can someone take a look? Thanks!!!
When DataStreamer's property allowOverwrite=false (default) it causes Persistent Store to be skipped.
Cache store is invoked only when AllowOverwrite is true. Link to doc
if you want to use readThrough, then use writeThrough. You can read about these modes in doc

Not able to find database while joining 2 cache in ignite

I am working on utilizing distributed joins between multiple ignite cache. I loaded the required data in both the caches and while performing the join it fails while parsing the SQL suggesting "Database not found" (Please see the stack trace).
Caused by: org.h2.jdbc.JdbcSQLException: Database "OFFER" not found; SQL statement:
SELECT "customOrganizationCache".Organization._key, "customOrganizationCache".Organization._val from Organization as organization, "customOfferCache".Offer as offer where organization._id = offer.relationships.customer.targets.key and organization._id = ? [90013-191]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:179)
Below is my ignite config file:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="offerCacheStoreFactory" class="javax.cache.configuration.FactoryBuilder" factory-method="factoryOf">
<constructor-arg><value>com.xyz.exploreignite.cache.CustomOfferCacheStore</value></constructor-arg>
</bean>
<bean id="organizationCacheStoreFactory" class="javax.cache.configuration.FactoryBuilder" factory-method="factoryOf">
<constructor-arg><value>com.xyz.exploreignite.cache.CustomOrganizationCacheStore</value></constructor-arg>
</bean>
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="peerClassLoadingEnabled" value="false"/>
<property name="clientMode" value="false"/>
<property name="gridName" value="clusterGrid"/>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="1"/>
<property name="name" value="customOrganizationCache"/>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="writeBehindEnabled" value="true"/>
<property name="copyOnRead" value="false"/>
<property name="memoryMode" value="OFFHEAP_TIERED"/>
<property name="atomicWriteOrderMode" value="PRIMARY"/>
<property name="indexedTypes" >
<list>
<value>java.lang.String</value>
<value>com.xyz.exploreignite.pojo.Organization</value>
</list>
</property>
<!-- Cache store. -->
<property name="cacheStoreFactory" ref="organizationCacheStoreFactory"/>
<property name="swapEnabled" value="false"/>
<property name="offHeapMaxMemory" value="0"/>
<property name="evictionPolicy">
<!-- LRU eviction policy. -->
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<!-- Set the maximum cache size to 1 million (default is 100,000). -->
<property name="maxSize" value="1000000"/>
</bean>
</property>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="1"/>
<property name="name" value="customOfferCache"/>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="writeBehindEnabled" value="true"/>
<property name="copyOnRead" value="false"/>
<property name="memoryMode" value="OFFHEAP_TIERED"/>
<property name="atomicWriteOrderMode" value="PRIMARY"/>
<property name="indexedTypes" >
<list>
<value>java.lang.String</value>
<value>com.xyz.exploreignite.pojo.Offer</value>
</list>
</property>
<!-- Cache store. -->
<property name="cacheStoreFactory" ref="offerCacheStoreFactory"/>
<property name="swapEnabled" value="false"/>
<property name="offHeapMaxMemory" value="0"/>
<property name="evictionPolicy">
<!-- LRU eviction policy. -->
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<!-- Set the maximum cache size to 1 million (default is 100,000). -->
<property name="maxSize" value="1000000"/>
</bean>
</property>
</bean>
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery. For information on all options refer
to our documentation: http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
Below is the code that I am using while performing the join:
try (Ignite ignite = Ignition.start(
// "/home/impadmin/ignite/apache-ignite-fabric-1.7.0-bin/examples/config/example-cache-ss-cluster.xml"))
// {
"/home/xyz/msheth/install/apache-ignite-fabric-1.7.0-bin/examples/config/example-cache-ss-cluster.xml")) {
try (IgniteCache<String, Offer> customOfferCache = ignite.getOrCreateCache("customOfferCache");
IgniteCache<String, Organization> customOrganizationCache = ignite
.getOrCreateCache("customOrganizationCache")) {
SqlFieldsQuery joinQuery = new SqlFieldsQuery("select organization.displayName "
+ "from Organization as organization, \"customOfferCache\".Offer as offer"
+ " where organization._id = offer.relationships.customer.targets.key "
+ "and organization._id = ?");
joinQuery.setDistributedJoins(true);
long startTime = System.currentTimeMillis();
try (QueryCursor<List<?>> joinCursor = customOrganizationCache
.query(joinQuery.setArgs("542de0b83b2d445f0a0e0f53"))) {
for (List<?> organizationEntry : joinCursor)
System.out.println("Organizations display name: " + organizationEntry);
}
System.out.println("Time to fetch join based record:" + (System.currentTimeMillis() - startTime));
}
}
Please help me to find the root cause.
The issue is in this expression:
offer.relationships.customer.targets.key
Couple of considerations around this:
You are allowed to have nested objects, however Ignite will create flat schema. E.g., customer field can be accessed as a member of Offer table, you should not provide this full path for this. Generally, you should try to avoid nested object when using SQL and create simple POJO classes instead.
However, you are not allowed to query inside a collection. In your case the whole collection of targets is saved as a single object and you can't select based on its contents. You should have separate cache for Target objects and join it with other tables.