We have build a 4 node Apache Ignite Cluster and able to do connect and perform the basic operation like creating a Cache from a Java program.
But it fails to connect to the Ignite cluster when I did the MySQL integration.
Following is the error message.
Exception in thread "main" class org.apache.ignite.IgniteException: Resource field is not assignable from the resource: class org.springframework.jdbc.datasource.DriverManagerDataSource
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:906)
at org.apache.ignite.Ignition.start(Ignition.java:350)
at PersonExample.PersonStoreExample.main(PersonStoreExample.java:16)
Caused by: class org.apache.ignite.IgniteCheckedException: Resource field is not assignable from the resource: class org.springframework.jdbc.datasource.DriverManagerDataSource
at org.apache.ignite.internal.processors.resource.GridResourceUtils.inject(GridResourceUtils.java:50)
at org.apache.ignite.internal.processors.resource.GridResourceSpringBeanInjector.inject(GridResourceSpringBeanInjector.java:67)
at org.apache.ignite.internal.processors.resource.GridResourceIoc.injectInternal(GridResourceIoc.java:172)
at org.apache.ignite.internal.processors.resource.GridResourceIoc.inject(GridResourceIoc.java:97)
at org.apache.ignite.internal.processors.resource.GridResourceProcessor.injectGeneric(GridResourceProcessor.java:257)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepare(GridCacheProcessor.java:539)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepare(GridCacheProcessor.java:528)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCache(GridCacheProcessor.java:1270)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:784)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:926)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1736)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1589)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1042)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:964)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:850)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:749)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:619)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:589)
at org.apache.ignite.Ignition.start(Ignition.java:347)
... 1 more
Following is the Ignite Config file:
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<!--
Alter configuration below as needed.
-->
<bean class="org.springframework.jdbc.datasource.DriverManagerDataSource" name="dataSource">
<property name="driverClassName" value="com.mysql.jdbc.Driver" />
<property name="url" value="jdbc:mysql://<<mysqk_host>>:3306/sample_db" />
<property name="username" value="root" />
<property name="password" value="hadoop" />
</bean>
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="personCache" />
<property name="readThrough" value="true" />
<property name="writeThrough" value="true" />
<property name="cacheStoreFactory">
<bean class="javax.cache.configuration.FactoryBuilder" factory-method="factoryOf">
<constructor-arg value="PersonExample.PersonStore" />
</bean>
</property>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="java.lang.Long" />
<property name="valueType" value="PersonExample.Person" />
<property name="fields">
<map>
<entry key="id" value="java.lang.Long" />
<entry key="name" value="java.lang.String" />
<entry key="orgId" value="java.lang.Long" />
<entry key="salary" value="java.lang.Integer" />
</map>
</property>
</bean>
</list>
</property>
</bean>
</list>
</property>
<!-- <property name="peerClassLoadingEnabled" value="true"></property> -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
<property name="zkConnectionString" value=“<<zk_host>>:2181" />
</bean>
</property>
</bean>
</property>
</bean>
How to resolve this error?
Error has been resolved with the following changes:
Added the latest MySQL JDBC jar in ignite Libs directory
Ensured Cluster config file and Client config files are same (Earlier client config file contained few additional configurations in cache configuration).
Added the Person class (Which is a MySQL table POJO, that will be stored in Ignite cache).
Related
I have a RHEL 7 server where I have ActiveMQ installed (version : apache-activemq-5.15.11).I am trying to run the web console over https and hence I went through the instructions mentioned in the documentation here and made changes in my jetty.xml and un-commented the following section:
<!--
Enable this connector if you wish to use https with web console
-->
<!-- bean id="SecureConnector" class="org.eclipse.jetty.server.ServerConnector">
<constructor-arg ref="Server" />
<constructor-arg>
<bean id="handlers" class="org.eclipse.jetty.util.ssl.SslContextFactory">
<property name="keyStorePath" value="${activemq.conf}/broker.ks" />
<property name="keyStorePassword" value="password" />
</bean>
</constructor-arg>
<property name="port" value="8162" />
</bean -->
when I am trying to access the web console like the following:
https://myserver.com:8162/admin/index.jsp
I am getting connection getting timed out in the browser.
Questions :
Am I trying to access it properly - via port 8162? It works fine with port 8161 and over http like the following:
http://myserver.com:8161/admin/index.jsp
Are there any additional changes needed to make it work over https?
My jetty.xml is below for reference:
<!--
Licensed to the Apache Software Foundation (ASF) under one or more contributor
license agreements. See the NOTICE file distributed with this work for additional
information regarding copyright ownership. The ASF licenses this file to You under
the Apache License, Version 2.0 (the "License"); you may not use this file except in
compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or
agreed to in writing, software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing permissions and
limitations under the License.
-->
<!--
An embedded servlet engine for serving up the Admin consoles, REST and Ajax APIs and
some demos Include this file in your configuration to enable ActiveMQ web components
e.g. <import resource="jetty.xml"/>
-->
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="securityLoginService" class="org.eclipse.jetty.security.HashLoginService">
<property name="name" value="ActiveMQRealm" />
<property name="config" value="${activemq.conf}/jetty-realm.properties" />
</bean>
<bean id="securityConstraint" class="org.eclipse.jetty.util.security.Constraint">
<property name="name" value="BASIC" />
<property name="roles" value="user,admin" />
<!-- set authenticate=false to disable login -->
<property name="authenticate" value="true" />
</bean>
<bean id="adminSecurityConstraint" class="org.eclipse.jetty.util.security.Constraint">
<property name="name" value="BASIC" />
<property name="roles" value="admin" />
<!-- set authenticate=false to disable login -->
<property name="authenticate" value="true" />
</bean>
<bean id="securityConstraintMapping" class="org.eclipse.jetty.security.ConstraintMapping">
<property name="constraint" ref="securityConstraint" />
<property name="pathSpec" value="/api/*,/admin/*,*.jsp" />
</bean>
<bean id="adminSecurityConstraintMapping" class="org.eclipse.jetty.security.ConstraintMapping">
<property name="constraint" ref="adminSecurityConstraint" />
<property name="pathSpec" value="*.action" />
</bean>
<bean id="rewriteHandler" class="org.eclipse.jetty.rewrite.handler.RewriteHandler">
<property name="rules">
<list>
<bean id="header" class="org.eclipse.jetty.rewrite.handler.HeaderPatternRule">
<property name="pattern" value="*"/>
<property name="name" value="X-FRAME-OPTIONS"/>
<property name="value" value="SAMEORIGIN"/>
</bean>
<bean id="header" class="org.eclipse.jetty.rewrite.handler.HeaderPatternRule">
<property name="pattern" value="*"/>
<property name="name" value="X-XSS-Protection"/>
<property name="value" value="1; mode=block"/>
</bean>
<bean id="header" class="org.eclipse.jetty.rewrite.handler.HeaderPatternRule">
<property name="pattern" value="*"/>
<property name="name" value="X-Content-Type-Options"/>
<property name="value" value="nosniff"/>
</bean>
</list>
</property>
</bean>
<bean id="secHandlerCollection" class="org.eclipse.jetty.server.handler.HandlerCollection">
<property name="handlers">
<list>
<ref bean="rewriteHandler"/>
<bean class="org.eclipse.jetty.webapp.WebAppContext">
<property name="contextPath" value="/admin" />
<property name="resourceBase" value="${activemq.home}/webapps/admin" />
<property name="logUrlOnStart" value="true" />
</bean>
<bean class="org.eclipse.jetty.webapp.WebAppContext">
<property name="contextPath" value="/api" />
<property name="resourceBase" value="${activemq.home}/webapps/api" />
<property name="logUrlOnStart" value="true" />
</bean>
<bean class="org.eclipse.jetty.server.handler.ResourceHandler">
<property name="directoriesListed" value="false" />
<property name="welcomeFiles">
<list>
<value>index.html</value>
</list>
</property>
<property name="resourceBase" value="${activemq.home}/webapps/" />
</bean>
<bean id="defaultHandler" class="org.eclipse.jetty.server.handler.DefaultHandler">
<property name="serveIcon" value="false" />
</bean>
</list>
</property>
</bean>
<bean id="securityHandler" class="org.eclipse.jetty.security.ConstraintSecurityHandler">
<property name="loginService" ref="securityLoginService" />
<property name="authenticator">
<bean class="org.eclipse.jetty.security.authentication.BasicAuthenticator" />
</property>
<property name="constraintMappings">
<list>
<ref bean="adminSecurityConstraintMapping" />
<ref bean="securityConstraintMapping" />
</list>
</property>
<property name="handler" ref="secHandlerCollection" />
</bean>
<bean id="contexts" class="org.eclipse.jetty.server.handler.ContextHandlerCollection">
</bean>
<bean id="jettyPort" class="org.apache.activemq.web.WebConsolePort" init-method="start">
<!-- the default port number for the web console -->
<property name="host" value="0.0.0.0"/>
<property name="port" value="8161"/>
</bean>
<bean id="Server" depends-on="jettyPort" class="org.eclipse.jetty.server.Server"
destroy-method="stop">
<property name="handler">
<bean id="handlers" class="org.eclipse.jetty.server.handler.HandlerCollection">
<property name="handlers">
<list>
<ref bean="contexts" />
<ref bean="securityHandler" />
</list>
</property>
</bean>
</property>
</bean>
<bean id="invokeConnectors" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetObject" ref="Server" />
<property name="targetMethod" value="setConnectors" />
<property name="arguments">
<list>
<bean id="Connector" class="org.eclipse.jetty.server.ServerConnector">
<constructor-arg ref="Server" />
<!-- see the jettyPort bean -->
<property name="host" value="#{systemProperties['jetty.host']}" />
<property name="port" value="#{systemProperties['jetty.port']}" />
</bean>
<!--
Enable this connector if you wish to use https with web console
-->
<!-- Enabling the following connector on Nov 11,2020 for HTTPS -->
<bean id="SecureConnector" class="org.eclipse.jetty.server.ServerConnector">
<constructor-arg ref="Server" />
<constructor-arg>
<bean id="handlers" class="org.eclipse.jetty.util.ssl.SslContextFactory">
<property name="keyStorePath" value="${activemq.conf}/broker.ks" />
<property name="keyStorePassword" value="password" />
</bean>
</constructor-arg>
<property name="port" value="8162" />
</bean>
</list>
</property>
</bean>
<bean id="configureJetty" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="staticMethod" value="org.apache.activemq.web.config.JspConfigurer.configureJetty" />
<property name="arguments">
<list>
<ref bean="Server" />
<ref bean="secHandlerCollection" />
</list>
</property>
</bean>
<bean id="invokeStart" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean"
depends-on="configureJetty, invokeConnectors">
<property name="targetObject" ref="Server" />
<property name="targetMethod" value="start" />
</bean>
</beans>
I'm new to Ignite and following various blogs to understand the configuration required.
As a starter, I have used "movie lens" data (Movies & Ratings) and used Ignite Web Console to generate the required configuration.
Please find the required details about my code below:
1) MySQL Table Structure
CREATE TABLE movies( movie_id INTEGER,
movie_name VARCHAR(250), genre VARCHAR(250),
CONSTRAINT pk_movie PRIMARY KEY (movie_id) );
CREATE TABLE ratings(
user_id INTEGER, movie_id INTEGER, rating FLOAT, timestamp TIMESTAMP,
CONSTRAINT pk_rating PRIMARY KEY (user_id, movie_id) );
2) Ignite Config File in Spring Boot Project (Auto Generated by Ignite Web Console)
<?xml version="1.0" encoding="UTF-8"?>
<!-- This file was generated by Ignite Web Console (05/15/2019, 18:37) -->
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<!-- Load external properties file. -->
<bean id="placeholderConfig" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:secret.properties"/>
</bean>
<!-- Data source beans will be initialized from external properties file. -->
<bean id="dsGeneric_Movielens" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="jdbcUrl" value="${dsGeneric_Movielens.jdbc.url}"/>
<property name="user" value="${dsGeneric_Movielens.jdbc.username}"/>
<property name="password" value="${dsGeneric_Movielens.jdbc.password}"/>
</bean>
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="ImportedCluster"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47510</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="MoviesCache"/>
<property name="cacheMode" value="REPLICATED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="cacheStoreFactory">
<bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
<property name="dataSourceBean" value="dsGeneric_Movielens"/>
<property name="dialect">
<bean class="org.apache.ignite.cache.store.jdbc.dialect.BasicJdbcDialect">
</bean>
</property>
<property name="types">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
<property name="cacheName" value="MoviesCache"/>
<property name="keyType" value="java.lang.Integer"/>
<property name="valueType" value="com.learnwithmanoj.ignite.model.Movies"/>
<property name="databaseSchema" value="movielens"/>
<property name="databaseTable" value="movies"/>
<property name="keyFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="movie_id"/>
<constructor-arg value="int"/>
<constructor-arg value="movieId"/>
</bean>
</list>
</property>
<property name="valueFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.VARCHAR"/>
</constructor-arg>
<constructor-arg value="movie_name"/>
<constructor-arg value="java.lang.String"/>
<constructor-arg value="movieName"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.VARCHAR"/>
</constructor-arg>
<constructor-arg value="genre"/>
<constructor-arg value="java.lang.String"/>
<constructor-arg value="genre"/>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</property>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="java.lang.Integer"/>
<property name="valueType" value="com.learnwithmanoj.ignite.model.Movies"/>
<property name="keyFieldName" value="movieId"/>
<property name="keyFields">
<list>
<value>movieId</value>
</list>
</property>
<property name="fields">
<map>
<entry key="movieName" value="java.lang.String"/>
<entry key="genre" value="java.lang.String"/>
<entry key="movieId" value="java.lang.Integer"/>
</map>
</property>
<property name="aliases">
<map>
<entry key="movieId" value="movie_id"/>
<entry key="movieName" value="movie_name"/>
</map>
</property>
</bean>
</list>
</property>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="RatingsCache"/>
<property name="cacheMode" value="REPLICATED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="cacheStoreFactory">
<bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
<property name="dataSourceBean" value="dsGeneric_Movielens"/>
<property name="dialect">
<bean class="org.apache.ignite.cache.store.jdbc.dialect.BasicJdbcDialect">
</bean>
</property>
<property name="types">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
<property name="cacheName" value="RatingsCache"/>
<property name="keyType" value="com.learnwithmanoj.ignite.model.RatingsKey"/>
<property name="valueType" value="com.learnwithmanoj.ignite.model.Ratings"/>
<property name="databaseSchema" value="movielens"/>
<property name="databaseTable" value="ratings"/>
<property name="keyFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="user_id"/>
<constructor-arg value="int"/>
<constructor-arg value="userId"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="movie_id"/>
<constructor-arg value="int"/>
<constructor-arg value="movieId"/>
</bean>
</list>
</property>
<property name="valueFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.REAL"/>
</constructor-arg>
<constructor-arg value="rating"/>
<constructor-arg value="java.lang.Double"/>
<constructor-arg value="rating"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.TIMESTAMP"/>
</constructor-arg>
<constructor-arg value="timestamp"/>
<constructor-arg value="java.sql.Timestamp"/>
<constructor-arg value="timestamp"/>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</property>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="com.learnwithmanoj.ignite.model.RatingsKey"/>
<property name="valueType" value="com.learnwithmanoj.ignite.model.Ratings"/>
<property name="keyFields">
<list>
<value>userId</value>
<value>movieId</value>
</list>
</property>
<property name="fields">
<map>
<entry key="userId" value="java.lang.Integer"/>
<entry key="movieId" value="java.lang.Integer"/>
<entry key="rating" value="java.lang.Double"/>
<entry key="timestamp" value="java.sql.Timestamp"/>
</map>
</property>
<property name="aliases">
<map>
<entry key="userId" value="user_id"/>
<entry key="movieId" value="movie_id"/>
</map>
</property>
<property name="indexes">
<list>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="fk_movie_id"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="movieId" value="false"/>
</map>
</property>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</beans>
3) Ignite Config File in Ignite Installation Folder
<?xml version="1.0" encoding="UTF-8"?>
<!-- This file was generated by Ignite Web Console (05/15/2019, 08:52) -->
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<!-- Load external properties file. -->
<bean id="placeholderConfig" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:secret.properties"/>
</bean>
<!-- Data source beans will be initialized from external properties file. -->
<bean id="datasource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property name="jdbcUrl" value="${datasource.jdbc.url}"/>
<property name="user" value="${datasource.jdbc.username}"/>
<property name="password" value="${datasource.jdbc.password}"/>
</bean>
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="MovieRatingsCluster"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47510</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
Problem Statement:
When I start my Spring Boot Application as a stand-alone Ignite node, then all the data from the MySQL database is getting synced into Ignite node as expected. Also, if I try to join between the Ratings and Movies table, it's joining properly and giving the correct result.
Join Query Used:
select r.movie_id, m.movie_name, count(r.rating) as count from
"RatingsCache".ratings r inner join "MoviesCache".movies m where
r.movie_id = m.movie_id group by r.movie_id order by count desc limit
5;
Now coming to the problem. If I do the same testing using more than one node, then the Join starts to give incorrect results due to the data not being colocated.
Can anyone guide me with the right set of configuration which needs to be added to fix the Join problem in Partition Mode?
I have even tried to add the below config.
<property name="cacheKeyConfiguration">
<list>
<bean class="org.apache.ignite.cache.CacheKeyConfiguration">
<property name="typeName" value="com.learnwithmanoj.ignite.model.RatingsKey" />
<property name="affinityKeyFieldName" value="movie_id" />
</bean>
</list>
</property>
But got the below error.
Caused by: org.apache.ignite.binary.BinaryObjectException: Binary type
has different affinity key fields
[typeName=com.learnwithmanoj.ignite.model.RatingsKey,
affKeyFieldName1=null, affKeyFieldName2=movie_id]
Your solution should work. First you need to clean marshaller/ directory under Ignite home dir on all nodes, then restart those nodes.
You might also need to specify key field name in caps: MOVIE_ID.
I'm using ignite version 2.0.0
I am trying to load about 40M data from db to my ignite instances(2 in this case). I read about the Ignite DataStreamers from https://apacheignite.readme.io/v1.2/docs/data-streamers
Because of the very low data loading speed of loadCache() i changed loadCache() with IgniteDataStreamer.addData().
Upon execution, i notice in the web console that the metrics for on-heap entries gets incrementing(i.e., it shows that data is being loaded). But when i query the ignite cache, i get the result as empty.
Also i notice that the server logs are showcasing this exception:
[01:46:19,541][ERROR][flusher-0-#66%RemoteIgniteCluster%][GridCacheWriteBehindStore] Unable to update underlying store: CacheJdbcPojoStore []
javax.cache.CacheException: Failed to read property value from non binary object [class=class java.lang.Integer, property=class_no]
at org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStore.extractBinaryParameter(CacheJdbcPojoStore.java:122)
at org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStore.extractParameter(CacheJdbcPojoStore.java:69)
at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.fillValueParameters(CacheAbstractJdbcStore.java:1414)
at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.writeAll(CacheAbstractJdbcStore.java:1081)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.updateStore(GridCacheWriteBehindStore.java:804)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.applyBatch(GridCacheWriteBehindStore.java:720)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.access$2400(GridCacheWriteBehindStore.java:75)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.flushCacheCoalescing(GridCacheWriteBehindStore.java:1108)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.body(GridCacheWriteBehindStore.java:1006)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
[01:50:00,945][ERROR][flusher-0-#66%RemoteIgniteCluster%][GridCacheWriteBehindStore] Unable to update underlying store: CacheJdbcPojoStore []
javax.cache.integration.CacheWriterException: Failed to write entries in database
at org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.writeAll(CacheAbstractJdbcStore.java:1151)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.updateStore(GridCacheWriteBehindStore.java:804)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.applyBatch(GridCacheWriteBehindStore.java:720)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.access$2400(GridCacheWriteBehindStore.java:75)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.flushCacheCoalescing(GridCacheWriteBehindStore.java:1108)
at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.body(GridCacheWriteBehindStore.java:1006)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
My LoadCache.java code is:
package load;
import javax.naming.event.ObjectChangeListener;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteDataStreamer;
import org.apache.ignite.Ignition;
public class LoadCaches {
public static void main(String[] args) throws Exception {
try (Ignite ignite = Ignition.start("RemoteIgniteCluster-client.xml");IgniteDataStreamer<Integer, Object> stmr = ignite.dataStreamer("PersonsCache")) {
System.out.println(">>> Loading caches...");
stmr.allowOverwrite(true);
stmr.autoFlushFrequency(1000);
System.out.println(">>> Loading cache: PersonsCache");
for (int i = 0; i < 5000000; i++)
stmr.addData(i, new Integer(i));
/*ignite.cache("PersonsCache").loadCache(null);*/
System.out.println(">>> All caches loaded!");
}
}
}
I tried changing
IgniteDataStreamer<Integer, Object> stmr = ignite.dataStreamer("PersonsCache"))
and
stmr.addData(i, new Integer(i));
with
IgniteDataStreamer<Integer, String> stmr = ignite.dataStreamer("PersonsCache"))
and
stmr.addData(i, Integer.toString(i));
and yet i get the same exception in the server logs, and when i try to query the cache i get empty results.
My spring xmls look like this:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<!-- Load external properties file. -->
<bean id="placeholderConfig" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="location" value="classpath:secret.properties"/>
</bean>
<!-- Data source beans will be initialized from external properties file. -->
<bean id="dsMySQL_DB" class="com.mysql.jdbc.jdbc2.optional.MysqlDataSource">
<property name="URL" value="${dsMySQL_DB.jdbc.url}"/>
<property name="user" value="${dsMySQL_DB.jdbc.username}"/>
<property name="password" value="${dsMySQL_DB.jdbc.password}"/>
</bean>
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="igniteInstanceName" value="testcluster"/>
<property name="peerClassLoadingEnabled" value="false" />
<property name="includeEventTypes">
<list>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED" />
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED" />
</list>
</property>
<!-- Configure internal thread pool. -->
<property name="publicThreadPoolSize" value="64" />
<!-- Configure system thread pool. -->
<property name="systemThreadPoolSize" value="32" />
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<!-- <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> -->
<property name="addresses">
<list>
<value>127.0.0.1:47500..47510</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="atomicConfiguration">
<bean class="org.apache.ignite.configuration.AtomicConfiguration">
<property name="backups" value="0"/>
</bean>
</property>
<property name="cacheKeyConfiguration">
<list>
<bean class="org.apache.ignite.cache.CacheKeyConfiguration">
<constructor-arg value="com.gmail.testcluster.model.Persons"/>
<constructor-arg value="age"/>
</bean>
</list>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<!-- Set rebalance batch size to 1 MB. -->
<property name="rebalanceBatchSize" value="#{1024 * 1024}" />
<!-- Explicitly disable rebalance throttling. -->
<property name="rebalanceThrottle" value="0" />
<!-- Set 4 threads for rebalancing. -->
<property name="rebalanceThreadPoolSize" value="4" />
<property name="name" value="PersonsCache"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="0"/>
<property name="affinity">
<bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
<property name="partitions" value="2048"/>
</bean>
</property>
<!-- <property name="queryDetailMetricsSize" value="50"/> -->
<property name="cacheStoreFactory">
<bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
<property name="dataSourceBean" value="dsMySQL_DB"/>
<property name="dialect">
<bean class="org.apache.ignite.cache.store.jdbc.dialect.MySQLDialect">
</bean>
</property>
<property name="types">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcType">
<property name="cacheName" value="PersonsCache"/>
<property name="keyType" value="java.lang.Integer"/>
<property name="valueType" value="com.gmail.testcluster.model.Persons"/>
<property name="databaseSchema" value="MY_DB"/>
<property name="databaseTable" value="PERSONS"/>
<property name="keyFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="ID"/>
<constructor-arg value="int"/>
<constructor-arg value="id"/>
</bean>
</list>
</property>
<property name="valueFields">
<list>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="CLASS_NO"/>
<constructor-arg value="java.lang.Integer"/>
<constructor-arg value="class_no"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.VARCHAR"/>
</constructor-arg>
<constructor-arg value="NAME"/>
<constructor-arg value="java.lang.String"/>
<constructor-arg value="name"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.INTEGER"/>
</constructor-arg>
<constructor-arg value="AGE"/>
<constructor-arg value="java.lang.Integer"/>
<constructor-arg value="age"/>
</bean>
<bean class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
<constructor-arg>
<util:constant static-field="java.sql.Types.DOUBLE"/>
</constructor-arg>
<constructor-arg value="R_AMT"/>
<constructor-arg value="java.lang.Double"/>
<constructor-arg value="rAmt"/>
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
</property>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="writeBehindEnabled" value="true"/>
<property name="writeBehindBatchSize" value="500"/>
<property name="writeBehindFlushSize" value="1000"/>
<property name="writeBehindFlushFrequency" value="60000"/>
<property name="queryEntities">
<list>
<bean class="org.apache.ignite.cache.QueryEntity">
<property name="keyType" value="java.lang.Integer"/>
<property name="valueType" value="com.gmail.testcluster.model.Persons"/>
<property name="fields">
<map>
<entry key="class_no" value="java.lang.Integer"/>
<entry key="name" value="java.lang.String"/>
<entry key="age" value="java.lang.Integer"/>
<entry key="rAmt" value="java.lang.Double"/>
</map>
</property>
<property name="aliases">
<map>
<entry key="rAmt" value="R_AMT"/>
</map>
</property>
<!-- <property name="indexes">
<list>
<bean class="org.apache.ignite.cache.QueryIndex">
<property name="name" value="queryindexing"/>
<property name="indexType" value="SORTED"/>
<property name="fields">
<map>
<entry key="class_no" value="true"/>
</map>
</property>
</bean>
</list>
</property> -->
</bean>
</list>
</property>
</bean>
</list>
</property>
</bean>
I am not sure where the problem is, can someone take a look? Thanks!!!
When DataStreamer's property allowOverwrite=false (default) it causes Persistent Store to be skipped.
Cache store is invoked only when AllowOverwrite is true. Link to doc
if you want to use readThrough, then use writeThrough. You can read about these modes in doc
I am working on utilizing distributed joins between multiple ignite cache. I loaded the required data in both the caches and while performing the join it fails while parsing the SQL suggesting "Database not found" (Please see the stack trace).
Caused by: org.h2.jdbc.JdbcSQLException: Database "OFFER" not found; SQL statement:
SELECT "customOrganizationCache".Organization._key, "customOrganizationCache".Organization._val from Organization as organization, "customOfferCache".Offer as offer where organization._id = offer.relationships.customer.targets.key and organization._id = ? [90013-191]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:179)
Below is my ignite config file:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="offerCacheStoreFactory" class="javax.cache.configuration.FactoryBuilder" factory-method="factoryOf">
<constructor-arg><value>com.xyz.exploreignite.cache.CustomOfferCacheStore</value></constructor-arg>
</bean>
<bean id="organizationCacheStoreFactory" class="javax.cache.configuration.FactoryBuilder" factory-method="factoryOf">
<constructor-arg><value>com.xyz.exploreignite.cache.CustomOrganizationCacheStore</value></constructor-arg>
</bean>
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="peerClassLoadingEnabled" value="false"/>
<property name="clientMode" value="false"/>
<property name="gridName" value="clusterGrid"/>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="1"/>
<property name="name" value="customOrganizationCache"/>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="writeBehindEnabled" value="true"/>
<property name="copyOnRead" value="false"/>
<property name="memoryMode" value="OFFHEAP_TIERED"/>
<property name="atomicWriteOrderMode" value="PRIMARY"/>
<property name="indexedTypes" >
<list>
<value>java.lang.String</value>
<value>com.xyz.exploreignite.pojo.Organization</value>
</list>
</property>
<!-- Cache store. -->
<property name="cacheStoreFactory" ref="organizationCacheStoreFactory"/>
<property name="swapEnabled" value="false"/>
<property name="offHeapMaxMemory" value="0"/>
<property name="evictionPolicy">
<!-- LRU eviction policy. -->
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<!-- Set the maximum cache size to 1 million (default is 100,000). -->
<property name="maxSize" value="1000000"/>
</bean>
</property>
</bean>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="1"/>
<property name="name" value="customOfferCache"/>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="writeBehindEnabled" value="true"/>
<property name="copyOnRead" value="false"/>
<property name="memoryMode" value="OFFHEAP_TIERED"/>
<property name="atomicWriteOrderMode" value="PRIMARY"/>
<property name="indexedTypes" >
<list>
<value>java.lang.String</value>
<value>com.xyz.exploreignite.pojo.Offer</value>
</list>
</property>
<!-- Cache store. -->
<property name="cacheStoreFactory" ref="offerCacheStoreFactory"/>
<property name="swapEnabled" value="false"/>
<property name="offHeapMaxMemory" value="0"/>
<property name="evictionPolicy">
<!-- LRU eviction policy. -->
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<!-- Set the maximum cache size to 1 million (default is 100,000). -->
<property name="maxSize" value="1000000"/>
</bean>
</property>
</bean>
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery. For information on all options refer
to our documentation: http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
Below is the code that I am using while performing the join:
try (Ignite ignite = Ignition.start(
// "/home/impadmin/ignite/apache-ignite-fabric-1.7.0-bin/examples/config/example-cache-ss-cluster.xml"))
// {
"/home/xyz/msheth/install/apache-ignite-fabric-1.7.0-bin/examples/config/example-cache-ss-cluster.xml")) {
try (IgniteCache<String, Offer> customOfferCache = ignite.getOrCreateCache("customOfferCache");
IgniteCache<String, Organization> customOrganizationCache = ignite
.getOrCreateCache("customOrganizationCache")) {
SqlFieldsQuery joinQuery = new SqlFieldsQuery("select organization.displayName "
+ "from Organization as organization, \"customOfferCache\".Offer as offer"
+ " where organization._id = offer.relationships.customer.targets.key "
+ "and organization._id = ?");
joinQuery.setDistributedJoins(true);
long startTime = System.currentTimeMillis();
try (QueryCursor<List<?>> joinCursor = customOrganizationCache
.query(joinQuery.setArgs("542de0b83b2d445f0a0e0f53"))) {
for (List<?> organizationEntry : joinCursor)
System.out.println("Organizations display name: " + organizationEntry);
}
System.out.println("Time to fetch join based record:" + (System.currentTimeMillis() - startTime));
}
}
Please help me to find the root cause.
The issue is in this expression:
offer.relationships.customer.targets.key
Couple of considerations around this:
You are allowed to have nested objects, however Ignite will create flat schema. E.g., customer field can be accessed as a member of Offer table, you should not provide this full path for this. Generally, you should try to avoid nested object when using SQL and create simple POJO classes instead.
However, you are not allowed to query inside a collection. In your case the whole collection of targets is saved as a single object and you can't select based on its contents. You should have separate cache for Target objects and join it with other tables.
Could you please help to solve the problem with XA transactions on Activemq and Oracle and Bitronix.
I have a activemq and using camel.xml embeded in for routing message from one queue to oracle db.
this is the content of camel.xml file in conf folder of standalone Activemq.
<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
<route id="partnerToDB">
<from uri="activemqXa:example.A" />
<transacted ref="PROPAGATION_REQUIRED"/>
<transform>
<simple>insert into tbl_1(body,type) values('${in.body}','P') </simple>
</transform>
<to uri="jdbc:dataSource" />
</route>
</camelContext>
<!-- TX configuration -->
<bean id="jtaTransactionManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionManager" ref="transactionManager"/>
<property name="userTransaction" ref="transactionManager" />
</bean>
<bean id="btmConfig" factory-method="getConfiguration" class="bitronix.tm.TransactionManagerServices">
<property name="serverId" value="spring-btm" />
</bean>
<bean id="transactionManager" factory-method="getTransactionManager" class="bitronix.tm.TransactionManagerServices" depends-on="btmConfig" destroy-method="shutdown" />
<bean id="PROPAGATION_REQUIRED" class="org.apache.camel.spring.spi.SpringTransactionPolicy">
<property name="transactionManager" ref="jtaTransactionManager"/>
<property name="propagationBehaviorName" value="PROPAGATION_REQUIRED"/>
</bean>
<!-- JMS configuration -->
<bean id="resourceManager" class="org.apache.activemq.pool.ActiveMQResourceManager" init-method="recoverResource">
<property name="transactionManager" ref="transactionManager" />
<property name="connectionFactory" ref="pooledJmsXaConnectionFactory" />
<property name="resourceName" value="activemq.default,java/testDS1" />
</bean>
<bean id="pooledJmsXaConnectionFactory" class="bitronix.tm.resource.jms.PoolingConnectionFactory" init-method="init" destroy-method="close" >
<property name="className" value="org.apache.activemq.ActiveMQXAConnectionFactory" />
<property name="uniqueName" value="activemq" />
<property name="maxPoolSize" value="8" />
<property name="driverProperties">
<props>
<prop key="brokerURL">tcp://172.16.9.17:61617</prop>
</props>
</property>
</bean>
<bean id="activemqXa" class="org.apache.activemq.camel.component.ActiveMQComponent">
<!-- because of https://issues.apache.org/jira/browse/AMQ-3251, we cannot use the XaPooledConnectionFactory in AMQ 5.5.1 -->
<property name="connectionFactory" ref="pooledJmsXaConnectionFactory"/>
<property name="transacted" value="false"/>
<property name="transactionManager" ref="jtaTransactionManager"/>
</bean>
<!-- JDBC configuration -->
<bean id="dataSource" class="bitronix.tm.resource.jdbc.PoolingDataSource"
init-method="init" destroy-method="close">
<property name="className" value="bitronix.tm.resource.jdbc.lrc.LrcXADataSource" />
<property name="uniqueName" value="java/testDS1" />
<property name="maxPoolSize" value="5" />
<property name="minPoolSize" value="0" />
<property name="allowLocalTransactions" value="false" />
<property name="testQuery" value="SELECT 1 FROM DUAL" />
<property name="driverProperties">
<props>
<prop key="user">test</prop>
<prop key="password">test</prop>
<prop key="url">jdbc:oracle:thin:#db1sh:1521/org.amin.org</prop>
<prop key="driverClassName">oracle.jdbc.OracleDriver</prop>
</props>
</property>
</bean>
when I run Activemq and send a message to example.A queue, nothing is inserted on db and message is dequeued from queue, I get this error in activemq log:
[org.apache.camel.RuntimeCamelException - java.sql.SQLException: cannot commit a resource
enlisted in a global transaction]
Also I have run these SQL scripts for Oracle DataBase:
$ORACLE_HOME/javavm/install/initxa.sql
$ORACLE_HOME/javavm/install/initjvm.sql
and the following grant statments:
grant select on pending_trans$ to public;
grant select on dba_2pc_pending to public;
grant select on dba_pending_transactions to public;
grant execute on dbms_system to <user>;
Any ideas/solutions are welcome!
Thanks
Looking at the first lines in the camel jdbc component docs I find this information box
This component can not be used as a Transactional Client. If you need transaction support in your route, you should use the SQL component instead.