I am using Spring, Hibernate for CRUD operations and using Apache 'BasicDataSource' for connection pooling
Now the problem is when use below following configuration in datasource
<property name="maxActive" value="100"/>
<property name="maxWait" value="10000"/>
<property name="removeAbandoned" value="true"/>
<property name="removeAbandonedTimeout" value="60"/>
<property name="logAbandoned" value="true"/>
<property name="maxIdle" value="10"/>
than after using all connections I m getting Error of "Connection reset on log" but it takes long time to get back.
And if I m removing following lines from datasource
<property name="removeAbandoned" value="true"/>
<property name="removeAbandonedTimeout" value="60"/>
<property name="logAbandoned" value="true"/>
And adding below line to SessionFactory(hibernateProperties)
<prop key="hibernate.connection.release_mode">after_statement</prop>
than i getting no error on console but the problem is It uses the connection and close as soon as it completes.
Related
I am facing an issue similar to the one jedis-1929. We are using JedisPool with maxTotal=400 and have ensured that after using jedis from pool.getResource() we are returning the connection back to the pool in finally block using jedis.close() method. Version of jedis is 3.0.0. The issue appears after continuously running the program for several days. We are getting/setting key-value pair in Redis around 0.1m times per minute. The key and values are both quite small, values approx 120 bytes. Use-case is mostly read heavy.
Wanted to set AbandonConfig to ensure leaked connections if any get closed after a default timeout, but don't see a way to set AbandonConfig related settings for JedisPool. Following is the exception we get, when numActives become equal to maxTotal
redis.clients.jedis.exceptions.JedisExhaustedPoolException: Could not get a resource since the pool is exhausted
at redis.clients.jedis.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234)
at com.til.ibeat.connection.RedisConnection.getconnection(RedisConnection.java:52)
at com.til.ibeat.service.impl.RedisCacheServiceImpl.pushToRedisSet(RedisCacheServiceImpl.java:188)
at com.til.ibeat.service.impl.MessageEnrichService.getVisitorType(MessageEnrichService.java:541)
at com.til.ibeat.service.impl.MessageEnrichService.populateVisitorType(MessageEnrichService.java:439)
at com.til.ibeat.service.impl.IbeatEventLogService.process(IbeatEventLogService.java:111)
at com.til.ibeat.service.impl.IbeatEventLogService$2.call(IbeatEventLogService.java:70)
at com.til.ibeat.service.impl.IbeatEventLogService$2.call(IbeatEventLogService.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException: Pool exhausted
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:452)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.jedis.util.Pool.getResource(Pool.java:50)
... 12 common frames omitted
Following is our configuration:
<bean id="redisPool2" class="redis.clients.jedis.JedisPool">
<constructor-arg index="0" type="org.apache.commons.pool2.impl.GenericObjectPoolConfig" ref="poolConfig2"/>
<constructor-arg index="1" type="java.lang.String" value="${jedis2.host}" />
<constructor-arg index="2" type="int" value="${jedis2.port:6379}"/>
<constructor-arg index="3" type="int" value="${jedis2.timeout.millis:200}"/>
<constructor-arg index="4" type="java.lang.String" value="${jedis2.password}" />
</bean>
<bean id="poolConfig2" class="org.apache.commons.pool2.impl.GenericObjectPoolConfig">
<property name="maxTotal" value="${jedis2.max.total:-1}" />
<property name="maxIdle" value="${jedis2.max.idle:50}" />
<property name="minIdle" value="${jedis2.min.idle:3}" />
<property name="testOnBorrow" value="${jedis2.test.on.borrow:true}" />
<property name="testOnReturn" value="${jedis2.test.on.return:false}" />
<property name="testWhileIdle" value="${jedis2.test.while.idle:false}" />
<property name="minEvictableIdleTimeMillis" value="${jedis2.min.evictable.idle.time.millis:2000}" />
<property name="timeBetweenEvictionRunsMillis" value="${jedis2.time.between.eviction.runs:30000}" />
<property name="numTestsPerEvictionRun" value="${jedis2.tests.per.eviction.run:10}" />
<property name="blockWhenExhausted" value="${jedis2.block.when.exhausted:false}" />
<property name="jmxEnabled" value="${jedis2.jmx.enabled:true}"/>
<property name="jmxNamePrefix" value="${jedis2.host}"/>
</bean>
I'm new with Ignite and I'm trying to test data quality and availability of Ignite cluster.
I use the below xml configuration for setting cluster,
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="socketTimeout" value="50000" />
<property name="networkTimeout" value="50000" />
<property name="reconnectCount" value="5" />
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>x.x.x.1:47500..47509</value>
<value>x.x.x.2:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
Also and jthe CacheConfiguration is,
<bean id="cache-template-bean" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="CACHE_TEMPLATE*"/>
<property name="cacheMode" value="PARTITIONED" />
<property name="backups" value="1" />
<!-- <property name="backups" value="2" />
<property name="backups" value="3" /> -->
<property name="atomicityMode" value="TRANSACTIONAL" />
<property name="writeSynchronizationMode" value="PRIMARY_SYNC" />
<property name="rebalanceBatchSize" value="#{4 * 1024 * 1024}" />
<property name="rebalanceMode" value="ASYNC" />
<property name="statisticsEnabled" value="true" />
<property name="rebalanceBatchesPrefetchCount" value="4" />
<property name="defaultLockTimeout" value="5000" />
<property name="readFromBackup" value="true" />
<property name="queryParallelism" value="6" />
<property name="nodeFilter">
<bean class="org.apache.ignite.util.AttributeNodeFilter">
<constructor-arg>
<map>
<entry key="ROLE" value="data.compute"/>
</map>
</constructor-arg>
</bean>
</property>
</bean>
My scenarios are,
Loaded the 5 million data when all the 3 nodes
Bring one node down
The count shows 3.75 million. (Data loss)
Bringing the node up counts 5 million again.
I tried backup 1,2,3 all resulted in the same data loss. As per Ignite documents, appears the data loss should not happen. If this fixed, I can try adding data when the node is down and check how it behaves.
Any suggestions, please?
Ash
The main idea of the baseline topology and persistence is to prevent unnecessary rebalance and store data only in specified server nodes. When a baseline node stopped, it is expected that one will back soon and the rebalance process is not triggered. You could exclude the node from the baseline using api or control.sh utility.
IgniteCache.size() returns the number of primary entries. So when a baseline node is stopped, size() shows a smaller number indicating that a number of primary entries is not accessible.
In your case the data is not lost by two reasons:
1. The data is persisted in backup entries on alive baseline nodes.
2. The primary and backup entries located on the stopped node will back to the cluster after the node started.
[1] https://apacheignite.readme.io/docs/baseline-topology
I am trying to modify default-config.xml by adding cacheConfiguration tags. Do i need to repeat cacheConfiguration XML tag for each data set RDD that i am tyring to keep to keep it in the memory ? Can i set backups to 0, if i don't want it.
ex:
<property name="cacheConfiguration">
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="TEST1_RDD"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="backups" value="0"/>
</bean>
</property> <property name="cacheConfiguration">
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="TEST2_RDD"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="backups" value="0"/>
</bean>
</property>
Also, do i need to specify explicitly write synchronization mode ? and by default which one Ignite consider ?
ex:
<property name="writeSynchronizationMode" value="FULL_SYNC"/>
Appreciate your response.
Yes, You have to write configuration for each cache as your cache may have different functionality/purpose and you have to set configuration according to it.
For backups it's default value is 0 and for CacheWriteSynchronizationMode default value is PRIMARY_SYNC
There is a possibility to define cache templates, if you don't want to provide the same configuration for caches: https://apacheignite.readme.io/docs/cache-template
I have 7 tables and some tables are empty and some has only one row. In table maximum 5 columns, no any extra indexes and I make quite simple SQL queries.
This is how I run h2.
java -jar h2-1.3.176.jar
Queries are generated and executed by eclipselink 2.6.3. These are my settings in persistence.xml
<properties>
<property name="eclipselink.connection-pool.default.initial" value="5"/>
<property name="eclipselink.connection-pool.default.min" value="5"/>
<property name="javax.persistence.jdbc.driver" value="org.h2.Driver"/>
<property name="javax.persistence.jdbc.url" value="jdbc:h2:tcp://localhost//home/somepath;DATABASE_TO_UPPER=FALSE"/>
<property name="javax.persistence.jdbc.user" value="sa"/>
<property name="javax.persistence.jdbc.password" value=""/>
<property name="eclipselink.logging.level" value="FINE"/>
<property name="eclipselink.logging.thread" value="false"/>
<property name="eclipselink.logging.exceptions" value="true"/>
<property name="eclipselink.orm.throw.exceptions" value="true"/>
<property name="eclipselink.left-join-fetch" value="true"/>
<property name="eclipselink.weaving" value="true"/>
</properties>
So every SQL query is executed about one second(!). Besides I hear that for every SQL query hard drive is working hard.
However, when I in H2 webconsole press connect to the same DB, then my java program works very fast. It works fast until I press disconnect in webconsole. How to explain this?
I use BoneCP in my Spring-based application.
<bean id="dataSource" class="com.jolbox.bonecp.BoneCPDataSource" destroy-method="close">
<property name="driverClass" value="com.mysql.jdbc.Driver" />
<property name="jdbcUrl" value="jdbc:mysql://ec2-23-21-211-???.compute-1.amazonaws.com:3306/?????" />
<property name="username" value="*****"/>
<property name="password" value="********"/>
<property name="idleConnectionTestPeriod" value="60"/>
<property name="idleMaxAge" value="240"/>
<property name="maxConnectionsPerPartition" value="3"/>
<property name="minConnectionsPerPartition" value="1"/>
<property name="partitionCount" value="1"/>
<property name="acquireIncrement" value="5"/>
<property name="statementsCacheSize" value="100"/>
<property name="releaseHelperThreads" value="3"/>
</bean>
Is there any short value for jdbcURL?
You can inject it via environmental variable through the CloudBees SDK.
1.Inject the datasource and the following environmental variables via bees app:bind
With the CloudBees SDK:
bees app:bind -a appName -db dbName -as mydb
It will automatically inject a datasource and will create these three environmental variables:
${DATABASE_URL_DB}
${DATABASE_USERNAME_DB}
${DATABASE_PASSWORD_DB}
Please, be aware that you will use on this way one active connection for the maxActive: '20' by default on the Tomcat JDBC Connection Pool.
2.Enable PlaceHolder on Spring framework and mark system-properties-mode as "OVERRIDE".
<context:property-placeholder location="classpath:spring/data-access.properties" system-properties-mode="OVERRIDE"/>
Example here.
3.On your datasource.xml configuration file, then you could use something like this:
value= "jdbc:"+ ${DATABASE_URL_DB}
Be aware that the recommended way to get the datasource on CloudBees is always using JNDI.
In this way, you will use our own implementation of the datasource, so you don't have to write the username, the password or the URL of the database. Instead of all these lines, you can just replace all of them for this one:
<jee:jndi-lookup id="dataSource" jndi-name="jdbc/mydb" resource-ref="true"/>