I'm seeing some unique constraint violations during a load test of 10,000 messages (only 8 out of 10,000 affected). I'm wondering if it's something I need to change in the DBCP data source settings, or Active MQ settings. The error is
org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during
JMS processing; nested exception is javax.jms.JMSException:
ORA-00001: unique constraint (WEBSVC.SYS_C00181594) violated
I've looked up this Constraint and tracked it to ACTIVEMQ_MSGS a table we let Active MQ create and maintain itself in Oracle. e.g.
select * from all_constraints
where constraint_name = 'SYS_C00181594'
OWNER CONSTRAINT_NAME CONSTRAINT_TYPE TABLE_NAME SEARCH_CONDITION R_OWNER R_CONSTRAINT_NAME DELETE_RULE STATUS DEFERRABLE DEFERRED VALIDATED GENERATED BAD RELY LAST_CHANGE INDEX_OWNER INDEX_NAME INVALID VIEW_RELATED
------ --------------- --------------- ------------- ---------------- ------- ----------------- ----------- ------- -------------- --------- --------- -------------- --- ---- -------------------- ----------- ------------- ------- ------------
WEBSVC SYS_C00181594 P ACTIVEMQ_MSGS ENABLED NOT DEFERRABLE IMMEDIATE VALIDATED GENERATED NAME 22-Mar-2013 16:16:22 SYS_C00181594
This is happening when I'm placing messages on the queue, full stack trace follows
Caused by: java.io.IOException: ORA-00001: unique constraint (WEBSVC.SYS_C00181594) violated
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
at org.apache.activemq.store.jdbc.TransactionContext.close(TransactionContext.java:138)
at org.apache.activemq.store.jdbc.JDBCMessageStore.addMessage(JDBCMessageStore.java:129)
at org.apache.activemq.store.memory.MemoryTransactionStore.addMessage(MemoryTransactionStore.java:328)
at org.apache.activemq.store.memory.MemoryTransactionStore$1.asyncAddQueueMessage(MemoryTransactionStore.java:155)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:744)
at org.apache.activemq.broker.region.Queue.send(Queue.java:717)
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:311)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:317)
at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:680)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:152)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:241)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
... 3 more
Caused by: java.sql.BatchUpdateException: ORA-00001: unique constraint (WEBSVC.SYS_C00181594) violated
at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10345)
at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:230)
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297)
at org.apache.activemq.store.jdbc.TransactionContext.executeBatch(TransactionContext.java:103)
at org.apache.activemq.store.jdbc.TransactionContext.executeBatch(TransactionContext.java:81)
at org.apache.activemq.store.jdbc.TransactionContext.close(TransactionContext.java:129)
I have a pretty paranoid DBCP BasicDataSource at the moment i.e.
<bean id="basicOracleDataSource" class="org.apache.commons.dbcp.BasicDataSource"
abstract="true" destroy-method="close">
<property name="driverClassName" value="oracle.jdbc.OracleDriver" />
<property name="poolPreparedStatements" value="true"/>
<property name="initialSize" value="5"/>
<property name="maxActive" value="30"/>
<property name="maxWait" value="60000"/> <!-- default indefinitely -->
<property name="timeBetweenEvictionRunsMillis" value="30000" />
<!-- verify connections in the pool to stop connection closed errors -->
<property name="logAbandoned" value="false"/> <!-- default false -->
<property name="maxIdle" value="5"/> <!-- default 8, minIdle defaults to 0 which is OK -->
<property name="numTestsPerEvictionRun" value="4"/> <!-- default 3 -->
<property name="removeAbandoned" value="true"/> <!-- default false -->
<property name="removeAbandonedTimeout" value="60"/><!-- default 300 seconds -->
<property name="testOnBorrow" value="true"/> <!-- default false -->
<property name="testOnReturn" value="true"/> <!-- default false -->
<property name="testWhileIdle" value="true"/> <!-- default false -->
<property name="validationQuery" value="SELECT 1 FROM dual"/>
</bean>
And just in case, here is my JMS Template too
<!-- Spring JMS Template -->
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory">
<!-- lets wrap in a pool to avoid creating a connection per send -->
<bean class="org.springframework.jms.connection.SingleConnectionFactory">
<property name="targetConnectionFactory">
<ref local="jmsFactory" />
</property>
</bean>
</property>
</bean>
Related
After all server nodes restart and rejoin the baseline, the cache still failed to be read or written:
javax.cache.CacheException: class org.apache.ignite.internal.processors.cache.CacheInvalidStateException: Failed to execute the cache operation (all partition owners have left the grid, partition data has been lost) [cacheName=SQL_DEV_MONI_MONITOR, partition=840, key=UserKeyCacheObjectImpl [part=840, val=iii, hasValBytes=false]]
at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1278)
Partitions owned by nodes is reblanced:
cacheGroups_public_LocalNodeOwningPartitionsCount{iin="fintech-test-grid-mcpl323", inci="7ffed004-2058-4045-875e-bca5ac37e12a", instance="10.16.23.47:9090", job="prometheus"}
544
cacheGroups_public_LocalNodeOwningPartitionsCount{iin="fintech-test-grid-mwpl037", inci="fcea873e-3822-4f1b-ab86-acf71d2e740b", instance="10.16.50.124:9090", job="prometheus"}
382
cacheGroups_public_LocalNodeOwningPartitionsCount{iin="fintech-test-grid-mwpl038", inci="d52c1baf-c34e-4c49-af93-89bff06f55f1", instance="10.16.50.123:9090", job="prometheus"}
210
The data region persistence was enabled, but reset_lost_partitions not works:
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<!--
Default memory region that grows endlessly. A cache is bound to this memory region
unless it sets another one in its CacheConfiguration.
-->
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Default_Region"/>
<property name="persistenceEnabled" value="true"/>
<!-- 1 GB memory region with disabled eviction -->
<property name="initialSize" value="#{100L * 1024 * 1024}"/>
<property name="maxSize" value="#{100L * 1024 * 1024 * 1024}"/>
<!-- Enabling SEGMENTED_LRU page replacement for this region. -->
<property name="pageReplacementMode" value="SEGMENTED_LRU"/>
<property name="metricsEnabled" value="true"/>
<property name="warmUpConfiguration">
<bean class="org.apache.ignite.configuration.LoadAllWarmUpConfiguration"/>
</property>
</bean>
</property>
</bean>
</property>
If a node has permanently left the grid, you need to update the baseline topology. There are a number of ways of doing that, but the simplest is the control script:
./control.sh --baseline
Then
./control.sh --baseline add [node]
./control.sh --baseline remove [node]
I run Apache Ignite to store large data set for computation & retrieval. For now I am trying to see if in-memory itself can address the caching problem.
I have partitioned the cache & set the backup count to 1. I believe, the data will be copied to another node to address any failure, this means, any one of the node goes down, the respective data should be available from the backup node. So, querying the cache should not be affected.
In my setup, when I shutdown one of the node, the data becomes unavailable & query to the cache returns null. Below is my ignite setup (run locally). What's the right way to configure the partition with right backup to address any node failures?
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util" xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean abstract="true" id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Set to true to enable distributed class loading for examples, default is false. -->
<property name="peerClassLoadingEnabled" value="true" />
<property name="rebalanceThreadPoolSize" value="4" />
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="mycache" />
<property name="cacheMode" value="PARTITIONED" />
<property name="backups" value="1" />
<property name="rebalanceMode" value="SYNC" />
<property name="writeSynchronizationMode" value="FULL_SYNC" />
<property name="partitionLossPolicy" value="READ_ONLY_SAFE" />
</bean>
</list>
</property>
<!-- Enable task execution events for examples. -->
<property name="includeEventTypes">
<list>
<!--Task execution events-->
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_TIMEDOUT" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_SESSION_ATTR_SET" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_REDUCED" />
<!--Cache events-->
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_READ" />
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED" />
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
Check that your nodes are a part of the same cluster. You should see info in the logs that the topology includes multiple servers.
Your understanding of the cache configuration is correct - the data should be available after one node goes down. Moreover, with the READ_ONLY_SAFE policy you cannot silently lose data even if the cluster doesn't have enough copies - your cache reads would start throwing errors.
Since you're getting null I'm guessing that your client just reconnected to the second server, which is not connected to the first one (and is therefore empty).
I am facing an issue similar to the one jedis-1929. We are using JedisPool with maxTotal=400 and have ensured that after using jedis from pool.getResource() we are returning the connection back to the pool in finally block using jedis.close() method. Version of jedis is 3.0.0. The issue appears after continuously running the program for several days. We are getting/setting key-value pair in Redis around 0.1m times per minute. The key and values are both quite small, values approx 120 bytes. Use-case is mostly read heavy.
Wanted to set AbandonConfig to ensure leaked connections if any get closed after a default timeout, but don't see a way to set AbandonConfig related settings for JedisPool. Following is the exception we get, when numActives become equal to maxTotal
redis.clients.jedis.exceptions.JedisExhaustedPoolException: Could not get a resource since the pool is exhausted
at redis.clients.jedis.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234)
at com.til.ibeat.connection.RedisConnection.getconnection(RedisConnection.java:52)
at com.til.ibeat.service.impl.RedisCacheServiceImpl.pushToRedisSet(RedisCacheServiceImpl.java:188)
at com.til.ibeat.service.impl.MessageEnrichService.getVisitorType(MessageEnrichService.java:541)
at com.til.ibeat.service.impl.MessageEnrichService.populateVisitorType(MessageEnrichService.java:439)
at com.til.ibeat.service.impl.IbeatEventLogService.process(IbeatEventLogService.java:111)
at com.til.ibeat.service.impl.IbeatEventLogService$2.call(IbeatEventLogService.java:70)
at com.til.ibeat.service.impl.IbeatEventLogService$2.call(IbeatEventLogService.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException: Pool exhausted
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:452)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.jedis.util.Pool.getResource(Pool.java:50)
... 12 common frames omitted
Following is our configuration:
<bean id="redisPool2" class="redis.clients.jedis.JedisPool">
<constructor-arg index="0" type="org.apache.commons.pool2.impl.GenericObjectPoolConfig" ref="poolConfig2"/>
<constructor-arg index="1" type="java.lang.String" value="${jedis2.host}" />
<constructor-arg index="2" type="int" value="${jedis2.port:6379}"/>
<constructor-arg index="3" type="int" value="${jedis2.timeout.millis:200}"/>
<constructor-arg index="4" type="java.lang.String" value="${jedis2.password}" />
</bean>
<bean id="poolConfig2" class="org.apache.commons.pool2.impl.GenericObjectPoolConfig">
<property name="maxTotal" value="${jedis2.max.total:-1}" />
<property name="maxIdle" value="${jedis2.max.idle:50}" />
<property name="minIdle" value="${jedis2.min.idle:3}" />
<property name="testOnBorrow" value="${jedis2.test.on.borrow:true}" />
<property name="testOnReturn" value="${jedis2.test.on.return:false}" />
<property name="testWhileIdle" value="${jedis2.test.while.idle:false}" />
<property name="minEvictableIdleTimeMillis" value="${jedis2.min.evictable.idle.time.millis:2000}" />
<property name="timeBetweenEvictionRunsMillis" value="${jedis2.time.between.eviction.runs:30000}" />
<property name="numTestsPerEvictionRun" value="${jedis2.tests.per.eviction.run:10}" />
<property name="blockWhenExhausted" value="${jedis2.block.when.exhausted:false}" />
<property name="jmxEnabled" value="${jedis2.jmx.enabled:true}"/>
<property name="jmxNamePrefix" value="${jedis2.host}"/>
</bean>
I'm new with Ignite and I'm trying to test data quality and availability of Ignite cluster.
I use the below xml configuration for setting cluster,
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="socketTimeout" value="50000" />
<property name="networkTimeout" value="50000" />
<property name="reconnectCount" value="5" />
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>x.x.x.1:47500..47509</value>
<value>x.x.x.2:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
Also and jthe CacheConfiguration is,
<bean id="cache-template-bean" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="CACHE_TEMPLATE*"/>
<property name="cacheMode" value="PARTITIONED" />
<property name="backups" value="1" />
<!-- <property name="backups" value="2" />
<property name="backups" value="3" /> -->
<property name="atomicityMode" value="TRANSACTIONAL" />
<property name="writeSynchronizationMode" value="PRIMARY_SYNC" />
<property name="rebalanceBatchSize" value="#{4 * 1024 * 1024}" />
<property name="rebalanceMode" value="ASYNC" />
<property name="statisticsEnabled" value="true" />
<property name="rebalanceBatchesPrefetchCount" value="4" />
<property name="defaultLockTimeout" value="5000" />
<property name="readFromBackup" value="true" />
<property name="queryParallelism" value="6" />
<property name="nodeFilter">
<bean class="org.apache.ignite.util.AttributeNodeFilter">
<constructor-arg>
<map>
<entry key="ROLE" value="data.compute"/>
</map>
</constructor-arg>
</bean>
</property>
</bean>
My scenarios are,
Loaded the 5 million data when all the 3 nodes
Bring one node down
The count shows 3.75 million. (Data loss)
Bringing the node up counts 5 million again.
I tried backup 1,2,3 all resulted in the same data loss. As per Ignite documents, appears the data loss should not happen. If this fixed, I can try adding data when the node is down and check how it behaves.
Any suggestions, please?
Ash
The main idea of the baseline topology and persistence is to prevent unnecessary rebalance and store data only in specified server nodes. When a baseline node stopped, it is expected that one will back soon and the rebalance process is not triggered. You could exclude the node from the baseline using api or control.sh utility.
IgniteCache.size() returns the number of primary entries. So when a baseline node is stopped, size() shows a smaller number indicating that a number of primary entries is not accessible.
In your case the data is not lost by two reasons:
1. The data is persisted in backup entries on alive baseline nodes.
2. The primary and backup entries located on the stopped node will back to the cluster after the node started.
[1] https://apacheignite.readme.io/docs/baseline-topology
I am currently trying different connection pools together with eclipselink. The one I am currently configuring is the one by eclipselink itself. Now I am not sure if it is meant to be used in production or if I should use another such as HikariCP? Also I am not sure how to detect if the Eclipselink connection pool is really running. My configuration is the following:
persistence.xml
...
<!-- Configuring Eclipse Link -->
<property name="javax.persistence.jdbc.driver" value="net.sourceforge.jtds.jdbc.Driver"/>
<property name="javax.persistence.jdbc.url" value="jdbc:jtds:sqlserver://SERVER/dev;instance=MSSQL2008R2"/>
<property name="javax.persistence.jdbc.user" value="user"/>
<property name="javax.persistence.jdbc.password" value="password"/>
<property name="eclipselink.persistence-context.flush-mode" value="commit" /> <!-- Flushing occurs at transaction commit. (https://www.eclipse.org/eclipselink/documentation/2.5/jpa/extensions/p_persistence_context_flushmode.htm)-->
<!-- We use the EclipseLink internal components (https://www.eclipse.org/eclipselink/documentation/2.6/jpa/extensions/persistenceproperties_ref.htm)-->
<!-- Eclipselink jdbc configuration -->
<property name="eclipselink.jdbc.batch-writing" value="JDBC"/> <!-- Batch-writing is supported by mssql and our jdbc driver (https://www.eclipse.org/eclipselink/documentation/2.5/jpa/extensions/p_jdbc_batchwriting.htm) -->
<!-- <property name="eclipselink.jdbc.batch-writing.size" value="10000"/> --> <!--Performance testing http://java-persistence-performance.blogspot.ch/2013/05/batch-writing-and-dynamic-vs.html -->
<property name="eclipselink.jdbc.bind-parameters" value="true"/> <!-- Parameterized sql queries -->
<property name="eclipselink.jdbc.cache-statements" value="true"/> <!-- https://docs.oracle.com/middleware/1212/core/ASPER/toplink.htm#ASPER99838 -->
<property name="eclipselink.jdbc.cache-statements.size" value="10000"/>
<property name="eclipselink.jdbc.allow-native-sql-queries" value="false"/> <!-- We explicitly set this, even though it is disabled by multitenancy by default -->
<!-- Eclipselink connection pool configuration-->
<property name="eclipselink.connection-pool.default.initial" value="50" />
<property name="eclipselink.connection-pool.default.min" value="50"/>
<property name="eclipselink.connection-pool.default.max" value="100"/>
<property name="eclipselink.connection-pool.default.url" value="jdbc:jtds:sqlserver://SERVER/dev;instance=MSSQL2008R2"/>
<!-- Eclipselink cache configuration -->
<property name="eclipselink.cache.shared.default" value="true" />
<!-- Eclipselink logging configuration -->
<property name="eclipselink.logging.level" value="OFF"/> <!-- How much log should be shown | from: https://wiki.eclipse.org/EclipseLink/Examples/JPA/Logging#Log_Level_Configuration-->
<property name="eclipselink.logging.level.sql" value="OFF"/> <!-- How to show the sql queries -->
<property name="eclipselink.target-database" value="SQLServer"/> <!-- What sql database is used | from: http://www.eclipse.org/eclipselink/documentation/2.5/jpa/extensions/p_target_database.htm-->
<property name="eclipselink.ddl-generation" value="none"/>
<property name="eclipselink.session.customizer" value="platform.data.EclipseLinkSessionCustomizer"/> <!-- Defines a naming strategy -->
<property name="eclipselink.multitenant.tenants-share-emf" value="false"/>
...
Thank you for your help!