I am using C3p0 connection pool and I am observing following exception in my server.
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#ed866f -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 1
Active Threads: 1
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#995ec2 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#11c3ceb
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#e392ed
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#c33853
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#a6f77d
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1c25bfc
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#12534a9
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#1a4564c
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
oracle.jdbc.driver.OracleStatement.close(OracleStatement.java:1338)
com.mchange.v2.c3p0.impl.NewPooledConnection.cleanupUncachedStatements(NewPooledConnection.java:651)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:539)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:234)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:470)
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:964)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
My configuration is
<property name="initialPoolSize" value="1" />
<property name="minPoolSize" value="1" />
<property name="maxPoolSize" value="20" />
<property name="maxIdleTime" value="240" />
<property name="checkoutTimeout" value="60000" />
<property name="acquireRetryAttempts" value="0" />
<property name="acquireRetryDelay" value="1000" />
<property name="debugUnreturnedConnectionStackTraces" value="true" />
<property name="unreturnedConnectionTimeout" value="300" />
<property name="numHelperThreads" value="1" />
<property name="preferredTestQuery" value="SELECT 1 FROM DUAL" />
After some googing I found following threads.
https://forum.hibernate.org/viewtopic.php?t=947246
It is suggested to disable the statement caching to avoid this problem. But in my configuration I didn't enable the statement caching(maxStatements, maxStatementsPerConnection are 0 by default).
Please suggest any alternatives.
Don't set numHelperThreads to 1.
If you set numHelperThreads to 1, any slow task will trigger an APPARENT DEADLOCK. c3p0 relies on an internal Thread pool. One thread does not make a pool. Let numHelperThreads have at least its default value of 3.
Specifically what is going on in your stack trace is that an attempt by the Connection pool to close() a Connection is taking long time, for some reason. There could be a real problem here, if this happens a lot, but it could just be an occasional slow operation, which a reasonably sized Thread pool would just handle and shrug off.
Good luck!
Related
I am facing an issue similar to the one jedis-1929. We are using JedisPool with maxTotal=400 and have ensured that after using jedis from pool.getResource() we are returning the connection back to the pool in finally block using jedis.close() method. Version of jedis is 3.0.0. The issue appears after continuously running the program for several days. We are getting/setting key-value pair in Redis around 0.1m times per minute. The key and values are both quite small, values approx 120 bytes. Use-case is mostly read heavy.
Wanted to set AbandonConfig to ensure leaked connections if any get closed after a default timeout, but don't see a way to set AbandonConfig related settings for JedisPool. Following is the exception we get, when numActives become equal to maxTotal
redis.clients.jedis.exceptions.JedisExhaustedPoolException: Could not get a resource since the pool is exhausted
at redis.clients.jedis.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234)
at com.til.ibeat.connection.RedisConnection.getconnection(RedisConnection.java:52)
at com.til.ibeat.service.impl.RedisCacheServiceImpl.pushToRedisSet(RedisCacheServiceImpl.java:188)
at com.til.ibeat.service.impl.MessageEnrichService.getVisitorType(MessageEnrichService.java:541)
at com.til.ibeat.service.impl.MessageEnrichService.populateVisitorType(MessageEnrichService.java:439)
at com.til.ibeat.service.impl.IbeatEventLogService.process(IbeatEventLogService.java:111)
at com.til.ibeat.service.impl.IbeatEventLogService$2.call(IbeatEventLogService.java:70)
at com.til.ibeat.service.impl.IbeatEventLogService$2.call(IbeatEventLogService.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException: Pool exhausted
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:452)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.jedis.util.Pool.getResource(Pool.java:50)
... 12 common frames omitted
Following is our configuration:
<bean id="redisPool2" class="redis.clients.jedis.JedisPool">
<constructor-arg index="0" type="org.apache.commons.pool2.impl.GenericObjectPoolConfig" ref="poolConfig2"/>
<constructor-arg index="1" type="java.lang.String" value="${jedis2.host}" />
<constructor-arg index="2" type="int" value="${jedis2.port:6379}"/>
<constructor-arg index="3" type="int" value="${jedis2.timeout.millis:200}"/>
<constructor-arg index="4" type="java.lang.String" value="${jedis2.password}" />
</bean>
<bean id="poolConfig2" class="org.apache.commons.pool2.impl.GenericObjectPoolConfig">
<property name="maxTotal" value="${jedis2.max.total:-1}" />
<property name="maxIdle" value="${jedis2.max.idle:50}" />
<property name="minIdle" value="${jedis2.min.idle:3}" />
<property name="testOnBorrow" value="${jedis2.test.on.borrow:true}" />
<property name="testOnReturn" value="${jedis2.test.on.return:false}" />
<property name="testWhileIdle" value="${jedis2.test.while.idle:false}" />
<property name="minEvictableIdleTimeMillis" value="${jedis2.min.evictable.idle.time.millis:2000}" />
<property name="timeBetweenEvictionRunsMillis" value="${jedis2.time.between.eviction.runs:30000}" />
<property name="numTestsPerEvictionRun" value="${jedis2.tests.per.eviction.run:10}" />
<property name="blockWhenExhausted" value="${jedis2.block.when.exhausted:false}" />
<property name="jmxEnabled" value="${jedis2.jmx.enabled:true}"/>
<property name="jmxNamePrefix" value="${jedis2.host}"/>
</bean>
I have a slightly strange warning from ActiveMQ 5.9.0 with JDBC Oracle backed persistence...
WARN [org.apache.activemq.store.jdbc.LeaseDatabaseLocker] LockableService
keep alive period: 2000, which renews the lease, is less than
lockAcquireSleepInterval: 1000, the lease duration.
These values will allow the lease to expire.
My question is why is LockableService reporting that 2000 < 1000? I think it should say "LockableService keep alive period: 2000, which renews the lease, is greater than lockAcquireSleepInterval: 1000, the lease duration. These values will allow the lease to expire.". What do you think, maybe I'm reading this wrong...
I do see a problem with my current settings (I have a Master and Slave, I shutdown the Master and the Slave takes over, but I startup the Master again and it doesn't become a Slave)... So I obviously need to tweak my settings, here is the current relevant configuration...
<bean id="jdbcPersistenceAdapter" class="org.apache.activemq.store.jdbc.JDBCPersistenceAdapter">
<property name="brokerName" value="messageCentreBroker" />
<property name="createTablesOnStartup" value="true" />
<property name="dataSource" ref="activeMqDataSource" />
<property name="lockKeepAlivePeriod" value="2000" />
<property name="locker" ref="leaseDatabaseLocker" />
</bean>
<bean id="leaseDatabaseLocker" class="org.apache.activemq.store.jdbc.LeaseDatabaseLocker">
<property name="lockAcquireSleepInterval" value="1000" />
</bean>
So I guess my lockAcquireSleepInterval should be greater than 2000? I'll try this, but interested to hear thoughts on the WARN message too, it seems wrong?
I think it's just a typo. Nothing more.
Our system will not shutdown when a "Stop" command is issued from the Tomcat Manager. I have determined that it is related to ActiveMQ/Spring. I have even figured out how to get it to shutdown, however my solution is a hack (at least I hope this isn't the "correct" way to do it). I would like to know the proper way to shutdown ActiveMQ so that I can remove my hack.
I inherited this component and I have no information about why certain architectural decisions were made, after a lot of digging I think I understand his thoughts, but I could be missing something. In other words, the real problem could be in the way that we are trying to use ActiveMQ/Spring.
We run in ServletContainer (Tomcat 6/7) and use ActiveMQ 5.9.1 and Spring 3.0.0 Multiple instances of our application can run in a "group", with each instance running on it's own server. ActiveMQ is used to facilitate communication between the multiple instances. Each instance has it's own embedded broker and it's own set of queues. Every queue on every instance has exactly 1 org.springframework.jms.listener.DefaultMessageListenerContainer listening to it, so 5 queues = 5 DefaultMessageListenerContainers for example.
Our system shut down properly until we fixed a bug by adding queuePrefetch="0" to the ConnectionFactory. At first I assumed that this change was incorrect in some way, but now that I understand the situation, I am confident that we should not be using the prefetch functionality.
I have created a test application to replicate the issue. Note that the information below makes no mention of message producers. That is because I can replicate the issue without ever sending/processing a single message. Simply creating the Broker, ConnectionFactory, Queues and Listeners during boot, is enough to keep the system from stopping properly.
Here is my sample configuration from my Spring XML. I will be happy to provide my entire project if someone wants it:
<amq:broker persistent="false" id="mybroker">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://0.0.0.0:61616"/>
</amq:transportConnectors>
</amq:broker>
<amq:connectionFactory id="ConnectionFactory" brokerURL="vm://localhost?broker.persistent=false" >
<amq:prefetchPolicy>
<amq:prefetchPolicy queuePrefetch="0"/>
</amq:prefetchPolicy>
</amq:connectionFactory>
<amq:queue id="lookup.mdb.queue.cat" physicalName="DogQueue"/>
<amq:queue id="lookup.mdb.queue.dog" physicalName="CatQueue"/>
<amq:queue id="lookup.mdb.queue.fish" physicalName="FishQueue"/>
<bean id="messageListener" class="org.springframework.jms.listener.DefaultMessageListenerContainer" abstract="true">
<property name="connectionFactory" ref="ConnectionFactory"/>
</bean>
<bean parent="messageListener" id="cat">
<property name="destination" ref="lookup.mdb.queue.dog"/>
<property name="messageListener">
<bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
</property>
<property name="concurrentConsumers" value="200"/>
<property name="maxConcurrentConsumers" value="200"/>
</bean>
<bean parent="messageListener" id="dog">
<property name="destination" ref="lookup.mdb.queue.cat"/>
<property name="messageListener">
<bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
</property>
<property name="concurrentConsumers" value="200"/>
<property name="maxConcurrentConsumers" value="200"/>
</bean>
<bean parent="messageListener" id="fish">
<property name="destination" ref="lookup.mdb.queue.fish"/>
<property name="messageListener">
<bean class="com.acteksoft.common.remote.jms.WorkerMessageListener"/>
</property>
<property name="concurrentConsumers" value="200"/>
<property name="maxConcurrentConsumers" value="200"/>
</bean>
My hack involves using a ServletContextListener to manually stop the objects. The hacky part is that I have to create additional threads to stop the DefaultMessageListenerContainers. Perhaps I'm stopping the objects in the wrong order, but I've tried everything that I can imagine. If I attempt to stop the objects in the main thread, then they hang indefinitely.
Thank you in advance!
UPDATE
I have tried the following based on boday's recommendation but it didn't work. I have also tried to specify the amq:transportConnector uri as tcp://0.0.0.0:61616?transport.daemon=true
<amq:broker persistent="false" id="mybroker" brokerName="localhost">
<amq:transportConnectors>
<amq:transportConnector uri="tcp://0.0.0.0:61616?daemon=true"/>
</amq:transportConnectors>
</amq:broker>
<amq:connectionFactory id="connectionFactory" brokerURL="vm://localhost" >
<amq:prefetchPolicy>
<amq:prefetchPolicy queuePrefetch="0"/>
</amq:prefetchPolicy>
</amq:connectionFactory>
At one point I tried to add similar properties to the brokerUrl parameter in the amq:connectionFactory element and the shutdown worked properly, however after further testing I learned that the properties were resulting in an exception to be thrown from VMTransportFactory. This resulted in improper initialization and the basic message functionality didn't work.
In case anyone else is wondering, as far as I can see it's not possible to have a daemon ListenerContainer using ActiveMQ.
When the ActiveMQConnection is started, it creates a ThreadPoolExecutor with non-daemon thread. This is seemingly to avoid issues when failing over the connection from one broker to another.
https://issues.apache.org/jira/browse/AMQ-796
executor = new ThreadPoolExecutor(1, 1, 5, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
#Override
public Thread newThread(Runnable r) {
Thread thread = new Thread(r, "ActiveMQ Connection Executor: " + transport);
//Don't make these daemon threads - see https://issues.apache.org/jira/browse/AMQ-796
//thread.setDaemon(true);
return thread;
}
});
try setting daemon=true on your TCP transport, this allows the process to run as a deamon thread which won't block the shutdown of your container
see http://activemq.apache.org/tcp-transport-reference.html
I have a an activemq camel route that stops receiving messages at some point and requires a restart. I am not sure how to programmatically detect and fix this situation.
My route looks like so:
from("activemq:queue:Consumer.app.VirtualTopic.msg?messageConverter=#convertMsg")
It's all configured in Spring like so:
<!-- Configure the Message Bus Factory -->
<bean id="jmsFactory" class="com.local.messaging.activemq.SpringSslContextConnectionFactory">
<property name="brokerURL" value="${jms.broker.url}" />
<property name="sslContext" ref="sslContext" />
</bean>
<!-- Connect the Message Bus Factory to Camel. The 'activemq' bean
name is necessary for Camel to pick it up automatically -->
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" depends-on="jmsFactory">
<property name="usePooledConnection" value="true" />
<property name="connectionFactory">
<bean class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="20" />
<property name="maximumActive" value="10" />
<property name="connectionFactory" ref="jmsFactory" />
</bean>
</property>
</bean>
Finally, the broker URL is configured like so:
jms.broker.url=failover://(ssl://amq1:61616,ssl://amq1:61616)
This starts up just fine and works like a champ most of the time. Every so often, though, I see this message in the logs:
Received a message on a connection which is not yet started. Have you forgotten to call Connection.start()? Connection: ActiveMQConnection {<details>}
I suspect strongly that this happens after the message bus has restarted, but as I have no direct access to the message bus, I don't know that for certain. I don't know that that matters.
The keys for me are:
How do I programmatically detect this situation? There doesn't appear to be any exception thrown or the like and the only way I've seen this is by parsing through log files.
After detecting it, how do I fix it? Do I need to start() and stop() the route or is there a cleaner way?
Finally, I did see some suggestions that this case should be handled by activemq, using the failover scheme. As shown above, I am using failover and this still happens.
How do I set the redeliveryPolicy in ActiveMQ on a Queue?
1) In the doc, see: activeMQ Redelivery, the explain that you should set it on the ConnectionFactory or Connection. But I want to use different value's for different Queue's.
2) Apart from that, I don't seem to get it work. Setting it on the connection factory in Spring (I am using activemq 5.4.2. with Spring 3.0) like this don't seem to have any effect:
<amq:connectionFactory id="amqConnectionFactory" brokerURL="${jms.factory.url}" >
<amq:properties>
<amq:redeliveryPolicy maximumRedeliveries="6" initialRedeliveryDelay="15000" useExponentialBackOff="true" backOffMultiplier="5"/>
</amq:properties>
</amq:connectionFactory>
I also tried to set it as property on the defined Queue, but that also seem to be ignored as the redelivery occurs sooner that the defined values:
<amq:queue id="jmsQueueDeclarationSnd" physicalName="${jms.queue.declaration.snd}" >
<amq:properties>
<amq:redeliveryPolicy maximumRedeliveries="6" initialRedeliveryDelay="15000" useExponentialBackOff="true" backOffMultiplier="5"/>
</amq:properties>
</amq:queue>
Thanks
I too was using the method shown by Ivan above for amq:connectionFactory
Whilst upgrading to ActiveMQ 5.7.0 I noticed this no longer works (since the implementation of https://issues.apache.org/jira/browse/AMQ-3224). Anyway after reading a better post on the ActiveMQ forums I currently use :-
<amq:queue id="emailQueue" physicalName="emailQueue" />
<amq:queue id="smsQueue" physicalName="smsQueue" />
<!-- Wait 15 seconds first re-delivery, then 45, 135, 405, 1215, 3645 seconds -->
<bean id="redeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<property name="backOffMultiplier" value="3" />
<property name="initialRedeliveryDelay" value="15000" />
<property name="maximumRedeliveries" value="6" />
<property name="queue" value="*" />
<property name="redeliveryDelay" value="15000" />
<property name="useExponentialBackOff" value="true" />
</bean>
<amq:connectionFactory id="jmsFactory" brokerURL="yourProtocol/BrokerURL">
<property name="redeliveryPolicy" ref="redeliveryPolicy" />
</amq:connectionFactory>
Note that for any messages that fail to be redelivered after 6 retries, ActiveMQ will create a DLQ.emailQueue' or DLQ.smsQueue and enqueue the message on that queue (dequeuing it from the original queue).
I got it working by setting it on the factory as done above but only when creating the connection factory as a Spring bean and not through XBean as shown above. This is because the xsd doesn't allow you to set the redeliveryPolicy as an object, but merely as a String.
After setting the cache level to Consumer in Spring's DefaultMessageListenerContainer, it all worked.
On the queue , it seems that you simple can set a delivery policy... Strange, as I would like to have different settings for different queue's/topics. Just imagine you have a slow and faster queue, or a external system that you connect to that needs more time to recover..
Maybe this feature is still to be implemented
You can set the redeliveryPolicy within the amq namespace like this:
<amq:connectionFactory id="jmsRedeliverConnectionFactory" brokerURL="vm://localhost">
<amq:redeliveryPolicy>
<amq:redeliveryPolicy maximumRedeliveries="5" initialRedeliveryDelay="1000" useExponentialBackOff="true" backOffMultiplier="5" />
</amq:redeliveryPolicy>
</amq:connectionFactory>
I could not get ActiveMQ (5.7.0) to recognize my redelivery policy when I defined it using <amq:properties> on the ConnectionFactory or the Queue (it kept using the default redelivery policy). What worked for me is this:
Create the RedeliveryPolicy as a standalone bean, then Spring-reference it in the ConnectionFactory
Create an explicit DLQ and Spring-reference it in the RedeliveryPolicy
Spring config as follows:
<amq:connectionFactory id="jmsFactory" brokerURL="vm://localhost" redeliveryPolicy="#activeMQRedeliveryPolicy" />
<amq:redeliveryPolicy id="activeMQRedeliveryPolicy" destination="#myDLQ" useExponentialBackOff="true" backOffMultiplier="3" maximumRedeliveries="4" />
<amq:queue id="myDLQ" physicalName="DLQ.myDLQ" />