Jedis connections not being released - redis

I am having issues using Redis with a Jedis connection pool.
The behaviour I am seeing is that after connections are established they are not being released back into the connection pool.
I know this because using the redis-cli client list command I can see the connections still hanging around.
Can someone please review my jedis connection pool to let me know if that might be causing the issue.
<spring:bean id="ElasticachePoolConfig" name="ElasticachePoolConfig" class="redis.clients.jedis.JedisPoolConfig" >
<!-- Minimum number of idle connections to Redis - these can be seen as always open and ready to serve -->
<spring:property name="minIdle" value="${redis.minIdle}" />
<!-- Number of connections to Redis that just sit there and do nothing -->
<spring:property name="maxIdle" value="${redis.maxIdle}" />
<!-- Maximum number of active connections that can be allocated from this pool at the same time -->
<spring:property name="maxTotal" value="${redis.maxTotal}" />
<!-- The minimum amount of time an object may sit idle in the pool before it is eligable for eviction by the idle object evictor.-->
<spring:property name="minEvictableIdleTimeMillis" value="${redis.minEvictableIdleTimeMillis}" />
<!-- The minimum amount of time a connection may sit idle in the pool before it is eligible for eviction by the idle connection evictor -->
<spring:property name="softMinEvictableIdleTimeMillis" value="${redis.softMinEvictableIdleTimeMillis}" />
<!-- The maximum number of milliseconds that the pool will wait (when there are no available connections) for a connection to be returned before throwing an exception -->
<spring:property name="maxWaitMillis" value="${redis.maxWaitMillis}" />
<!-- Maximum number of connections to test in each idle check -->
<spring:property name="numTestsPerEvictionRun" value="${redis.numTestsPerEvictionRun}" />
<!-- Tests whether connection is dead when connection retrieval method is called -->
<spring:property name="testOnBorrow" value="${redis.testOnBorrow}" />
<!-- Tests whether connection is dead when returning a connection to the pool -->
<spring:property name="testOnReturn" value="${redis.testOnReturn}" />
<!-- Tests whether connections are dead during idle periods -->
<spring:property name="testWhileIdle" value="${redis.testWhileIdle}" />
<!-- Idle connection checking period -->
<spring:property name="timeBetweenEvictionRunsMillis" value="${redis.timeBetweenEvictionRunsMillis}" />
<spring:property name="blockWhenExhausted" value="${redis.blockWhenExhausted}" />
</spring:bean>
Where the properties are set as follows...
redis.port=6379
redis.timeout=2000
redis.ttl=600
redis.host=localhost
redis.repeater.maxRetries=3
redis.repeater.millisBetweenRetries=2000
redis.minIdle=5
redis.maxIdle=10
redis.maxTotal=8
redis.minEvictableIdleTimeMillis=30000
redis.softMinEvictableIdleTimeMillis=-1
redis.maxWaitMillis=5000
redis.numTestsPerEvictionRun=10
redis.testOnBorrow=true
redis.testOnReturn=true
redis.testWhileIdle=false
redis.timeBetweenEvictionRunsMillis=50000
redis.blockWhenExhausted=true

Your maxIdle setting is larger than your maxTotal setting, so effectively no connections will be released on idle. The pool will maintain maxIdle connections.
JedisPool uses Apache Commons Pooling internally. See this answer regarding the differences between maxIdle and maxTotal.

Related

Mule exception strategy

On one particular queue alone the Mule JMS consumer count goes to 0 and comes back to default numbers after a point in time
My question is, why is it going down to 0 and I have message stuck in the queue? After few moments I find it comes back to 32 (default numberOfConcurrentTransactedRecievers) and processes the message and then again goes back to 0 consumers. What could be the problem?
I've other consumers in the same mule which are able to process cleanly without the count reducing.
My Configs:
<!-- ActiveMQ Connection factory -->
<spring:bean id="domainConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL" value="tcp://78.99.88.72:61616?jms.prefetchPolicy.all=2&jms.useAsyncSend=true&socketBufferSize=131072" />
<spring:property name="redeliveryPolicy" ref="domainRedeliveryPolicy" />
</spring:bean>
<!-- amqExceptionConnectionFactory Connection factory -->
<spring:bean id="amqExceptionConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" lazy-init="true">
<spring:property name="brokerURL" value="failover:(${domain.amq.failover.url}?jms.prefetchPolicy.all=2)" />
<spring:property name="redeliveryPolicy" ref="domainRedeliveryPolicy" />
</spring:bean>
<jms:activemq-connector
name="com.KKT.jmsConnector"
username="${domain.amq.user.id}"
password="${domain.amq.user.password}"
connectionFactory-ref="pooledConnectionFactory"
validateConnections="true"
numberOfConcurrentTransactedReceivers="32"
acknowledgementMode="AUTO_ACKNOWLEDGE"
persistentDelivery="false"
doc:name="AMQ Connector for KKTInterface"
specification="1.1">
<receiver-threading-profile maxThreadsActive="32" poolExhaustedAction="WAIT"/>
<dispatcher-threading-profile maxThreadsActive="128" poolExhaustedAction="WAIT"/>
<reconnect count="15" frequency="5000" blocking="true"/>
</jms:activemq-connector>
And I see the below error:
mule.log.5:2020-05-29 20:24:49,392 [ERROR] KKT.router [ActiveMQ Session Task-6] | ed938f60-a1d8-11ea-8ad7-525400aef612: KKT Interface: Unrecoverable Error: Please check ASAP: org.mule.transport.jms.redelivery.MessageRedeliveredException: "Message with id "ID:localhost-35511-1590775568949-1:1:162:1:3367" has been redelivered 1 times on endpoint "jms://com.KKT.router.inbox", which exceeds the maxRedelivery setting of 0 on the connector "com.KKT.router.jmsConnector".
mule.log.5:2020-05-29 20:24:49,216 [ERROR] exception.CatchMessagingExceptionStrategy [ActiveMQ Session Task-5] |
mule.log.5:2020-05-29 20:24:49,216 [ERROR] exception.CatchMessagingExceptionStrategy [ActiveMQ Session Task-4] |
mule.log.9:2020-05-29 20:24:32,753 [ERROR] exception.CatchMessagingExceptionStrategy [ActiveMQ Session Task-57] |
mule.log.9:2020-05-29 20:24:32,753 [ERROR] KKT.router [ActiveMQ Session Task-57] | db927d80-a1d8-11ea-8ad7-525400aef612: KKT Interface: Unrecoverable Error: Please check ASAP: org.mule.api.service.FailedToQueueEventException: Interrupted while queueing event for "SEDA Stage com.KKT.router.requestGateway.stage1".
The issue seems to be that some error is triggering a redelivery of the message. When the redelivery count exceeds the maximum then you get a redelivery error.
You should get familiar with the JMS connector redelivery configuration: https://docs.mulesoft.com/mule-runtime/3.9/jms-transport-reference. Then check the Rollback Exception strategy https://docs.mulesoft.com/mule-runtime/3.9/rollback-exception-strategy on how you can handle those errors.
Note that ActiveMQ may have its own redelivery configurations.

How to put the message got from jdbc to rabbitmq?

In the spring integration reference on JDBC Support
it has the following code to get the records out of a table and convert them into message and sent them to a channel. But my system is rabbitmq so I want the message from jdbc stored in rabbitmq and consumed by rabbitmq client. How to do that using spring integration and spring rabbitmq?
<int-jdbc:inbound-channel-adapter query="select * from item where status=2"
channel="target" data-source="dataSource"
update="update item set status=10 where id in (:id)" />
I know there are AMQP Backed Message Channels, and defined as below.
<int-amqp:channel id="p2pChannel"/>
But how to put them together? So that the message created from a table will be stored in a rabbitmq queue?
update: I used the following code:
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.postgresql.Driver"/>
<property name="url" value="jdbc:postgresql://localhost/example"/>
<property name="username" value="postgres"/>
<property name="password" value="oracle"/>
</bean>
<int-amqp:channel id="myMessageChannel_test"/>
<int-jdbc:inbound-channel-adapter query="select * from my_files where message_created=0"
channel="myMessageChannel_test" data-source="dataSource"
update="update my_files set message_created=1 where file_id in (:file_id)">
<int:poller fixed-rate="1000">
<int:transactional/>
</int:poller>
</int-jdbc:inbound-channel-adapter>
But it complains:
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for amqp-channel 'myMessageChannel_test'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.amqp.channel.AbstractSubscribableAmqpChannel$DispatchingMessageListener.onMessage(AbstractSubscribableAmqpChannel.java:197)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:737)
... 10 common frames omitted
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:107)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.amqp.channel.AbstractSubscribableAmqpChannel$DispatchingMessageListener.onMessage(AbstractSubscribableAmqpChannel.java:181)
... 11 common frames omitted
2014-12-30 19:23:18.985 WARN 29203 --- [cTaskExecutor-1] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener threw exception
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:802)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:740)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:628)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1196)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:600)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:960)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:944)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1058)
So your channel "myMessageChannel_test" has messages from database but that needs some config like below to be able to receive/handle/subscribe to messages.
There are many ways to do it - one way as below
<rabbit:connection-factory id="rabbitConnFactory" addresses="${rabbit.addresses}"
virtual-host="${rabbit.host}" username="${rabbit.username}" password="${rabbit.password}"/>
<rabbit:admin id="admin" connection-factory="rabbitConnFactory" />
<rabbit:template id="amqpTemplate" connection-factory="rabbitConnFactory" routing-key="some.key" exchange="some.exchange" queue="some.queue" />
<int-amqp:outbound-channel-adapter routing-key="some.key" amqp-template="amqpTemplate"
channel="myMessageChannel_test" />
You need to do respective rabbit configs as well

C3p0 APPARENT DEADLOCK exception even though statement caching is disabled

I am using C3p0 connection pool and I am observing following exception in my server.
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#ed866f -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 1
Active Threads: 1
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask#995ec2 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#11c3ceb
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#e392ed
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#c33853
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#a6f77d
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#1c25bfc
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask#12534a9
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask#1a4564c
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
oracle.jdbc.driver.OracleStatement.close(OracleStatement.java:1338)
com.mchange.v2.c3p0.impl.NewPooledConnection.cleanupUncachedStatements(NewPooledConnection.java:651)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:539)
com.mchange.v2.c3p0.impl.NewPooledConnection.close(NewPooledConnection.java:234)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.destroyResource(C3P0PooledConnectionPool.java:470)
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask.run(BasicResourcePool.java:964)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
My configuration is
<property name="initialPoolSize" value="1" />
<property name="minPoolSize" value="1" />
<property name="maxPoolSize" value="20" />
<property name="maxIdleTime" value="240" />
<property name="checkoutTimeout" value="60000" />
<property name="acquireRetryAttempts" value="0" />
<property name="acquireRetryDelay" value="1000" />
<property name="debugUnreturnedConnectionStackTraces" value="true" />
<property name="unreturnedConnectionTimeout" value="300" />
<property name="numHelperThreads" value="1" />
<property name="preferredTestQuery" value="SELECT 1 FROM DUAL" />
After some googing I found following threads.
https://forum.hibernate.org/viewtopic.php?t=947246
It is suggested to disable the statement caching to avoid this problem. But in my configuration I didn't enable the statement caching(maxStatements, maxStatementsPerConnection are 0 by default).
Please suggest any alternatives.
Don't set numHelperThreads to 1.
If you set numHelperThreads to 1, any slow task will trigger an APPARENT DEADLOCK. c3p0 relies on an internal Thread pool. One thread does not make a pool. Let numHelperThreads have at least its default value of 3.
Specifically what is going on in your stack trace is that an attempt by the Connection pool to close() a Connection is taking long time, for some reason. There could be a real problem here, if this happens a lot, but it could just be an occasional slow operation, which a reasonably sized Thread pool would just handle and shrug off.
Good luck!

Restarting activemq route in camel when receiving a "Connection.start()" message

I have a an activemq camel route that stops receiving messages at some point and requires a restart. I am not sure how to programmatically detect and fix this situation.
My route looks like so:
from("activemq:queue:Consumer.app.VirtualTopic.msg?messageConverter=#convertMsg")
It's all configured in Spring like so:
<!-- Configure the Message Bus Factory -->
<bean id="jmsFactory" class="com.local.messaging.activemq.SpringSslContextConnectionFactory">
<property name="brokerURL" value="${jms.broker.url}" />
<property name="sslContext" ref="sslContext" />
</bean>
<!-- Connect the Message Bus Factory to Camel. The 'activemq' bean
name is necessary for Camel to pick it up automatically -->
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" depends-on="jmsFactory">
<property name="usePooledConnection" value="true" />
<property name="connectionFactory">
<bean class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="20" />
<property name="maximumActive" value="10" />
<property name="connectionFactory" ref="jmsFactory" />
</bean>
</property>
</bean>
Finally, the broker URL is configured like so:
jms.broker.url=failover://(ssl://amq1:61616,ssl://amq1:61616)
This starts up just fine and works like a champ most of the time. Every so often, though, I see this message in the logs:
Received a message on a connection which is not yet started. Have you forgotten to call Connection.start()? Connection: ActiveMQConnection {<details>}
I suspect strongly that this happens after the message bus has restarted, but as I have no direct access to the message bus, I don't know that for certain. I don't know that that matters.
The keys for me are:
How do I programmatically detect this situation? There doesn't appear to be any exception thrown or the like and the only way I've seen this is by parsing through log files.
After detecting it, how do I fix it? Do I need to start() and stop() the route or is there a cleaner way?
Finally, I did see some suggestions that this case should be handled by activemq, using the failover scheme. As shown above, I am using failover and this still happens.

How to set the ActiveMQ redeliveryPolicy on a queue?

How do I set the redeliveryPolicy in ActiveMQ on a Queue?
1) In the doc, see: activeMQ Redelivery, the explain that you should set it on the ConnectionFactory or Connection. But I want to use different value's for different Queue's.
2) Apart from that, I don't seem to get it work. Setting it on the connection factory in Spring (I am using activemq 5.4.2. with Spring 3.0) like this don't seem to have any effect:
<amq:connectionFactory id="amqConnectionFactory" brokerURL="${jms.factory.url}" >
<amq:properties>
<amq:redeliveryPolicy maximumRedeliveries="6" initialRedeliveryDelay="15000" useExponentialBackOff="true" backOffMultiplier="5"/>
</amq:properties>
</amq:connectionFactory>
I also tried to set it as property on the defined Queue, but that also seem to be ignored as the redelivery occurs sooner that the defined values:
<amq:queue id="jmsQueueDeclarationSnd" physicalName="${jms.queue.declaration.snd}" >
<amq:properties>
<amq:redeliveryPolicy maximumRedeliveries="6" initialRedeliveryDelay="15000" useExponentialBackOff="true" backOffMultiplier="5"/>
</amq:properties>
</amq:queue>
Thanks
I too was using the method shown by Ivan above for amq:connectionFactory
Whilst upgrading to ActiveMQ 5.7.0 I noticed this no longer works (since the implementation of https://issues.apache.org/jira/browse/AMQ-3224). Anyway after reading a better post on the ActiveMQ forums I currently use :-
<amq:queue id="emailQueue" physicalName="emailQueue" />
<amq:queue id="smsQueue" physicalName="smsQueue" />
<!-- Wait 15 seconds first re-delivery, then 45, 135, 405, 1215, 3645 seconds -->
<bean id="redeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<property name="backOffMultiplier" value="3" />
<property name="initialRedeliveryDelay" value="15000" />
<property name="maximumRedeliveries" value="6" />
<property name="queue" value="*" />
<property name="redeliveryDelay" value="15000" />
<property name="useExponentialBackOff" value="true" />
</bean>
<amq:connectionFactory id="jmsFactory" brokerURL="yourProtocol/BrokerURL">
<property name="redeliveryPolicy" ref="redeliveryPolicy" />
</amq:connectionFactory>
Note that for any messages that fail to be redelivered after 6 retries, ActiveMQ will create a DLQ.emailQueue' or DLQ.smsQueue and enqueue the message on that queue (dequeuing it from the original queue).
I got it working by setting it on the factory as done above but only when creating the connection factory as a Spring bean and not through XBean as shown above. This is because the xsd doesn't allow you to set the redeliveryPolicy as an object, but merely as a String.
After setting the cache level to Consumer in Spring's DefaultMessageListenerContainer, it all worked.
On the queue , it seems that you simple can set a delivery policy... Strange, as I would like to have different settings for different queue's/topics. Just imagine you have a slow and faster queue, or a external system that you connect to that needs more time to recover..
Maybe this feature is still to be implemented
You can set the redeliveryPolicy within the amq namespace like this:
<amq:connectionFactory id="jmsRedeliverConnectionFactory" brokerURL="vm://localhost">
<amq:redeliveryPolicy>
<amq:redeliveryPolicy maximumRedeliveries="5" initialRedeliveryDelay="1000" useExponentialBackOff="true" backOffMultiplier="5" />
</amq:redeliveryPolicy>
</amq:connectionFactory>
I could not get ActiveMQ (5.7.0) to recognize my redelivery policy when I defined it using <amq:properties> on the ConnectionFactory or the Queue (it kept using the default redelivery policy). What worked for me is this:
Create the RedeliveryPolicy as a standalone bean, then Spring-reference it in the ConnectionFactory
Create an explicit DLQ and Spring-reference it in the RedeliveryPolicy
Spring config as follows:
<amq:connectionFactory id="jmsFactory" brokerURL="vm://localhost" redeliveryPolicy="#activeMQRedeliveryPolicy" />
<amq:redeliveryPolicy id="activeMQRedeliveryPolicy" destination="#myDLQ" useExponentialBackOff="true" backOffMultiplier="3" maximumRedeliveries="4" />
<amq:queue id="myDLQ" physicalName="DLQ.myDLQ" />