Redis, Jedis connection pool optimizeand monitoring tools - redis

Question

I am new to Redis, Jedis. Today, I saw an error in my log
Caused by: redis.clients.jedis.exceptions.JedisExhaustedPoolException: redis.clients.jedis.exceptions.JedisExhaustedPoolException: Could not get a resource since the pool is exhausted
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_191]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_191]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_191]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735) ~[?:1.8.0_191]
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160) ~[?:1.8.0_191]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174) ~[?:1.8.0_191]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) ~[?:1.8.0_191]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_191]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583) ~[?:1.8.0_191]
at com.nokia.snmpapp.VariableDataUtils.processSnmpDataByOP(VariableDataUtils.java:372) ~[snmpapp.jar:?]
at com.nokia.snmpapp.consumerservice.DocsisQos3MibData.processDocsQos3ParamSetTableObjectEvent(DocsisQos3MibData.java:172) ~[snmpapp.jar:?]
at com.nokia.snmpapp.KafkaConsumer.onSnmpappDocsQos3ParamSetTableObjectEvent(KafkaConsumer.java:315) ~[snmpapp.jar:?]
... 16 more
Caused by: redis.clients.jedis.exceptions.JedisExhaustedPoolException: Could not get a resource since the pool is exhausted
at redis.clients.jedis.util.Pool.getResource(Pool.java:53) ~[snmpapp.jar:?]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234) ~[snmpapp.jar:?]
at com.nokia.snmpapp.RedisUtils.add2list(RedisUtils.java:27) ~[snmpapp.jar:?]
at com.nokia.snmpapp.VariableDataUtils.lambda$processSnmpDataByOP$5(VariableDataUtils.java:379) ~[snmpapp.jar:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:1.8.0_191]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_191]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_191]
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) ~[?:1.8.0_191]
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[?:1.8.0_191]
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:439) ~[snmpapp.jar:?]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349) ~[snmpapp.jar:?]
at redis.clients.jedis.util.Pool.getResource(Pool.java:50) ~[snmpapp.jar:?]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234) ~[snmpapp.jar:?]
at com.nokia.snmpapp.RedisUtils.add2list(RedisUtils.java:27) ~[snmpapp.jar:?]
at com.nokia.snmpapp.VariableDataUtils.lambda$processSnmpDataByOP$5(VariableDataUtils.java:379) ~[snmpapp.jar:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:1.8.0_191]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_191]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_191]
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) ~[?:1.8.0_191]
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_191]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[?:1.8.0_191]
My current configuration is
maxTotal: 10
maxIdle: 5
minIdle: 1
maxWaitMillis: 2000
I know I can increase my connection pool size to resolve the error message. However, I would like to understand the resource usage a bit more before determining a good number for my env. I have been google about this, but I am unable to find much talking about the monitoring tool or how to monitor the performance and pool usage. Does that mean there is no tools to monitor? If there is no way to monitor connection pool usage, what can I do to get a better idea on what to optimize on Redis connection?
Thanks in advance

Answer 1

Monitoring Jedis Pool with JMX MBeans
Jedis Pool is based on Apache Generic-Pool API, so you can use JMX to monitor the state of your Connection pool.
For example, start your application with JMX enabled:
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Then connect with JConsole or any other JMX compliant tool:
Connect to your application/process
Go to MBeans
Go to the org.apache.commons.pool2.GenericObjectPool.pool MBeans
You will be able to monitor what is happening to your application/JedisPool.
You have many monitoring tools that are compliant with JMX allowing you to put alert based on a threshold
Sizing and Configuration
The configuration of your pool is really application-specific, and to configure the maxTotal property, that is the maximum number of connection you have to look at:
how many conccurent connection is needed by your services
how long it take each call to execute the commands you are sending
And you need to be sure you do not overload your system (client-side and server-side.)
JedisPool: Be sure you release the connection ;)
Also, it is not about monitoring but more about best-practice:
does your application release the connection to the pool after each use?
/// Jedis implements Closeable. Hence, the jedis instance will be auto-closed after the last statement.
try (Jedis jedis = pool.getResource()) {
// use your jedis
jedis.set("foo", "bar");
} // the resource will be released back to the pool
If not using the try with the resource you have to close the connection (to put it back to the pool)
Jedis jedis = null;
try {
jedis = pool.getResource();
// work with Redis
jedis.set("foo", "bar");
} finally {
// you must close the connection to put it back to the pool
if (jedis != null) {
jedis.close();
}
}
Also when you close your application:
pool.close()