Tomcat Connection Pool Exhausted - apache

I'm using Apache Tomcat JDBC connection pooling in my project. I'm confused because under heavy load I keep seeing the following error:
12:26:36,410 ERROR [] (http-/XX.XXX.XXX.X:XXXXX-X) org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-/XX.XXX.XXX.X:XXXXX-X] Timeout: Pool empty. Unable to fetch a connection in 10 seconds, none available[size:4; busy:4; idle:0; lastwait:10000].
12:26:36,411 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/APP].[AppConf]] (http-/XX.XXX.XXX.X:XXXXX-X) JBWEB000236: Servlet.service() for servlet AppConf threw exception: org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException
My expectation was that with pooling, requests for new connections would be held in a queue until a connection became available. Instead it seems that requests are rejected when the pool has reached capacity. Can this behaviour be changed?
Thanks,
Dal
This is my pool configuration:
PoolProperties p = new PoolProperties();
p.setUrl("jdbc:oracle:thin:#" + server + ":" + port + ":" + SID_SVC);
p.setDriverClassName("oracle.jdbc.driver.OracleDriver");
p.setUsername(username);
p.setPassword(password);
p.setMaxActive(4);
p.setInitialSize(1);
p.setMaxWait(10000);
p.setRemoveAbandonedTimeout(300);
p.setMinEvictableIdleTimeMillis(150000);
p.setTestOnBorrow(true);
p.setValidationQuery("SELECT 1 from dual");
p.setMinIdle(1);
p.setMaxIdle(2);
p.setRemoveAbandoned(true);
p.setJdbcInterceptors(
"org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;"
+ "org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer;"
+ "org.apache.tomcat.jdbc.pool.interceptor.ResetAbandonedTimer");

This working as per design/implementation, if you see the log Timeout: Pool empty. Unable to fetch a connection in 10 seconds and your configuration is p.setMaxWait(10000);. The requesting thread waits for 10seconds(10000 millseconds, maxwait) before giving up waiting for connection.
Now you have two solutions, increase the number of maxActive connection or check if there are any connection leaks/long running queries(which you do not expect).

Related

org.apache.ignite.client.ClientConnectionException: Ignite cluster is unavailable inside kubernetes environment

We have a setup wherein, one ignite server node serves 15 to 20 thick client nodes and 40 to 50 thin client nodes, thin client connection is singlton,
In operation, some times we get below error,
org.apache.ignite.client.ClientConnectionException: Ignite cluster is unavailable [sock=Socket[addr=hostnm19.hostx.com/10.13.10.19,port=30519,localport=57552]]
On the Server node, we are inserting data inside a third party store using CacheStoreAdapters
Don't know where it goes wrong since out of 100 operations one operation fails with the above error.
Also, let me know what can we do for this failure handling.
Apache Ignite version: 2.8
Edits: (Code Snippet)
ClientConfiguration cfg = new ClientConfiguration()
.setAddresses("host:port");
IgniteClient client = Ignition.startClient(cfg); // this client is singleton
client.getOrCreateCache("ABC_CACHE").put(key, val);
StatckTrace:
org.apache.ignite.client.ClientConnectionException: Ignite cluster is unavailable [sock=Socket[addr=hostnm19.hostx.com/10.13.10.19,port=30519,localport=57552]]
at org.apache.ignite.internal.client.thin.TcpClientChannel.handleIOError(TcpClientChannel.java:499)
at org.apache.ignite.internal.client.thin.TcpClientChannel.handleIOError(TcpClientChannel.java:491)
at org.apache.ignite.internal.client.thin.TcpClientChannel.access$100(TcpClientChannel.java:92)
at org.apache.ignite.internal.client.thin.TcpClientChannel$ByteCountingDataInput.read(TcpClientChannel.java:538)
at org.apache.ignite.internal.client.thin.TcpClientChannel$ByteCountingDataInput.readInt(TcpClientChannel.java:572)
at org.apache.ignite.internal.client.thin.TcpClientChannel.processNextResponse(TcpClientChannel.java:272)
at org.apache.ignite.internal.client.thin.TcpClientChannel.receive(TcpClientChannel.java:234)
at org.apache.ignite.internal.client.thin.TcpClientChannel.service(TcpClientChannel.java:171)
at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:160)
at org.apache.ignite.internal.client.thin.ReliableChannel.request(ReliableChannel.java:187)
at org.apache.ignite.internal.client.thin.TcpIgniteClient.getOrCreateCache(TcpIgniteClient.java:114)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.ignite.internal.client.thin.TcpClientChannel$ByteCountingDataInput.read(TcpClientChannel.java:535)
... 36 more
You probably have network or NAT configured which will reset connections when not used, or even sporadically.
In this case, you will have to reconnect.
Another option, are you sure you are connecting to thin client port and not some other port?

Ignite Nodes disconnects after some time

Hi peoples I am facing an issue with my clusters .
Whenever the ignite nodes goes on idle they gets disconnected and after that time if i try to connect my first request gets failed. but after first request all requests are sucessfull until they become idle for some time.
Please help
SEVERE: Servlet.service() for servlet [dispatcherServlet] in context
with path [] threw exception [Request processing failed; nested exception is
org.springframework.transaction.CannotCreateTransactionException: Could not
open JDBC Connection for transaction; nested exception is
java.sql.SQLException: Failed to communicate with Ignite cluster.] with root
cause
commons-api_research.1.tmu590s6e7pi#cc-dev-platform03.netlink.com |
java.net.SocketException: Connection reset
commons-api_research.1.tmu590s6e7pi#cc-dev-platform03.netlink.com | at
java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
commons-api_research.1.tmu590s6e7pi#cc-dev-platform03.netlink.com | at
java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
commons-api_research.1.tmu590s6e7pi#cc-dev-platform03.netlink.com | at
java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
commons-api_research.1.tmu590s6e7pi#cc-dev-platform03.netlink.com | at
java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
commons-api_research.1.tmu590s6e7pi#cc-dev-platform03.netlink.com | at
java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
commons-api_research.1.tmu590s6e7pi#cc-dev-platform03.netlink.com | at
org.apache.ignite.internal.jdbc.thin.JdbcThinTcpIo.read
You should not keep raw JDBC connections around indefinitely, instead you should use a proper connection pool, such as c3p0, with support for connection recycling, etc.
Well its not about DataSource but setting idle connection timeout to max worked for me.
TcpCommunicationSpi spi = new TcpCommunicationSpi();
spi.setUsePairedConnections(true);
spi.setIdleConnectionTimeout(999999999999999999l);
cfg.setCommunicationSpi(spi);

Client timeout in Gemfire logs

​
I am currently in the process of trouble shooting Gemfire logs for client timeout:
​
Logs:​
​[warning 2018/06/11 23:45:05.685 CDT xxxxsrv01_instance_42404_cacheserver <ClientHealthMonitor Thread> tid=0x97] Server connection from [identity(xx.xx.xx.xx(5338:loner):33228:4cae4ef2,connection=2; port=33261] is terminated because its client timeout of 10,000 has expired.
Even increasing the ​"read-timeout" value in client pool config to 200000 gives "client time out of 200000 has expired".
Also noticed below logs:
xxxxx_myinstance_42404_cacheserver <Handshaker /0:0:0:0:0:0:0:0:42404 Thread 229> tid=0x2c29cc] Bridge server: failed accepting client connection {0}
​Can you please help me on possible causes of the issue.
Regards,​
Rahul N.​

Monit cannot open connection errors resulting in alerts from M/Monit that server is down

I'm using monit and M/Monit to monitor my application infrastructure. But every once in a while, M/Monit will show a "No report" error from a server and mark it down. A few seconds later, the issue clears at the next check in for the server to M/Monit.
The monit logs on some of the servers have these events in them:
Oct 14 12:19:11 ip-10-203-51-199 monit[30307]: M/Monit: cannot open a
connection to http://example.com:8080/collector -- Connection timed out
Oct 14 12:20:16 ip-10-203-51-199 monit[30307]: M/Monit: cannot open a
connection to http://example.com:8080/collector -- Connection timed out
Oct 14 12:22:21 ip-10-203-51-199 monit[30307]: M/Monit: cannot open a
connection to http://example.com:8080/collector -- Connection timed out
What config do I need to tune to increase the threshold until M/Monit considers the server actually down?
Here is the config from the server that has the most trouble:
set httpd port 2812 and
allow xxx:xxx
set mailserver xxx.xxx.xxx port xxx username "xxx" password "xxx" using tlsv1 with timeout 15 seconds
set daemon 30
with start delay 120
set logfile syslog facility log_daemon
set alert xxx
set mail-format {
subject: $EVENT $SERVICE on $HOST
from: monit#$HOST
message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
}
set mmonit http://xxx:xxx#example.com:8080/collector
There doesn't appear to be any problem with config file.
The intermittent problem you are experiencing is because monit is failing to open a socket on the port and timing out. See the source code for reference (handle_mmonit()):
http://fossies.org/linux/privat/monit-5.6.tar.gz:a/monit-5.6/src/collector.c
Search for the string "M/Monit: cannot open a connection to".
The timeout value appears to be fixed at 5 seconds in the code. But 5 seconds is ample time to open a socket connection on that port.
How often does monit post events to mmonit?
Had the same problem
[MST Apr 5 11:24:11] error : 'apache' failed protocol test [APACHESTATUS] at [phoenix.example.com]:80 [TCP/IP] -- APACHE-STATUS: error -- no scoreboard found
[MST Apr 5 11:24:16] error : Cannot create socket to [10x.xx.xx.x4]:8080 -- Connection timed out
We had another firewall on top of iptables. Opened up the 8080 in the input and the output side and it fixed it!

Apache + Tomcat with mod_jk: maxThread setting upon load balancing

I have Apache + Tomcat setup with mod_jk on 2 servers. Each server has its own Apache+Tomcat pair, and every request is being served by Tomcat load balancing workers on 2 servers.
I have a question about how Apache's maxClient and Tomcat's maxThread should be set.
The default numbers are,
Apache: maxClient=150, Tomcat: maxThread=200
In this configuration, if we have only 1 server setup, it would work just fine as Tomcat worker never receives the incoming connections more than 150 at once. However, if we are load balancing between 2 servers, could it be possible that Tomcat worker receives 150 + (some number from another server) and make the maxThread overflow as SEVERE: All threads (200) are currently busy?
If so, should I set Tomcat's maxThread=300 in this case?
Thanks
Setting maxThreads to 300 should be fine - there are no fixed rules. It depends on whether you see any connections being refused.
Increasing too much causes high memory consumption but production Tomcats are known to run with 750 threads. See here as well. http://java-monitor.com/forum/showthread.php?t=235
Have you actually got the SEVERE error? I've tested on our Tomcat 6.0.20 and it throws an INFO message when the maxThreads is crossed.
INFO: Maximum number of threads (200) created for connector with address null and port 8080
It does not refuse connections until the acceptCount value is crossed. The default is 100.
From the Tomcat docs http://tomcat.apache.org/tomcat-5.5-doc/config/http.html
The maximum queue length for incoming
connection requests when all possible
request processing threads are in use.
Any requests received when the queue
is full will be refused. The default
value is 100.
The way it works is
1) As the number of simultaneous requests increase, threads will be created up to the configured maximum (the value of the maxThreads attribute).
So in your case, the message "Maximum number of threads (200) created" will appear at this point. However requests will still be queued for service.
2) If still more simultaneous requests are received, they are queued up to the configured maximum (the value of the acceptCount attribute).
Thus a total of 300 requests can be accepted without failure. (assuming your acceptCount is at default of 100)
3) Crossing this number throws Connection Refused errors, until resources are available to process them.
So you should be fine until you hit step 3