java.net.SocketTimeoutException: Read timed out just after few milliseconds - axis2

I have an Axis2 1.5.5 instance running on Tomcat5.5.
Tomcat connector configuration is like below.
<Connector port="8080" maxHttpHeaderSize="8192"
maxThreads="100" minSpareThreads="15" maxSpareThreads="40"
enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" />
Client send post request to my service endpoint deployed in Axis2, but receive read timed out just after few milliseconds.
I check this through TCP dumps.
Access logs looks like below.
http-5905-Processor28 "POST /axis2/services/my_service_endpoint HTTP/1.1" 500 4234 20003
Access logs shows that it waited for 20 seconds as per connectionTiemout attribute.
This is the stack trace received by client.
org.apache.axis2.AxisFault: Read timed out
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:75)
at org.apache.axis2.transport.TransportUtils.createDocumentElement(TransportUtils.java:197)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:145)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:168)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:144)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:783)
at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:813)
at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:226)
at org.apache.coyote.http11.filters.ChunkedInputFilter.parseChunkHeader(ChunkedInputFilter.java:249)
at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:130)
at org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer.java:748)
at org.apache.coyote.Request.doRead(Request.java:419)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:270)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:293)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at org.apache.axiom.om.util.DetachableInputStream.read(DetachableInputStream.java:147)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.apache.axis2.builder.BuilderUtil.getCharSetEncoding(BuilderUtil.java:294)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:56)
... 24 more
First I thought it could be due to some network issue, but later our network admin confirmed he didn't find any issue within network.
Not sure why its getting read timed out just after few milliseconds. Retrying the same request after sometime will go through successfully. This is an intermittent behavior but frequency is very high.
I am looking for a hint to further investigate this issue.
Thanks for your help.

Related

SocketTimeout Exception on Jedis del operation

I'm regularly getting SocketTimeout Exception on del operation on redis server using Jedis. It's only 1% of all the delete operations done.
Exception:-
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_77]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_77]
at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_77]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_77]
at java.net.SocketInputStream.read(SocketInputStream.java:127) ~[?:1.8.0_77]
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:195) ~[jedis-2.x.jar:?]
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) ~[jedis-2.x.jar:?]
at redis.clients.jedis.Protocol.process(Protocol.java:141) ~[jedis-2.x.jar:?]
at redis.clients.jedis.Protocol.read(Protocol.java:205) ~[jedis-2.x.jar:?]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297) ~[jedis-2.x.jar:?]
at redis.clients.jedis.Connection.getIntegerReply(Connection.java:222) ~[jedis-2.x.jar:?]
at redis.clients.jedis.Jedis.del(Jedis.java:145) ~[jedis-2.x.jar:?]
at redis.clients.jedis.ShardedJedis.del(ShardedJedis.java:252) ~[jedis-2.x.jar:?]
I checked my shared pool configuration and it is enough to handle the traffic. I did a client list on the server and do not see any client with age over 180 sec. Any broken resource is dropped from the pool.
What could be the issue here and how can I investigate this further?

Tomcat 8: SSL session ID not available

I'm having a problem trying to migrate from Tomcat7 to Tomcat8. Things seems to work fine, but when I access my application it logs this Exception:
09-Nov-2016 13:23:09.192 WARNING [https-openssl-nio-9443-exec-3] org.apache.coyote.AbstractProcessor.populateSslRequestAttributes Exception getting SSL attributes
java.lang.IllegalStateException: SSL session ID not available
at org.apache.tomcat.util.net.openssl.OpenSSLEngine$OpenSSLSession.getId(OpenSSLEngine.java:1048)
at org.apache.tomcat.util.net.jsse.JSSESupport.getSessionId(JSSESupport.java:156)
at org.apache.coyote.AbstractProcessor.populateSslRequestAttributes(AbstractProcessor.java:619)
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:359)
at org.apache.coyote.Request.action(Request.java:392)
at org.apache.catalina.connector.Request.getAttribute(Request.java:900)
at org.apache.catalina.connector.RequestFacade.getAttribute(RequestFacade.java:282)
at org.apache.cxf.transport.http.AbstractHTTPDestination.propogateSecureSession(AbstractHTTPDestination.java:411)
at org.apache.cxf.transport.http.AbstractHTTPDestination.setupMessage(AbstractHTTPDestination.java:395)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:238)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:589)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
My connector is defined this way:
<Connector port="9443" URIEncoding="utf-8" protocol="HTTP/1.1"
scheme="https" secure="true" SSLEnabled="true"
maxPostSize="-1"
useAprConnector ="false"
sslProtocol="TLSv1.2" clientAuth="want"
disableSessionTickets="true"
keystoreFile="${catalina.home}/conf/keystore.jks"
keystorePass="..."
truststoreFile="${catalina.home}/conf/truststore.jks"
truststorePass="..."
/>
Things works fine on Tomcat7, and also in tomcat8 the secure conection is fine.
It looks like you are hitting Bug 59811 - TLS Session ID not available if session tickets are used.
I suspect that this is because in switching to Tomcat 8.5.x OpenSSL is used for TLS if it is available.
You have a couple of options:
Add an explicit SSLHostConfig element and include disableSessionTickets="true" to disable session tickets
Set sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation" on the Connector to use JSSE for TLS rather than OpenSSL
The second option is closest to what you had with Tomcat 7.

Apache httpd connection refused during stress test using jmeter

Now, I do stress test using jemter in windows(jmeter.bat).
Number of Threads = 100
Ramp-up period = 0
Loop count = 1000
But raise connection refused error in jmeter, after 20~30 seconds.
I can not find connection refused error in /usr/local/apache/logs/error.log or /usr/local/apache/logs/access.log.
Test is HTTP GET, index.html("It's works")
Apache MPM is worker and setting below:
<IfModule mpm_worker_module>
StartServers 8
MinSpareThreads 64
MaxSpareThreads 1024
ThreadsPerChild 64
MaxRequestWorkers 512
MaxConnectionsPerChild 0
jmeter problem? any apache setting problem?
jmeter exception below:
org.apache.http.conn.HttpHostConnectException: Connection to https://checkout.teledit.com refused
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.open(MeasuringConnectionManager.java:107)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:517)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:331)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1146)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1135)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:434)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:414)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
... 14 more
It's not a jmeter issue:
check components between jmeter and apache (firewall, load balancer, ddos protection ...)
check apache settings, to check this rampup slowly and see starting from which load you get those errors.

Distributed Testing with Jmeter

I am using Jmeter 2.8 and trying to do distributed (master-slave) testing for the first time.
I setup everything fine
set Ip addresses in JMeter.properties
run Jmater-server.bat on all machines
but when I run test(a very simple request) it gets failed:
Request:
http://localhost:8080/web/login?credentials=<credential><userId>WillieWonka
</userId><passwd><![CDATA[WillieWonka]]></passwd></credential>
Error I am getting is
org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8080 refused
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:284)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1075)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1064)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:426)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:255)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
While server is up and running (I tested this request running from localMachine only, It is working fine).
What is wrong?
What I am missing?
I have resolved this issue, replaced
localhost:8080
with
computerName:8080
and start application server via start.bat instead run.bat
now it is working fine.

Error Starting Tomcat 6 with SSL Enabled

I recently got a ssl certificate from godaddy. I created the keystore properly and put the following into my server.xml:
<Connector port="8443" protocol="org.apache.coyote.Http11Protocol" SSLEnabled="true"
maxThreads="200" scheme="https" secure="true" keystoreFile="C:\SSLCerts\mykeystore.keystore" keystorePass="password"
clientAuth="false" sslProtocol="TLS" />
I originally had protocol="HTTP/1.1" but I was receiving an error stating that the keystoreFile and the keystorePass didn't have a matching value. Which caused a binding error on port 8443. I ran netstat and made sure nothing is running on 8443; did some googling and found to fix that issue to change the protocol as it is above.
After adding that change I'm receiving the following error:
SEVERE: Failed to start connector [Connector[org.apache.coyote.Http11Protocol-84
43]]
LifecycleException: service.getName(): "Catalina"; Protocol handler start fail
ed: java.lang.NullPointerException
at org.apache.catalina.connector.Connector.start(Connector.java:1102)
at org.apache.catalina.core.StandardService.start(StandardService.java:5
40)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754
)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Jun 21, 2012 1:23:57 PM org.apache.coyote.ajp.AjpAprProtocol start
Does anyone have any idea what needs to be done in order to fix this?
protocol="org.apache.coyote.Http11Protocol" is not a valid protocol. You want protocol="org.apache.coyote.http11.Http11Protocol". Note the extra http11 package you are missing.
That will probably bring you back to the problem with your key store.