Apache httpd connection refused during stress test using jmeter - apache

Now, I do stress test using jemter in windows(jmeter.bat).
Number of Threads = 100
Ramp-up period = 0
Loop count = 1000
But raise connection refused error in jmeter, after 20~30 seconds.
I can not find connection refused error in /usr/local/apache/logs/error.log or /usr/local/apache/logs/access.log.
Test is HTTP GET, index.html("It's works")
Apache MPM is worker and setting below:
<IfModule mpm_worker_module>
StartServers 8
MinSpareThreads 64
MaxSpareThreads 1024
ThreadsPerChild 64
MaxRequestWorkers 512
MaxConnectionsPerChild 0
jmeter problem? any apache setting problem?
jmeter exception below:
org.apache.http.conn.HttpHostConnectException: Connection to https://checkout.teledit.com refused
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.open(MeasuringConnectionManager.java:107)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:517)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:331)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1146)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1135)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:434)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:414)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
... 14 more

It's not a jmeter issue:
check components between jmeter and apache (firewall, load balancer, ddos protection ...)
check apache settings, to check this rampup slowly and see starting from which load you get those errors.

Related

Redis broken pipe error when running in istio mesh

Connection to redis works when I run the client in Kubernetes cluster but I get 'broken pipe' error when ran inside a istio mesh
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Broken pipe
at redis.clients.jedis.Connection.flush(Connection.java:334)
at redis.clients.jedis.Connection.getIntegerReply(Connection.java:263)
at redis.clients.jedis.Jedis.exists(Jedis.java:179)
at com.swift.ems.microservices.CreateSessionServer$AuthServiceImpl.createSessionAuth(CreateSessionServer.java:59)
at com.swift.ems.microservices.AuthServiceGrpc$MethodHandlers.invoke(AuthServiceGrpc.java:204)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at redis.clients.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52)
at redis.clients.util.RedisOutputStream.flush(RedisOutputStream.java:216)
at redis.clients.jedis.Connection.flush(Connection.java:331)
... 12 more
ERROR ConnectingSep 12, 2018 9:04:19 PM io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed#525a867b
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Broken pipe
at redis.clients.jedis.Connection.flush(Connection.java:334)
at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:257)
at redis.clients.jedis.Connection.getBulkReply(Connection.java:248)
at redis.clients.jedis.Jedis.get(Jedis.java:153)
at com.swift.ems.redis.SessionHandler.getStatus(SessionHandler.java:61)
at com.swift.ems.microservices.CreateSessionServer$AuthServiceImpl.createSessionAuth(CreateSessionServer.java:74)
at com.swift.ems.microservices.AuthServiceGrpc$MethodHandlers.invoke(AuthServiceGrpc.java:204)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at redis.clients.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52)
at redis.clients.util.RedisOutputStream.flush(RedisOutputStream.java:216)
at redis.clients.jedis.Connection.flush(Connection.java:331)
... 14 more
I believe it needs to give more time for connecting but not sure, thanks in advance.
Well, Istio was blocking the connection to redis so created a service entry and it worked!
https://istio.io/blog/2018/egress-tcp/#mesh-external-service-entry-for-an-external-mysql-instance

Unable to create a successful connection for data source in Weblogic server 12.2.1.3 console

I am getting the below error when I create a new data source in WebLogic Server 12.2.1.3.0 after I installed the server today.
I have already created and tested the same in Java 1.7 using WebLogic Server 12.1.3.0.0.
This is failing now that I have updated to Java 1.8 using WebLogic 12.2.1.3.0.
Error:
Messages
Message icon - Error Connection test failed.
Message icon - Error [FMWGEN][DB2 JDBC Driver]Error establishing socket to host and port: powud03a.apmoller.net:60004. Reason: Connection timed out: connect
weblogic.jdbc.db2base.ddcp.b(Unknown Source)
weblogic.jdbc.db2base.ddcp.a(Unknown Source)
weblogic.jdbc.db2base.ddco.b(Unknown Source)
weblogic.jdbc.db2base.ddco.a(Unknown Source)
weblogic.jdbc.db2.drda.ddf.(Unknown Source)
weblogic.jdbc.db2.DB2ImplConnection.k(Unknown Source)
weblogic.jdbc.db2.DB2ImplConnection.d(Unknown Source)
weblogic.jdbc.db2base.BaseConnection.b(Unknown Source)
weblogic.jdbc.db2base.BaseConnection.k(Unknown Source)
weblogic.jdbc.db2base.BaseConnection.b(Unknown Source)
weblogic.jdbc.db2base.BaseConnection.a(Unknown Source)
weblogic.jdbcx.db2base.ddx.a(Unknown Source)
weblogic.jdbcx.db2base.ddx.getXAConnection(Unknown Source)
weblogic.jdbc.common.internal.DataSourceUtil.testConnection0(DataSourceUtil.java:387)
weblogic.jdbc.common.internal.DataSourceUtil.access$000(DataSourceUtil.java:24)
weblogic.jdbc.common.internal.DataSourceUtil$1.run(DataSourceUtil.java:285)
java.security.AccessController.doPrivileged(Native Method)
weblogic.jdbc.common.internal.DataSourceUtil.testConnection(DataSourceUtil.java:282)
com.bea.console.utils.jdbc.JDBCUtils.testConnection(JDBCUtils.java:937)
com.bea.console.actions.jdbc.datasources.createjdbcdatasource.CreateJDBCDataSource.testConnectionConfiguration(CreateJDBCDataSource.java:524)
...

java.net.SocketTimeoutException: Read timed out just after few milliseconds

I have an Axis2 1.5.5 instance running on Tomcat5.5.
Tomcat connector configuration is like below.
<Connector port="8080" maxHttpHeaderSize="8192"
maxThreads="100" minSpareThreads="15" maxSpareThreads="40"
enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" />
Client send post request to my service endpoint deployed in Axis2, but receive read timed out just after few milliseconds.
I check this through TCP dumps.
Access logs looks like below.
http-5905-Processor28 "POST /axis2/services/my_service_endpoint HTTP/1.1" 500 4234 20003
Access logs shows that it waited for 20 seconds as per connectionTiemout attribute.
This is the stack trace received by client.
org.apache.axis2.AxisFault: Read timed out
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:75)
at org.apache.axis2.transport.TransportUtils.createDocumentElement(TransportUtils.java:197)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:145)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:168)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:144)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:783)
at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:813)
at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:226)
at org.apache.coyote.http11.filters.ChunkedInputFilter.parseChunkHeader(ChunkedInputFilter.java:249)
at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:130)
at org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer.java:748)
at org.apache.coyote.Request.doRead(Request.java:419)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:270)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:293)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at org.apache.axiom.om.util.DetachableInputStream.read(DetachableInputStream.java:147)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.apache.axis2.builder.BuilderUtil.getCharSetEncoding(BuilderUtil.java:294)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:56)
... 24 more
First I thought it could be due to some network issue, but later our network admin confirmed he didn't find any issue within network.
Not sure why its getting read timed out just after few milliseconds. Retrying the same request after sometime will go through successfully. This is an intermittent behavior but frequency is very high.
I am looking for a hint to further investigate this issue.
Thanks for your help.

Distributed Testing with Jmeter

I am using Jmeter 2.8 and trying to do distributed (master-slave) testing for the first time.
I setup everything fine
set Ip addresses in JMeter.properties
run Jmater-server.bat on all machines
but when I run test(a very simple request) it gets failed:
Request:
http://localhost:8080/web/login?credentials=<credential><userId>WillieWonka
</userId><passwd><![CDATA[WillieWonka]]></passwd></credential>
Error I am getting is
org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:8080 refused
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:284)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1075)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1064)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:426)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:255)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
While server is up and running (I tested this request running from localMachine only, It is working fine).
What is wrong?
What I am missing?
I have resolved this issue, replaced
localhost:8080
with
computerName:8080
and start application server via start.bat instead run.bat
now it is working fine.

Using Apache Directory Studio

I have followed http://confluence.atlassian.com/display/CROWD/Creating+a+Connection+to+your+LDAP+Directory
for basic understanding and trying to set up LDAP for my office use.
scenario:
I have started Apache 2.2 (Apache Directory Server) at localhost:389 and it's running fine.
Next, I installed Apache DirectoryStudio and trying to create new connection. But I am getting the following error while opening the connection:
Error while opening connection - localhost:389; socket closed
javax.naming.ServiceUnavailableException: localhost:389; socket closed
at com.sun.jndi.ldap.Connection.readReply(Unknown Source)
at com.sun.jndi.ldap.LdapClient.ldapBind(Unknown Source)
at com.sun.jndi.ldap.LdapClient.authenticate(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.ensureOpen(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.ensureOpen(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.reconnect(Unknown Source)
at javax.naming.ldap.InitialLdapContext.reconnect(Unknown Source)
at org.apache.directory.studio.connection.core.io.jndi.JNDIConnectionWrapper$7.run(JNDIConnectionWrapper.java:1055)
at org.apache.directory.studio.connection.core.io.jndi.JNDIConnectionWrapper.runAndMonitor(JNDIConnectionWrapper.java:1272)
at org.apache.directory.studio.connection.core.io.jndi.JNDIConnectionWrapper.doBind(JNDIConnectionWrapper.java:1065)
at org.apache.directory.studio.connection.core.io.jndi.JNDIConnectionWrapper.bind(JNDIConnectionWrapper.java:254)
at org.apache.directory.studio.connection.core.jobs.OpenConnectionsRunnable.run(OpenConnectionsRunnable.java:114)
at org.apache.directory.studio.connection.core.jobs.StudioConnectionJob.run(StudioConnectionJob.java:114)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Please guide me!
I resolved this by changing theport numbers.
Now I am getting the following error:
Error while opening connection - [LDAP: error code 49 - cannot bind the principalDn.]
javax.naming.AuthenticationException: [LDAP: error code 49 - cannot bind the principalDn.]
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.ensureOpen(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.ensureOpen(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.reconnect(Unknown Source)
at javax.naming.ldap.InitialLdapContext.reconnect(Unknown Source)
at org.apache.directory.studio.connection.core.io.jndi.JNDIConnectionWrapper$7.run(JNDIConnectionWrapper.java:1055)
at org.apache.directory.studio.connection.core.io.jndi.JNDIConnectionWrapper.runAndMonitor(JNDIConnectionWrapper.java:1272)
at org.apache.directory.studio.connection.core.io.jndi.JNDIConnectionWrapper.doBind(JNDIConnectionWrapper.java:1065)
at org.apache.directory.studio.connection.core.io.jndi.JNDIConnectionWrapper.bind(JNDIConnectionWrapper.java:254)
at org.apache.directory.studio.connection.core.jobs.OpenConnectionsRunnable.run(OpenConnectionsRunnable.java:114)
at org.apache.directory.studio.connection.core.jobs.StudioConnectionJob.run(StudioConnectionJob.java:114)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
This means that your connection to the ldap server is wrong. I know the default username and password for Apache DS is admin and secret respectively.
Try the following as the bind DN or user string:
uid=admin,ou=system