We are running Infinispan 7.2.5 with 3 instances in REPL cluster.
Spark Client is connected to the cluster using HotRod.
Suddenly, the view is updated and one instance is removed from the cluster and from client view as well.
The instance was up & running but was not able to connect to other instances and was giving timed out exception.
What could have caused the instance to leave the cluster and not let it join back ? If someone can provide some insights on this.
Meanwhile, CPU was too high on the instance. Is it because of re-join attempts(if it actually takes much of CPU) or something else could be the reason ?
Client connected to Infinispan using HotRod observed below excpetion :
WARN (ClientListenerNotifier.java:266) - ISPN004039: Unable to complete reading event from server null
java.nio.channels.IllegalBlockingModeException
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126)
at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:237)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
WARN (ClientListenerNotifier.java:266) - ISPN004039: Unable to complete reading event from server null
java.nio.channels.IllegalBlockingModeException
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126)
at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:237)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
During the sametime, Infinispan server instance received below logs :
WARN [org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p3-t13) ISPN000071: Caught exception when handling command StateResponseCommand{cache=AsrlEnbTopologyCache, origin=asr-1-asrltopologyservice-24247, topologyId=9}: org.infinispan.util.concurrent.TimeoutException: Timed out applying state
at org.infinispan.statetransfer.StateConsumerImpl.applyState(StateConsumerImpl.java:542) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.StateResponseCommand.perform(StateResponseCommand.java:62) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:85) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:32) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_131]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_131]
whereas, other instance available which seems to be separated from the cluster has received below logs:
ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread--p2-t12) Failed to send entries to node asr-2-asrltopologyservice-2286 : Node asr-2-asrltopologyservice-2286 timed out: org.infinispan.util.concurrent.TimeoutException: Node asr-2-asrltopologyservice-2286 timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:248) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:239) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntry(OutboundTransferTask.java:195) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:149) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_131]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_131]
Caused by: org.jgroups.TimeoutException: timeout waiting for response from asr-2-asrltopologyservice-2286, request: org.jgroups.blocks.UnicastRequest#6ac2f8a3, req_id=1067, mode=GET_ALL, target=asr-2-asrltopologyservice-2286
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:427) [jgroups-3.6.2.Final.jar:3.6.2.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:433) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:241) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
... 12 more
ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread--p2-t14) Failed to send entries to node asr-2-asrltopologyservice-2286 : Node asr-2-asrltopologyservice-2286 timed out: org.infinispan.util.concurrent.TimeoutException: Node asr-2-asrltopologyservice-2286 timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:248) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:239) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.sendEntry(OutboundTransferTask.java:195) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:149) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_131]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_131]
Caused by: org.jgroups.TimeoutException: timeout waiting for response from asr-2-asrltopologyservice-2286, request: org.jgroups.blocks.UnicastRequest#1984efac, req_id=1069, mode=GET_ALL, target=asr-2-asrltopologyservice-2286
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:427) [jgroups-3.6.2.Final.jar:3.6.2.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:433) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:241) [infinispan-core-7.2.5.Final.jar:7.2.5.Final]
... 12 more
Related
Im using Spring session, Spring Data redis and redis clients for session management. Here im getting belw exception while deploy the application.
Version:
spring-session : 1.3.5.RELEASE
spring-data-redis : 2.5.4
jedis : 3.6.3
lettuce-core : 6.1.4.RELEASE
spring-webmvc: 5.3.9
Exception:
java.lang.NoSuchMethodError: 'void org.springframework.data.redis.core.RedisOperations.delete(java.lang.Object)'
at org.springframework.session.data.redis.RedisSessionExpirationPolicy.cleanExpiredSessions(RedisSessionExpirationPolicy.java:132)
at org.springframework.session.data.redis.RedisOperationsSessionRepository.cleanupExpiredSessions(RedisOperationsSessionRepository.java:398)
at jdk.internal.reflect.GeneratedMethodAccessor196.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
I configured hive on tez as following.
I can see my queries in HiveServer2 Web UI.
But, I cannot see DAGs in Tez UI after I submit hive queries.
What configurations am I missing?
$HADOOP/etc/hadoop/mapred-site.xml
mapreduce.framework.name=yarn-tez
$YARN/conf/yarn-site.xml
yarn.timeline-service.enabled=true
yarn.timeline-service.hostname=localhost
yarn.timeline-service.http-cross-origin.enabled=true
yarn.resourcemanager.system-metrics-publisher.enabled=true
$HIVE/conf/hive-site.xml
hive.execution.engine=tez
$TEZ/conf/tez-site.xml
tez.history.logging.service.class=
org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService
tez.tez-ui.history-url.base=http://localhost/tez-ui/
I tried simple select query, and join query.
The join query got this errors.
java.sql.SQLException: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257)
at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296)
at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:291)
at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:291)
at org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInterpreter.java:581)
at org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInterpreter.java:692)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:97)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:498)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:162)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I tried to set offset 3 and 0, all works fine with statistics with REST setup for now. However, the DAS wso2carbon.log keeps throwing the following error messages:
TID: [-1] [] [2016-09-15 16:27:30,727] ERROR {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker} -
Error while trying to connect to the endpoint. Cannot borrow client for ssl://localhost:7711 {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://localhost:7711
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:100)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointSecurityException: Error while trying to connect to ssl://localhost:7711
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:61)
at org.wso2.carbon.databridge.agent.client.AbstractClientPoolFactory.makeObject(AbstractClientPoolFactory.java:39)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:92)
... 6 more
Caused by: org.apache.thrift.transport.TTransportException: Could not connect to localhost on port 7711
at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:212)
at org.apache.thrift.transport.TSSLTransportFactory.getClientSocket(TSSLTransportFactory.java:166)
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:56)
... 9 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:637)
at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:425)
at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:208)
... 11 more
I am wondering what caused this, and how to fix it?
What I did is to remove port 7711 from data receiver config in apim admin-dashabord/Analytics. It is now only list tcp://localhost:7611 . That seems to fix it, but don't know why it did not respond at 7711 as it is actually configured in DAS conf/data-bridge/data-bridge-config.xml.
If you have installed any car files, you may need to update the port of receiver URL in scripts inside them, to match with your offset.
I am running load tests with JMeter, and responds well Mule with 12 threads but with 14 not responding, and then I have to restart. Someone help me.
And no errors, the log stands still.
The configuration of the components
<jms:activemq-connector name="ActiveMQ" brokerURL="${org.xyz.esb.jms.url}"
validateConnections="true" createMultipleTransactedReceivers="true"
numberOfConcurrentTransactedReceivers="30" doc:name="AMQ BE">
<receiver-threading-profile maxThreadsActive="30" maxThreadsIdle="5" doThreading="false" threadTTL="20000" poolExhaustedAction="DISCARD_OLDEST" />
<reconnect frequency="5000"/>
</jms:activemq-connector>
<jms:activemq-connector name="ActiveMQLog" brokerURL="${org.xyz.esb.jms.url}" validateConnections="false" disableTemporaryReplyToDestinations="true" doc:name="AMQ Log"/>
<db:mysql-config name="MySQL_Cache" doc:name="MySQL Configuration" dataSource-ref="dataSMule"/>
<db:mysql-config name="MySQL_Cache2" doc:name="MySQL Configuration" dataSource-ref="dataSMule2"/>
<vm:connector name="vm_cnn_op" validateConnections="false"
createMultipleTransactedReceivers="true"
numberOfConcurrentTransactedReceivers="30">
<vm:queue-profile maxOutstandingMessages="500" />
</vm:connector>
<configuration doc:name="conf_ssb">
<default-threading-profile maxThreadsActive="30" maxThreadsIdle="5" doThreading="false" threadTTL="20000" poolExhaustedAction="DISCARD_OLDEST" />
</configuration>
<flow name="agent-xdf-main" processingStrategy="synchronous">
...
</flow>
--- this is the execution of jstack
2015-09-18 23:03:10
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.71-b01 mixed mode):
"Attach Listener" daemon prio=5 tid=0x00007ff231c14800 nid=0xdd07 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"ActiveMQ InactivityMonitor Worker" daemon prio=5 tid=0x00007ff231afa000 nid=0xab07 waiting on condition [0x0000000155c24000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000014c287758> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"[esb-xyz-r4].HTTP_HTTPS_4.receiver.15" prio=5 tid=0x00007ff2320ce000 nid=0xdb03 in Object.wait() [0x0000000159fb9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000001469fa630> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
at java.lang.Object.wait(Object.java:503)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1118)
- locked <0x00000001469fa630> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:79)
WorkerContext.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"[esb-xyz-r4].http.request.dispatch.8085.14" prio=5 tid=0x00007ff2320cd800 nid=0xd903 waiting on condition [0x0000000159ec0000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000001468c24a8> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
at org.mule.transport.http.HttpMessageProcessTemplate.awaitTermination(HttpMessageProcessTemplate.java:492)
at org.mule.transport.http.HttpMessageReceiver.processRequest(HttpMessageReceiver.java:60)
at org.mule.transport.http.HttpRequestDispatcherWork.run(HttpRequestDispatcherWork.java:73)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Statistics Thread" daemon prio=5 tid=0x00007ff22b810800 nid=0x2b0b waiting on condition [0x0000000155a1e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000010f1d9a00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"__DEFAULT__" daemon prio=5 tid=0x00007ff22afc3800 nid=0x2c0b in Object.wait() [0x0000000154de6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000010f1cea50> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x000000010f1cea50> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Abandoned connection cleanup thread" daemon prio=5 tid=0x00007ff22b772000 nid=0x2f07 in Object.wait() [0x0000000151c47000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x000000010eb6f1a8> (a java.lang.ref.ReferenceQueue$Lock)
at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)
"[esb-xyz-r4].Mule.01" prio=5 tid=0x00007ff22b759800 nid=0x6203 waiting on condition [0x0000000156559000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000010ec222b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:519)
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:682)
at org.mule.context.notification.ServerNotificationManager.run(ServerNotificationManager.java:267)
at org.mule.work.WorkerContext.run(WorkerContext.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"[esb-xyz-r4].log4j.config.monitor" daemon prio=5 tid=0x00007ff22b40b800 nid=0x6003 waiting on condition [0x0000000156456000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.mule.module.launcher.log4j.ArtifactAwareRepositorySelector$ConfigWatchDog.run(ArtifactAwareRepositorySelector.java:411)
"[default].processing.time.monitor" daemon prio=5 tid=0x00007ff22b6d7800 nid=0x5e03 in Object.wait() [0x0000000154bb5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000010ed30460> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x000000010ed30460> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at org.mule.management.stats.DefaultProcessingTimeWatcher$ProcessingTimeChecker.run(DefaultProcessingTimeWatcher.java:76)
at java.lang.Thread.run(Thread.java:745)
"[default].Mule.01" prio=5 tid=0x00007ff22adc2000 nid=0x5c03 waiting on condition [0x00000001551c4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000010e1efb40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:519)
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:682)
at org.mule.context.notification.ServerNotificationManager.run(ServerNotificationManager.java:267)
at org.mule.work.WorkerContext.run(WorkerContext.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"[default].log4j.config.monitor" daemon prio=5 tid=0x00007ff22ad5f000 nid=0x5a03 waiting on condition [0x00000001550c1000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.mule.module.launcher.log4j.ArtifactAwareRepositorySelector$ConfigWatchDog.run(ArtifactAwareRepositorySelector.java:411)
"[default].log4j.config.monitor" daemon prio=5 tid=0x00007ff22ad03800 nid=0x5807 waiting on condition [0x0000000154fbe000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.mule.module.launcher.log4j.ArtifactAwareRepositorySelector$ConfigWatchDog.run(ArtifactAwareRepositorySelector.java:411)
"Mule.log.slf4j.ref.handler" prio=5 tid=0x00007ff22b1e4800 nid=0x5103 in Object.wait() [0x0000000154944000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000010e1ea228> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x000000010e1ea228> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at org.mule.module.logging.LoggerReferenceHandler$1.run(LoggerReferenceHandler.java:49)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Mule.log.clogging.ref.handler" prio=5 tid=0x00007ff22aced800 nid=0x4f03 in Object.wait() [0x0000000154841000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000010e1e8898> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x000000010e1e8898> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at org.mule.module.logging.LoggerReferenceHandler$1.run(LoggerReferenceHandler.java:49)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Mule.system.log4j.config.monitor" daemon prio=5 tid=0x00007ff22acc2800 nid=0x4d03 waiting on condition [0x000000015473e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.mule.module.launcher.log4j.ArtifactAwareRepositorySelector$ConfigWatchDog.run(ArtifactAwareRepositorySelector.java:411)
"DestroyJavaVM" prio=5 tid=0x00007ff22b0e0000 nid=0x1303 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Wrapper-Connection" daemon prio=5 tid=0x00007ff22b0d7800 nid=0x4903 runnable [0x0000000153cba000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.io.DataInputStream.readByte(DataInputStream.java:265)
at org.tanukisoftware.wrapper.WrapperManager.handleSocket(WrapperManager.java:3737)
at org.tanukisoftware.wrapper.WrapperManager.run(WrapperManager.java:4084)
at java.lang.Thread.run(Thread.java:745)
"Wrapper-Control-Event-Monitor" daemon prio=5 tid=0x00007ff22b072800 nid=0x4503 waiting on condition [0x0000000153ab4000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.tanukisoftware.wrapper.WrapperManager$3.run(WrapperManager.java:731)
"Service Thread" daemon prio=5 tid=0x00007ff22a8a8000 nid=0x4103 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=5 tid=0x00007ff22a87d000 nid=0x3f03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=5 tid=0x00007ff22b001000 nid=0x3d03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=5 tid=0x00007ff22a8a2800 nid=0x300f runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=5 tid=0x00007ff22b010800 nid=0x2903 in Object.wait() [0x0000000151b03000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000010e120260> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x000000010e120260> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" daemon prio=5 tid=0x00007ff22a85e800 nid=0x2703 in Object.wait() [0x0000000151a00000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000010e11fd18> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x000000010e11fd18> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=5 tid=0x00007ff22a85c000 nid=0x2503 runnable
"GC task thread#0 (ParallelGC)" prio=5 tid=0x00007ff22a818800 nid=0x2103 runnable
"GC task thread#1 (ParallelGC)" prio=5 tid=0x00007ff22b00a000 nid=0x2303 runnable
"VM Periodic Task Thread" prio=5 tid=0x00007ff22a8b1000 nid=0x4303 waiting on condition
JNI global references: 471
You're not providing any info so here is a general answer: make sure Mule's thread pools are properly configured.
Assuming you are doing http calls, you can use jetty-inbound instead of http-inbound transport which handles more load.
No matter the transport you use, you should tune the threading profile and run jmeter from a different host.
Also you can check more info on the performance-tuning-guide.
In general, these kind of issues needs to be figured out using narrow-down approach. Since there is no clarity on what each thread is doing as part of testing in your problem statement here are couple of pointers which you can try
check whether JMeter has enough resources or not (check using Resource Monitor utility - CPU, Memory)
server reaction to requests
Try hitting the application url from same machine where jmeter is running. This will show whether there is a problem from jmeter or not
If point 3 fails, try hitting the application url from server / someother machine if it works its a problem with jmeter & machine resources else server
See memory allocation to jmeter. Default allocations will be very less
There are some tips. Please try if possible
I am testing the Performance of a B2B example by sending FPML messages. I have tried with different no of nodes(2,3,4) with different no of messages. I can able to send upto 10,000 messages.But when i was trying with 50,000 messages with different no of nodes, i am getting the below error.
ERROR - JMSMessageSender Error closing JMS Connection after send
javax.jms.JMSException: org.apache.activemq.transport.RequestTimedOutIOException
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at org.apache.activemq.ActiveMQConnection.doSyncSendPacket(ActiveMQConnection.java:1431)
at org.apache.activemq.ActiveMQConnection.close(ActiveMQConnection.java:675)
at org.apache.axis2.transport.jms.JMSMessageSender.close(JMSMessageSender.java:250)
at org.apache.axis2.transport.jms.JMSSender.sendMessage(JMSSender.java:156)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine$TransportNonBlockingInvocationWorker.run(AxisEngine.java:626)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.activemq.transport.RequestTimedOutIOException
at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:54)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:92)
at org.apache.activemq.ActiveMQConnection.doSyncSendPacket(ActiveMQConnection.java:1418)
... 8 more
[2013-03-05 22:34:14,351] ERROR - JMSSender Unable to create a JMSMessageSender for : null
javax.jms.JMSException: java.io.EOFException
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1377)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1481)
at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:323)
at org.apache.activemq.ActiveMQConnection.createQueueSession(ActiveMQConnection.java:1257)
at org.apache.axis2.transport.jms.JMSOutTransportInfo.createJMSSender(JMSOutTransportInfo.java:378)
at org.apache.axis2.transport.jms.JMSSender.sendMessage(JMSSender.java:124)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine$TransportNonBlockingInvocationWorker.run(AxisEngine.java:626)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:229)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:221)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
... 1 more
[2013-03-05 22:34:14,357] INFO - AxisEngine [MessageContext: logID=4daccf302df9e230a0e9d2d3c35edeb7306de0841cfe1b3f] Unable to create a JMSMessageSender for : null
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to /home/vithya/wso2/wso2esb-4.6.0/repository/logs/heap-dump.hprof ...
Unable to create /home/vithya/wso2/wso2esb-4.6.0/repository/logs/heap-dump.hprof: File exists
[2013-03-05 22:35:02,899] ERROR - JMSSender Unable to create a JMSMessageSender for : null
javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format.
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1395)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1481)
at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:323)
at org.apache.activemq.ActiveMQConnection.createQueueSession(ActiveMQConnection.java:1257)
at org.apache.axis2.transport.jms.JMSOutTransportInfo.createJMSSender(JMSOutTransportInfo.java:378)
at org.apache.axis2.transport.jms.JMSSender.sendMessage(JMSSender.java:124)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine$TransportNonBlockingInvocationWorker.run(AxisEngine.java:626)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:98)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1366)
... 10 more
Exception in thread "ActiveMQ Transport: tcp:///10.200.208.124:61616#33434" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200)
at java.lang.Thread.run(Thread.java:662)
[2013-03-05 22:35:02,901] INFO - AxisEngine [MessageContext: logID=4daccf302df9e230aab5c2d3c35edeb7f06de0841cfe1b3f] Unable to create a JMSMessageSender for : null
[2013-03-05 22:35:02,903] ERROR - JMSSender Unable to create a JMSMessageSender for : null
javax.jms.JMSException: java.io.EOFException
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1377)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1481)
at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:323)
at org.apache.activemq.ActiveMQConnection.createQueueSession(ActiveMQConnection.java:1257)
at org.apache.axis2.transport.jms.JMSOutTransportInfo.createJMSSender(JMSOutTransportInfo.java:378)
at org.apache.axis2.transport.jms.JMSSender.sendMessage(JMSSender.java:124)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine$TransportNonBlockingInvocationWorker.run(AxisEngine.java:626)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:229)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:221)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204)
... 1 more
[2013-03-05 22:35:02,906] ERROR - JMSSender Unable to create a JMSMessageSender for : null
javax.jms.JMSException: Cannot send, channel has already failed: tcp://10.200.208.124:61616
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1395)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1481)
at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:323)
at org.apache.activemq.ActiveMQConnection.createQueueSession(ActiveMQConnection.java:1257)
at org.apache.axis2.transport.jms.JMSOutTransportInfo.createJMSSender(JMSOutTransportInfo.java:378)
at org.apache.axis2.transport.jms.JMSSender.sendMessage(JMSSender.java:124)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine$TransportNonBlockingInvocationWorker.run(AxisEngine.java:626)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already failed: tcp://10.200.208.124:61616
at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:282)
at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:271)
at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1366)
... 10 more
[2013-03-05 22:35:02,906] INFO - AxisEngine [MessageContext: logID=4daccf302df9e2305ab5c2d3c35edeb7906de0841cfe1b3f] Unable to create a JMSMessageSender for : null
[2013-03-05 22:35:02,909] INFO - AxisEngine [MessageContext: logID=4daccf302df9e2302ab5c2d3c35edeb7106de0841cfe1b3f] Unable to create a JMSMessageSender for : null
Please help me to resolve this problem................
Thanks in advance.....
Please refer document [1] and tune your ESB. Let us know if you are still getting this error.
http://docs.wso2.org/wiki/display/ESB460/Performance+Tuning