I'm using AMQ message groups with AMQ 5.6 with Kahadb persistent messaging and periodically see this WARN in the logs. I don't see any specific errors that result from this, but am wondering what the side effects of this are, etc.
My consumer is a Camel (2.9.2) route with maxConcurrentConsumers=5 (using AMQ connection pool, etc). I'll try to boil this down to a unit test at some point...for now, here is the stack trace
Async error occurred: java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException
java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:785)
at org.apache.activemq.broker.region.Queue.send(Queue.java:707)
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:306)
at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:453)
at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:231)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$KeyIterator.next(HashMap.java:828)
at org.apache.activemq.util.MarshallingSupport.marshalPrimitiveMap(MarshallingSupport.java:64)
at org.apache.activemq.command.Message.beforeMarshall(Message.java:210)
at org.apache.activemq.command.ActiveMQObjectMessage.beforeMarshall(ActiveMQObjectMessage.java:199)
at org.apache.activemq.openwire.v6.MessageMarshaller.looseMarshal(MessageMarshaller.java:277)
at org.apache.activemq.openwire.v6.ActiveMQMessageMarshaller.looseMarshal(ActiveMQMessageMarshaller.java:111)
at org.apache.activemq.openwire.v6.ActiveMQObjectMessageMarshaller.looseMarshal(ActiveMQObjectMessageMarshaller.java:111)
at org.apache.activemq.openwire.OpenWireFormat.marshal(OpenWireFormat.java:168)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:429)
at org.apache.activemq.store.kahadb.KahaDBStore$StoreQueueTask.run(KahaDBStore.java:1177)
logged a JIRA for this (AMQ-4092), it sounds like an issue with a combination of KahaDB and message groups. setting concurrentStoreAndDispatchQueues=false solves the issue (thanks to Gary Tully for the suggestion)
Related
I am getting ClassCastException from AMQP internals while using Spring RabbitMQ. We found this in application logs on production and were not able to reproduce this, but it's a real pain for us.
Basically, it seems like the entire stack fails once this happens, because then we see invalid DeliveryTag being sent on that channel, which results in RabbitMQ closing this channel.
We are using AMQPClient 5.4.3 and SpringRabbit 2.1.2, RabbitMQ caching mode is Connection (because of HAProxy and failover), ACK mode is manual, Publisher returns are enabled and Publisher Confirms as well.
java.lang.ClassCastException: com.rabbitmq.client.impl.AMQImpl$Channel$OpenOk cannot be cast to com.rabbitmq.client.impl.AMQImpl$Confirm$SelectOk
at com.rabbitmq.client.impl.ChannelN.confirmSelect(ChannelN.java:1552)
at com.rabbitmq.client.impl.ChannelN.confirmSelect(ChannelN.java:52)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.doCreateBareChannel(CachingConnectionFactory.java:658)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:649)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:608)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:499)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$1600(CachingConnectionFactory.java:100)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:1331)
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1967)
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1935)
at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:889)
at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:955)
at org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:948)
at net.homecredit.lisa.messaging.amqp.SaveApplicationResponseProducer.sendResponse(SaveApplicationResponseProducer.kt:40)
at net.homecredit.lisa.messaging.amqp.SaveApplicationConsumer.onReceiveRequestMessage(SaveApplicationConsumer.kt:87)
at sun.reflect.GeneratedMethodAccessor372.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:170)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:120)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:190)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:127)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1521)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1444)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1431)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1410)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:848)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:832)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:78)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1073)
at java.lang.Thread.run(Thread.java:748)
This is complete mystery for me.
Does anyone know what's wrong?
The RabbitMQ client can be configured to check the response type for RPC via channelShouldCheckRpcResponseType, see also https://github.com/rabbitmq/rabbitmq-java-client/issues/708.
My consumer worked fine for about one week, then the consumer thread died and logged the below error:
It seems the Authentication failure is caused by
BufferedInputStream.read timeout, and I want to know if there is a way
to treat the Authentication failure as a non-fatal and do not kill the
consumer thread. and I think the timeout issue is just caused by
network issue not the Authentication failure, since this consumer
already worked for one week.
org.springframework.amqp.rabbit.listener.exception.FatalListenerStartupException: Authentication failure
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:460)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1171)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused by authentication failure
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:61)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:296)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:524)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:85)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:135)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:71)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:456)
... 2 more
Caused by: com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused by authentication failure
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:341)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:824)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:736)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:283)
... 7 more
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36) at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:366) at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:233) at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:211) at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:326)
The target code looks like:
} catch (ShutdownSignalException e) {
...
throw new PossibleAuthenticationFailureException(e);
}
So, indeed, it isn't necessary that the issue with the connection is about authentication.
There is just ShutdownSignalException by the Connection timed out reason.
So, you should just try to increase connection timeout, which is 60000 by default.
But that is likely there is some problem with resources on the broker since we just can't connect.
We can't treat this kind of errors (ShutdownSignalException) as non-fatal, since your problem happens exactly on the start().
EDIT
In case of fatal failure the container emits ListenerContainerConsumerFailedEvent:http://docs.spring.io/spring-amqp/reference/html/_reference.html#consumer-events. You can handle it and restart the container on purpose.
I'm using Apache Ignite to cluster our web sessions. Occasionally one of our caches crashes, and I keep receiving the following exception messages in my Tomcat application log file:
2016-05-23 15:04:02,200 ERROR root/error 495 - Failed to update web session: null
java.lang.IllegalStateException: Cache has been closed or destroyed: session-cache
at org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(GridCacheGateway.java:160)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.onEnter(IgniteCacheProxy.java:1923)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:855)
at org.apache.ignite.cache.websession.WebSessionFilter.doFilter0(WebSessionFilter.java:341)
at org.apache.ignite.cache.websession.WebSessionFilter.doFilter(WebSessionFilter.java:315)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
2016-05-23 15:04:06,497 ERROR root/error 495 - Failed to update web session: null
java.lang.IllegalStateException: Cache has been closed or destroyed: session-cache
...
It worries me because I cannot find any further error messages explaining the cause of the crash, not even in the folder IGNITE_HOME/work
(By the way this crash happened before due to running out of memory, and I fixed it by increasing the application JVM heap size.)
My questions are:
Where I can find the logs that explain the cause of the crash?
If no logs available, shall I need to register certain events so as to receive the notification/explanation of the crash, how?
When receiving such notification in the event of such cache crash, is there any way to restore the cache without having to restart the whole application?
Many thanks!
Update (2016-08-04)
As Valentin pointed out, by default Ignite inherits the application's logging settings if the node is embedded in the application. And indeed Ignite writes its logs in the same application log file.
If I would like to output Ignite logs to a separate log file, I could add configuration to log4j.properties as below:
log4j.appender.IGNITE=org.apache.log4j.DailyRollingFileAppender
log4j.appender.IGNITE.File=${IGNITE_HOME}/work/logs/${tomcat.hostname}/ignite.log
log4j.appender.IGNITE.DatePattern='.'yyyy-MM-dd
log4j.appender.IGNITE.Threshold=DEBUG
log4j.appender.IGNITE.layout=org.apache.log4j.PatternLayout
log4j.appender.IGNITE.layout.ConversionPattern=[%d{ABSOLUTE}][%-5p][%t][%c{1}] %m%n
log4j.logger.org.apache.ignite=INFO, IGNITE
But there is still one thing I do not quite understand still is that, the configuration aforementioned would output the Ignite logging information to both ignite.log as specified above and the application's log file. #Valentin, wonder if you know why? Thanks. Is there any way that I could output the Ignite logs to ignite.log only, not to the application's log file at all?
If a node is embedded in the application, it will by default inherit the its logging settings. So most likely Ignite writes the log in the same file where the application writes it.
Note that if you're using Log4J, you should include ignite-log4j dependency in you project.
As for restoring the data, you should not lose any data if only one node in the cluster fails and you have at least one backup. To add backups use CacheConfiguration.backups configuration property.
I don't know if this will solve your specific problem but I found logs in IGNITE_HOME/work/logs
I'm using JBoss AS 7 Hornetq. Our standalone java application interacts with a queue and sends messages. We had the entire environment setup and it was working pretty smoothly. Suddenly, one fine day, our standalone application failed with the below exception:
Caused by: java.io.NotSerializableException: org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy
Detailed exception stack trace is below
javax.naming.NamingException: Failed to lookup [Root exception is java.io.NotSerializableException: org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy]
at org.jboss.naming.remote.client.ClientUtil.namingException(ClientUtil.java:36)
at org.jboss.naming.remote.protocol.v1.Protocol$1.execute(Protocol.java:104)
at org.jboss.naming.remote.protocol.v1.RemoteNamingStoreV1.lookup(RemoteNamingStoreV1.java:79)
at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:79)
at org.jboss.naming.remote.client.RemoteContext.lookup(RemoteContext.java:83)
at javax.naming.InitialContext.lookup(Unknown Source)
at com.infosys.lbs.publishing.LocationProcessor.postMessageInQueue(LocationProcessor.java:377)
at com.infosys.lbs.publishing.LocationProcessor.process(LocationProcessor.java:69)
at com.infosys.lbs.publishing.main.Publisher.main(Publisher.java:34)
Caused by: java.io.NotSerializableException: org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:891)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:998)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
at org.jboss.naming.remote.protocol.v1.Protocol$1$2.write(Protocol.java:138)
at org.jboss.naming.remote.protocol.v1.WriteUtil.write(WriteUtil.java:61)
at org.jboss.naming.remote.protocol.v1.Protocol$1.handleServerMessage(Protocol.java:128)
at org.jboss.naming.remote.protocol.v1.RemoteNamingServerV1$MessageReciever$1.run(RemoteNamingServerV1.java:73)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: an exception which occurred:
in field loadBalancingPolicy
in field serverLocator
in object org.hornetq.jms.client.HornetQJMSConnectionFactory#ea074d
Exception was happening when the app was trying to lookup the connection factory
QueueConnectionFactory qcf = (QueueConnectionFactory)context.lookup("jms/RemoteConnectionFactory");
Below are the steps on how we resolved the issue
There was almost zero help for resolving this issue. A web search on this exception returned next to nothing. However, this particular thread on JBoss AS Dev site spinned a thought in my head: RemoteConnectionFactory is not found when looking up in a remote client
The scenario mentioned in this thread was not same as ours. (In our app this is the first and the only lookup happening.) This thread got me thinking towards a possible connection factory initialization issue. While there is nothing I could do to debug or find the issue around it, I thought that if I could reinitialize it, that would help.
So I tried lookup with java:jboss/exported/jms/RemoteConnectionFactory As expected it failed with a NamingException. Hoping that this naming syntax (using java:/) would have resulted in a reinitialization, I tried lookup again with jms/RemoteConnectionFactory. And bingo!!! it worked!
Unfortunately, we still don't know why it happened, and if it is just a one-off case! Documenting it here just in case some mortal soul hits this issue.
I am using a wsdl2java generated web service client for the axis2 web service (axis2 version is 1.6.1). So for http connection the old version of commons-httpclient 3.1 is used. While doing some load tests to my web service client. In some cases i came across this exception
org.apache.axis2.AxisFault: org.apache.axiom.ext.io.StreamCopyException: Error reading from source
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at com.ardic.arcsp.carbon.registry.ws.stub.WSStorageServiceStub.fromOM(WSStorageServiceStub.java:16706)
at com.ardic.arcsp.carbon.registry.ws.stub.WSStorageServiceStub.wSgetById(WSStorageServiceStub.java:6659)
at com.ardic.arcsp.update.wsclient.WSStorageServiceClient.getResourceById(WSStorageServiceClient.java:177)
at com.ardic.arcsp.update.wsclient.OTAUpdateTestServlet.downloadResource(OTAUpdateTestServlet.java:162)
at com.ardic.arcsp.update.wsclient.OTAUpdateTestServlet.doGet(OTAUpdateTestServlet.java:91)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:273)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:305)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:165)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:298)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:57)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.wso2.carbon.server.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:154)
at org.wso2.carbon.server.TomcatServer$1.invoke(TomcatServer.java:254)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:399)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:396)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:356)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1534)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:705)
Caused by: org.apache.axiom.om.OMException: org.apache.axiom.ext.io.StreamCopyException: Error reading from source
at org.apache.axiom.attachments.impl.PartFactory.createPart(PartFactory.java:172)
at org.apache.axiom.attachments.Attachments.getPart(Attachments.java:705)
at org.apache.axiom.attachments.Attachments.getNextPartDataHandler(Attachments.java:623)
at org.apache.axiom.attachments.Attachments.getDataHandler(Attachments.java:350)
at org.apache.axiom.om.impl.builder.OMAttachmentAccessorMimePartProvider.getDataHandler(OMAttachmentAccessorMimePartProvider.java:45)
at org.apache.axiom.util.stax.xop.XOPDecodingStreamReader$DataHandlerProviderImpl.getDataHandler(XOPDecodingStreamReader.java:81)
at org.apache.axiom.util.stax.xop.XOPDecodingStreamReader.getDataHandler(XOPDecodingStreamReader.java:472)
at org.apache.axiom.om.impl.SwitchingWrapper.getDataHandler(SwitchingWrapper.java:1360)
at org.apache.axiom.util.stax.XMLStreamReaderUtils.getDataHandlerFromElement(XMLStreamReaderUtils.java:230)
at com.ardic.arcsp.carbon.registry.ws.stub.xsd.WSResource$Factory.parse(WSResource.java:2087)
at com.ardic.arcsp.carbon.registry.ws.stub.WSgetByIdResponse$Factory.parse(WSgetByIdResponse.java:417)
at com.ardic.arcsp.carbon.registry.ws.stub.WSStorageServiceStub.fromOM(WSStorageServiceStub.java:16273)
... 37 more
Caused by: org.apache.axiom.ext.io.StreamCopyException: Error reading from source
at org.apache.axiom.attachments.utils.BAAOutputStream.readFrom(BAAOutputStream.java:114)
at org.apache.axiom.attachments.impl.BufferUtils.inputStream2OutputStream(BufferUtils.java:76)
at org.apache.axiom.attachments.impl.PartFactory.createPart(PartFactory.java:136)
... 48 more
Caused by: java.io.IOException: CRLF expected at end of chunk: 116/161
at org.apache.commons.httpclient.ChunkedInputStream.readCRLF(ChunkedInputStream.java:207)
at org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:219)
at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.axiom.om.util.DetachableInputStream.read(DetachableInputStream.java:147)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at java.io.PushbackInputStream.read(PushbackInputStream.java:169)
at org.apache.axiom.attachments.BoundaryPushbackInputStream.readFromStream(BoundaryPushbackInputStream.java:114)
at org.apache.axiom.attachments.BoundaryPushbackInputStream.read(BoundaryPushbackInputStream.java:248)
at org.apache.axiom.attachments.MIMEBodyPartInputStream.read(MIMEBodyPartInputStream.java:87)
at org.apache.axiom.attachments.utils.BAAOutputStream.readFrom(BAAOutputStream.java:112)
... 50 more
The web service client request files from the server and downloads the file. These exceptions occurs on the response while receiving files in chunks.
While searching i came across this fix mentioned here.
https://issues.apache.org/jira/browse/HTTPCORE-190
However the exceptions still occur. So what else can cause this exceptions?
UPDATE:
Ok, i got the real reason for exception.On the client side if the memory allocation for jvm is low, the response from the server cannot be processed due to SocketTimeOut. But the exception itself could not stress this. Increasing the heap on the client side can solve the problem temporary.
You definitely need enough memory to hold all the data that is chucked to the client. Java can give really weird results, especially the older JVMs, when there isn't enough memory to go around. It can be tricky sometimes to identify that as the root cause so bravo on that find.
There are a few helpful big on the Apache Axis2 User's Guide - Creating Clients page that might be helpful if you are new to wsdl2java.
Our next step is to learn what command line options are available. One thing you can try is using the -u option to generate separate classes for the data bindings instead of creating them as inner classes. (Take a look at this Stack Overflow question.)
Since this question is 10 years old if you run into this or another error please create a new question specific to that context.