I have created a simple JMS to JMS proxy that listen to JMS from Glassfish sourceQueue and Send JMS to Active MQ targetQueue. Whenever I switchon the endpoint in WSO2 OR bring down the Active MQ and send a message to the sourceQueue, the system start writing unlimited log files that grows so quickly.
Here my proxy code and also underneath, I added the generated log file;
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="JMS2JMSProxy2"
transports="jms"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<property name="OUT_ONLY" value="true"/>
<property name="ClientApiNonBlocking" scope="axis2" action="remove"/>
<send>
<endpoint key="ActiveMQ"/>
</send>
<log level="custom">
<property name="Transaction Action" value="Committed"/>
</log>
</inSequence>
<outSequence/>
<faultSequence>
<log level="custom">
<property name="Transaction Action" value="Rollbacked"/>
</log>
<property name="SET_ROLLBACK_ONLY" value="true" scope="axis2"/>
</faultSequence>
</target>
<parameter name="serviceType">proxy</parameter>
<parameter name="transport.jms.Destination">GFQueue</parameter>
<description/>
</proxy>
Here is the log file extract;
2014-05-07 16:22:52,451 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,451 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,457 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,458 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,458 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,459 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,467 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,467 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,468 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,468 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,476 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,476 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,477 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,477 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,485 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,485 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,486 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,486 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,494 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,494 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,495 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,495 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,500 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,500 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,503 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,503 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,508 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,508 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,509 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,509 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,514 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,515 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,515 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,515 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,523 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,523 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,524 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,524 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,534 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,534 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,535 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,535 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,540 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,541 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,541 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,542 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,547 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,547 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,550 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,550 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,556 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,556 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,557 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,557 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,567 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,568 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,568 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,569 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,575 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,575 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,575 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,575 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,583 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,583 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,584 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,584 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,592 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,592 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,593 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,593 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,601 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,601 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,602 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,602 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,608 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,608 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,610 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,610 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,615 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl: org.apache.synapse.mediators.MediatorFaultHandler
2014-05-07 16:22:52,616 [-] [jms-Worker-9] WARN JMS2JMSProxy Executing fault sequence mediator : org.apache.synapse.mediators.base.SequenceMediator
2014-05-07 16:22:52,618 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Rollbacked
2014-05-07 16:22:52,618 [-] [jms-Worker-9] INFO JMS2JMSProxy Transaction Action = Committed
2014-05-07 16:22:52,624 [-] [jms-Worker-9] INFO JMS2JMSProxy FaultHandler executing impl:
Extract from Error log file;
2014-05-08 15:34:46,983 [-] [jms-Worker-1] ERROR JMSConnectionFactory Error acquiring a Connection from the JMS CF : default using properties : {java.naming.provider.url=tcp://localhost:61616, java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory, transport.jms.ConnectionFactoryType=queue, transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory}
javax.jms.JMSException: Could not connect to broker URL: tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused: connect
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:283)
at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:227)
at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:175)
at org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
at org.apache.axis2.transport.jms.JMSConnectionFactory.createConnection(JMSConnectionFactory.java:319)
at org.apache.axis2.transport.jms.JMSConnectionFactory.getSharedConnection(JMSConnectionFactory.java:425)
at org.apache.axis2.transport.jms.JMSConnectionFactory.getConnection(JMSConnectionFactory.java:382)
at org.apache.axis2.transport.jms.JMSMessageSender.<init>(JMSMessageSender.java:86)
at org.apache.axis2.transport.jms.JMSSender.sendMessage(JMSSender.java:123)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:297)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:482)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:59)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:338)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:333)
at org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:59)
at org.apache.synapse.endpoints.IndirectEndpoint.send(IndirectEndpoint.java:54)
at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:97)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:77)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:166)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.base.AbstractTransportListener.handleIncomingMessage(AbstractTransportListener.java:328)
at org.apache.axis2.transport.jms.JMSMessageReceiver.processThoughEngine(JMSMessageReceiver.java:200)
at org.apache.axis2.transport.jms.JMSMessageReceiver.onMessage(JMSMessageReceiver.java:122)
at org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.handleMessage(ServiceTaskManager.java:535)
at org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:438)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:435)
at org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:401)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:53)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:263)
... 33 more
2014-05-08 15:34:47,138 [-] [jms-Worker-1] ERROR Axis2Sender Unexpected error during sending message out
org.apache.axis2.transport.jms.AxisJMSException: Error acquiring a Connection from the JMS CF : default using properties : {java.naming.provider.url=tcp://localhost:61616, java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory, transport.jms.ConnectionFactoryType=queue, transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory}
at org.apache.axis2.transport.jms.JMSConnectionFactory.handleException(JMSConnectionFactory.java:248)
at org.apache.axis2.transport.jms.JMSConnectionFactory.createConnection(JMSConnectionFactory.java:330)
at org.apache.axis2.transport.jms.JMSConnectionFactory.getSharedConnection(JMSConnectionFactory.java:425)
at org.apache.axis2.transport.jms.JMSConnectionFactory.getConnection(JMSConnectionFactory.java:382)
at org.apache.axis2.transport.jms.JMSMessageSender.<init>(JMSMessageSender.java:86)
at org.apache.axis2.transport.jms.JMSSender.sendMessage(JMSSender.java:123)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:297)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:482)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:59)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:338)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:333)
at org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:59)
at org.apache.synapse.endpoints.IndirectEndpoint.send(IndirectEndpoint.java:54)
at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:97)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:77)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:166)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.base.AbstractTransportListener.handleIncomingMessage(AbstractTransportListener.java:328)
at org.apache.axis2.transport.jms.JMSMessageReceiver.processThoughEngine(JMSMessageReceiver.java:200)
at org.apache.axis2.transport.jms.JMSMessageReceiver.onMessage(JMSMessageReceiver.java:122)
at org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.handleMessage(ServiceTaskManager.java:535)
at org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:438)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: javax.jms.JMSException: Could not connect to broker URL: tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused: connect
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:283)
at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:227)
at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:175)
at org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
at org.apache.axis2.transport.jms.JMSConnectionFactory.createConnection(JMSConnectionFactory.java:319)
... 29 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:435)
at org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:401)
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:53)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:263)
... 33 more
2014-05-08 15:34:47,158 [-] [jms-Worker-1] WARN ProxyServiceMessageReceiver Executing fault handler due to exception encountered
2014-05-08 15:34:47,228 [-] [jms-Worker-1] WARN FaultHandler ERROR_CODE : 0
2014-05-08 15:34:47,228 [-] [jms-Worker-1] WARN FaultHandler ERROR_MESSAGE : Unexpected error during sending message out
2014-05-08 15:34:47,228 [-] [jms-Worker-1] WARN FaultHandler ERROR_DETAIL : org.apache.synapse.SynapseException: Unexpected error during sending message out
at org.apache.synapse.core.axis2.Axis2Sender.handleException(Axis2Sender.java:172)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:71)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:338)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:333)
at org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:59)
at org.apache.synapse.endpoints.IndirectEndpoint.send(IndirectEndpoint.java:54)
at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:97)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:77)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:166)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.base.AbstractTransportListener.handleIncomingMessage(AbstractTransportListener.java:328)
at org.apache.axis2.transport.jms.JMSMessageReceiver.processThoughEngine(JMSMessageReceiver.java:200)
at org.apache.axis2.transport.jms.JMSMessageReceiver.onMessage(JMSMessageReceiver.java:122)
at org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.handleMessage(ServiceTaskManager.java:535)
at org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:438)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.axis2.transport.jms.AxisJMSException: Error acquiring a Connection from the JMS CF : default using properties : {java.naming.provider.url=tcp://localhost:61616, java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory, transport.jms.ConnectionFactoryType=queue, transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory}
at org.apache.axis2.transport.jms.JMSConnectionFactory.handleException(JMSConnectionFactory.java:248)
at org.apache.axis2.transport.jms.JMSConnectionFactory.createConnection(JMSConnectionFactory.java:330)
at org.apache.axis2.transport.jms.JMSConnectionFactory.getSharedConnection(JMSConnectionFactory.java:425)
at org.apache.axis2.transport.jms.JMSConnectionFactory.getConnection(JMSConnectionFactory.java:382)
at org.apache.axis2.transport.jms.JMSMessageSender.<init>(JMSMessageSender.java:86)
at org.apache.axis2.transport.jms.JMSSender.sendMessage(JMSSender.java:123)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:297)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:482)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:59)
... 19 more
Caused by: javax.jms.JMSException: Could not connect to broker URL: tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused: connect
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:283)
at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:227)
at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:175)
at org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
at org.apache.axis2.transport.jms.JMSConnectionFactory.createConnection(JMSConnectionFactory.java:319)
... 29 more
Related
path picture
I'd like to know a reason of the following error on Mac:
Docker ggr logs:
2022/11/01 00:02:12 [-] [-] [INIT] [-] [-] [-] [-] [-] [-] [Users file is "/etc/grid-router/users.htpasswd"]
2022/11/01 00:02:12 [-] [-] [INIT] [-] [-] [-] [-] [-] [-] [Users file "/etc/grid-router/users.htpasswd" does not exist or is a directory]
Sequence of commands on Mac:
1. cd /etc/grid-router/quota
2. htpasswd -bc users.htpasswd test test-password
3. cat users.htpasswd
test:$apr1$KmVucZGe$S4MXZhc0DHC4J/6i5KW9h/
4. $ pwd
/etc/grid-router/quota
5. test.xml file was created appropriately in quota directory
6. execute this command from quota directory:
docker run -d --name ggr -p 0.0.0.0:4445:4444 -v /etc/grid-router/:/etc/grid-router:ro --net host aerokube/ggr:latest-release
Docker ggr logs:
2022/11/01 00:02:12 [-] [-] [INIT] [-] [-] [-] [-] [-] [-] [Users file is "/etc/grid-router/users.htpasswd"]
2022/11/01 00:02:12 [-] [-] [INIT] [-] [-] [-] [-] [-] [-] [Users file "/etc/grid-router/users.htpasswd" does not exist or is a directory]
Path to the file users.htpasswd on Mac:
please see a pic in the attachment
Thanks in advance
File with users list should be in /etc/grid-router/users.htpasswd. On your screenshot it is in /etc/grid-router/quota/users.htpasswd. That's obviously an error.
I am trying to create a bare-bones skeleton integration test for Kafka with TestContainers: just publish message to topic and check it arrives to it (entire setup below).
SkeletonTests.kt
#Testcontainers
class SkeletonTests {
#Container
private val kafka = KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:6.2.1"))
#Test
fun `do nothing special`() {
// Arrange
val producer = KafkaProducer(
mapOf(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG to kafka.bootstrapServers),
StringSerializer(),
StringSerializer()
)
val consumer = KafkaConsumer(
mapOf(
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG to kafka.bootstrapServers,
ConsumerConfig.MAX_POLL_RECORDS_CONFIG to 1,
ConsumerConfig.AUTO_OFFSET_RESET_CONFIG to "earliest",
ConsumerConfig.GROUP_ID_CONFIG to "test-group-id"
),
StringDeserializer(),
StringDeserializer()
).apply { subscribe(listOf("topic")) }
// Act
producer.send(ProducerRecord("topic", "Hello there!"))
producer.flush()
// Assert
assertEquals(consumer.poll(Duration.ofSeconds(3)).first().value(), "Hello there!")
}
}
build.gradle.kts
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
kotlin("jvm") version "1.5.31"
}
repositories {
mavenCentral()
}
dependencies {
implementation("org.apache.kafka:kafka-clients:3.1.0")
implementation("ch.qos.logback:logback-core:1.2.11")
implementation("ch.qos.logback:logback-classic:1.2.11")
implementation("org.slf4j:slf4j-api:1.7.36")
testImplementation("org.junit.jupiter:junit-jupiter:5.8.2")
testImplementation("org.testcontainers:kafka:1.17.1")
testImplementation("org.testcontainers:junit-jupiter:1.17.1")
}
tasks.test {
useJUnitPlatform()
}
tasks.withType<KotlinCompile>() {
kotlinOptions.jvmTarget = "11"
}
logback.xml
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<root level="debug">
<appender-ref ref="STDOUT"/>
</root>
</configuration>
Test passes (ProducerTests > do nothing special() PASSED) however log is flooded with producer and consumer warnings. Is this expected? Am I missing some configuration for broker/leader to make this errors go away?
Producer:
02:12:11.204 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available
02:12:11.255 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node localhost:61785 (id: 1 rack: null) for sending metadata request
02:12:11.255 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node localhost:61785 (id: 1 rack: null) using address localhost/127.0.0.1
02:12:11.255 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.common.network.Selector - [Producer clientId=producer-1] Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:224)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:526)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
at java.base/java.lang.Thread.run(Thread.java:829)
02:12:11.256 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Node 1 disconnected.
02:12:11.256 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:61785) could not be established. Broker may not be available.
02:14:39.670 [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Error while fetching metadata with correlation id 3 : {topic=LEADER_NOT_AVAILABLE}
02:14:39.670 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Requesting metadata update for topic topic due to error LEADER_NOT_AVAILABLE
Consumer:
02:21:11.351 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Initiating connection to node localhost:61785 (id: 1 rack: null) using address localhost/127.0.0.1
02:21:11.352 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:224)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:526)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:306)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1374)
02:21:11.352 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Node 1 disconnected.
02:21:11.352 [kafka-coordinator-heartbeat-thread | test-group-id] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Connection to node 1 (localhost/127.0.0.1:61785) could not be established. Broker may not be available.
02:21:11.353 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] No broker available to send FindCoordinator request
02:21:11.543 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] No broker available to send FindCoordinator request
02:21:11.543 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Give up sending metadata request since no node is available
02:21:11.544 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Sending FindCoordinator request to broker localhost:61785 (id: 1 rack: null)
02:21:11.544 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Initiating connection to node localhost:61785 (id: 1 rack: null) using address localhost/127.0.0.1
02:21:11.545 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Connection with localhost/127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:224)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:526)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:306)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1374)
02:21:11.545 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Node 1 disconnected.
02:21:11.545 [kafka-coordinator-heartbeat-thread | test-group-id] WARN org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Connection to node 1 (localhost/127.0.0.1:61785) could not be established. Broker may not be available.
02:21:11.545 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] Cancelled request with header RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=4, clientId=consumer-test-group-id-1, correlationId=14) due to node 1 being disconnected
02:21:11.545 [kafka-coordinator-heartbeat-thread | test-group-id] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=consumer-test-group-id-1, groupId=test-group-id] FindCoordinator request failed due to org.apache.kafka.common.errors.DisconnectException
Update: I removed Spring dependencies completely however the problem persists and that suggests I am misconfiguring TestContainers.
When I am trying to run oozie Job it's throwing following error
Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
My shell script runs fine when run independently.
I am new to oozie, Any help will be greatly appreciated
Workflow.xml
<workflow-app name="DQ-wf" xmlns="uri:oozie:workflow:0.4">
<start to = 'validate-tables' />
<action name="validate-tables">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>test.sh</exec>
<file>test.sh</file>
<capture-output/>
</shell>
<ok to="decide"/>
<error to="fail"/>
</action>
<decision name="decide">
<switch>
<case to="shell-file">
${wf:actionData("shell-file")["isMatch"]}
</case>
<default to="end"/>
</switch>
</decision>
<action name="shell-file">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>copy-output.sh</exec>
<file>copy-output.sh</file>
</shell>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name='end' />
</workflow-app>
Job.Properties
===============
nameNode=hdfs://dev-devwork:8020
jobTracker=isperf.xxx.com:8050
hdfs=hdfs://dev-devwork
queueName=default
workflowRoot=/etc/hive/conf
oozie.use.system.libpath=true
oozieProjectRoot=${nameNode}/user/gdsds/oozie/ing
appPath=${oozieProjectRoot}
oozie.wf.application.path=${appPath}
test.sh
========
DATE=`date +%Y_%m_%d_%H_%M_%S`;
. ./env_settings.ini
destdir=/home/gdsds/oozie/$DATE.txt
Q1=$( hive -S -e "select count(1) from gpc.TABLE_A;")
echo $Q1;
Q1T=$( hive -S -e "select count(1) from gpc.TABLE_B;")
if [ "$Q1" -eq "$Q1T" ]
then
echo "$Q1 TABLE_A VS TABLE_B MATCH" >> "$destdir";
else
echo "$Q1 TABLE_A VS TABLE_B UNMATCH" >> "$destdir";
echo "isMatch=false";
exit 0
fi
Q2=$( hive -S -e "select count(1) from gpc.TABLE_X;")
echo $Q2
Q2T=$( hive -S -e "select count(1) from gpc.TABLE_Y")
if [ "$Q2" -eq "$Q2T" ]
then
echo "$Q2 TABLE_X VS TABLE_Y MATCH" >> "$destdir";
echo "isMatch=true";
else
echo "$Q2 TABLE_X VS TABLE_Y UNMATCH" >> "$destdir";
echo "isMatch=false";
exit 0
fi
Error Logs
2016-05-04 23:04:47,771 INFO ActionStartXCommand:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#:start:] Start action [0186090-160418004347476-oozie-oozi-W#:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-04 23:04:47,774 INFO ActionStartXCommand:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#:start:] [***0186090-160418004347476-oozie-oozi-W#:start:***]Action status=DONE
2016-05-04 23:04:47,774 INFO ActionStartXCommand:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#:start:] [***0186090-160418004347476-oozie-oozi-W#:start:***]Action updated in DB!
2016-05-04 23:04:47,799 INFO WorkflowNotificationXCommand:520 - SERVER[isperf.xxx.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#:start:] No Notification URL is defined. Therefore nothing to notify for job 0186090-160418004347476-oozie-oozi-W#:start:
2016-05-04 23:04:47,801 INFO WorkflowNotificationXCommand:520 - SERVER[isperf.xxx.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0186090-160418004347476-oozie-oozi-W
2016-05-04 23:04:47,817 INFO ActionStartXCommand:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#validate-tables] Start action [0186090-160418004347476-oozie-oozi-W#validate-tables] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-04 23:04:49,257 INFO ShellActionExecutor:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#validate-tables] checking action, hadoop job ID [job_1461261058027_467475] status [RUNNING]
2016-05-04 23:04:49,258 INFO ActionStartXCommand:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#validate-tables] [***0186090-160418004347476-oozie-oozi-W#validate-tables***]Action status=RUNNING
2016-05-04 23:04:49,258 INFO ActionStartXCommand:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#validate-tables] [***0186090-160418004347476-oozie-oozi-W#validate-tables***]Action updated in DB!
2016-05-04 23:04:49,262 INFO WorkflowNotificationXCommand:520 - SERVER[isperf.xxx.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#validate-tables] No Notification URL is defined. Therefore nothing to notify for job 0186090-160418004347476-oozie-oozi-W#validate-tables
2016-05-04 23:05:16,380 INFO CallbackServlet:520 - SERVER[isperf.xxx.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#validate-tables] callback for action [0186090-160418004347476-oozie-oozi-W#validate-tables]
2016-05-04 23:05:16,550 INFO ShellActionExecutor:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#validate-tables] action completed, external ID [job_1461261058027_467475]
2016-05-04 23:05:16,554 WARN ShellActionExecutor:523 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#validate-tables] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2016-05-04 23:05:16,600 INFO ActionEndXCommand:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#validate-tables] ERROR is considered as FAILED for SLA
2016-05-04 23:05:16,627 INFO ActionStartXCommand:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#fail] Start action [0186090-160418004347476-oozie-oozi-W#fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-04 23:05:16,630 INFO ActionStartXCommand:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#fail] [***0186090-160418004347476-oozie-oozi-W#fail***]Action status=DONE
2016-05-04 23:05:16,631 INFO ActionStartXCommand:520 - SERVER[isperf.xxx.com] USER[gdsds] GROUP[-] TOKEN[] APP[DQ-wf] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#fail] [***0186090-160418004347476-oozie-oozi-W#fail***]Action updated in DB!
2016-05-04 23:05:16,678 INFO WorkflowNotificationXCommand:520 - SERVER[isperf.xxx.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#fail] No Notification URL is defined. Therefore nothing to notify for job 0186090-160418004347476-oozie-oozi-W#fail
2016-05-04 23:05:16,678 INFO WorkflowNotificationXCommand:520 - SERVER[isperf.xxx.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0186090-160418004347476-oozie-oozi-W
2016-05-04 23:05:16,678 INFO WorkflowNotificationXCommand:520 - SERVER[isperf.xxx.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0186090-160418004347476-oozie-oozi-W] ACTION[0186090-160418004347476-oozie-oozi-W#validate-tables] No Notification URL is defined. Therefore nothing to notify for job 0186090-160418004347476-oozie-oozi-W#validate-tables
I getting following error in my server.log after deployed my application in JBoss and while trying to login this error occurs....
Basically JBoss says two-phase commit fails while inifinispan tries to marshal the object.... it raises the exceptionNotSerializableException...
Consequently, JBoss-Infinispan says DummyTransaction.java:287 ISPN000111: afterCompletion() failed for SynchronizationAdapter
13:28:38,186 ERROR [org.infinispan.transaction.TransactionCoordinator] (http-/192.nnn.nnn.nn:8080-3) ISPN000188: Error while processing a commit in a two-phase transaction: org.infinispan.marshall.NotSerializableException: com.singtel.um.transaction.TransactionManager
Caused by: an exception which occurred:
in field transactionManager
in object java.util.HashMap#f1852ac6
in object org.jboss.as.clustering.SimpleMarshalledValue#f1852ac6
-> toString = {User=com.singtel.um.dto.UserDto#4c08e73b, Userid=1, Username=sysadmin, org.apache.struts.action.LOCALE=en_US, Response=com.singtel.um.dto.ResponsePageDto#72c5cec1, userManager=com.singtel.um.UserManager#418634d5}
in object org.infinispan.util.FastCopyHashMap#373deff8
-> toString = {0=11, 1=1444195716345, 2=org.jboss.as.clustering.web.DistributableSessionMetadata#40843124, 3={User=com.singtel.um.dto.UserDto#4c08e73b, Userid=1, Username=sysadmin, org.apache.struts.action.LOCALE=en_US, Response=com.singtel.um.dto.ResponsePageDto#72c5cec1, userManager=com.singtel.um.UserManager#418634d5}}
in object org.infinispan.atomic.AtomicHashMap#3345fa74
-> toString = AtomicHashMap
in object org.infinispan.container.entries.ImmortalCacheEntry#cca5812b
-> toString = ImmortalCacheEntry{key=ZVx0ehVWVUvmITSBGMT9pQmB, value=ImmortalCacheValue {value=AtomicHashMap}}
in object org.infinispan.loaders.bucket.Bucket#18891bc5
-> toString = Bucket{entries={ZVx0ehVWVUvmITSBGMT9pQmB=ImmortalCacheEntry{key=ZVx0ehVWVUvmITSBGMT9pQmB, value=ImmortalCacheValue {value=AtomicHashMap}}}, bucketId='1607025664'}
13:28:38,210 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) ERROR [http-/192.nnn.nnn.nn:8080-3] (TransactionCoordinator.java:213) - ISPN000188: Error while processing a commit in a two-phase transaction
13:28:38,213 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) org.infinispan.marshall.NotSerializableException: com.singtel.um.transaction.TransactionManager
13:28:38,216 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) Caused by: an exception which occurred:
13:28:38,218 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) in field transactionManager
13:28:38,219 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) in object java.util.HashMap#f1852ac6
13:28:38,221 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) in object org.jboss.as.clustering.SimpleMarshalledValue#f1852ac6
13:28:38,224 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) -> toString = {User=com.singtel.um.dto.UserDto#4c08e73b, Userid=1, Username=sysadmin, org.apache.struts.action.LOCALE=en_US, Response=com.singtel.um.dto.ResponsePageDto#72c5cec1, userManager=com.singtel.um.UserManager#418634d5}
13:28:38,228 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) in object org.infinispan.util.FastCopyHashMap#373deff8
13:28:38,231 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) -> toString = {0=11, 1=1444195716345, 2=org.jboss.as.clustering.web.DistributableSessionMetadata#40843124, 3={User=com.singtel.um.dto.UserDto#4c08e73b, Userid=1, Username=sysadmin, org.apache.struts.action.LOCALE=en_US, Response=com.singtel.um.dto.ResponsePageDto#72c5cec1, userManager=com.singtel.um.UserManager#418634d5}}
13:28:38,237 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) in object org.infinispan.atomic.AtomicHashMap#3345fa74
13:28:38,239 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) -> toString = AtomicHashMap
13:28:38,241 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) in object org.infinispan.container.entries.ImmortalCacheEntry#cca5812b
13:28:38,243 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) -> toString = ImmortalCacheEntry{key=ZVx0ehVWVUvmITSBGMT9pQmB, value=ImmortalCacheValue {value=AtomicHashMap}}
13:28:38,246 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) in object org.infinispan.loaders.bucket.Bucket#18891bc5
13:28:38,249 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) -> toString = Bucket{entries={ZVx0ehVWVUvmITSBGMT9pQmB=ImmortalCacheEntry{key=ZVx0ehVWVUvmITSBGMT9pQmB, value=ImmortalCacheValue {value=AtomicHashMap}}}, bucketId='1607025664'}
13:28:38,253 ERROR [org.infinispan.transaction.tm.DummyTransaction] (http-/192.nnn.nnn.nn:8080-3) ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=-1, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction#42} org.infinispan.transaction.synchronization.SynchronizationAdapter#61: org.infinispan.CacheException: Could not commit.
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83)
at org.infinispan.transaction.tm.DummyTransaction.notifyAfterCompletion(DummyTransaction.java:285)
at org.infinispan.transaction.tm.DummyTransaction.runCommitTx(DummyTransaction.java:334)
at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:91)
at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
at org.jboss.as.clustering.web.impl.TransactionBatchingManager.endBatch(TransactionBatchingManager.java:75)
at org.jboss.as.web.session.DistributableSessionManager.processSessionRepl(DistributableSessionManager.java:1574) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at org.jboss.as.web.session.DistributableSessionManager.storeSession(DistributableSessionManager.java:872) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at org.jboss.as.web.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:47) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:142) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:99) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:92) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:64) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.4.8.Final-redhat-4.jar:7.4.8.Final-redhat-4]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_80]
Caused by: javax.transaction.xa.XAException
at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:224)
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:185)
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81)
... 21 more
13:28:38,308 INFO [stdout] (http-/192.nnn.nnn.nn192.nnn.nnn.nn:8080-3) ERROR [http-/192.nnn.nnn.nn:8080-3] (DummyTransaction.java:287) - ISPN000111: afterCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=-1, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction#42} org.infinispan.transaction.synchronization.SynchronizationAdapter#61
13:28:38,317 INFO [stdout] (http-/192.nnn.nnn.nn:8080-3) org.infinispan.CacheException: Could not commit.
Please someone help me to solve this problem.
One of your classes (not sure what's the name but it holds com.singtel.um.dto.ResponsePageDto, com.singtel.um.dto.UserDto, and com.singtel.um.UserManager) seems to have a field transactionManager, and your TM implementation is not serializable. You should make that field transient.
Not sure why the HashMap is in the stack, though, unless something is instrumenting Java classes.
Ideally, you should also use Externalizable or AdvancedExternalizer to optimize serialization of your objects. See Infinispan User Guide
I have a synchronized Mule flow which reads messages from a sonic topic and publish to a Rabbit exchange.
I am loosing messages when the Rabbit is brought up/down.
Rabbit exchange is publishing to HA queues.
How can I make sure Mule is not consuming the message until proper "Ack" is received from Rabbit Broker?
Here is the flow.
<jms:connector name="sonicMQConnectorSub" validateConnections="true" connectionFactory-ref="factorySub" doc:name="JMS" clientId="testClient" durable="true" maxRedelivery="-1" >
<reconnect-forever frequency="30000"/>
</jms:connector>
<spring:beans>
<spring:bean id="soniqMQConnectionFactoryBeanSub" name="factorySub" class="progress.message.jclient.ConnectionFactory">
<spring:property name="connectionURLs" value="tcp://server1:7800" />
<spring:property name="defaultUser" value="user" />
<spring:property name="defaultPassword" value="pass" />
</spring:bean>
</spring:beans>
<amqp:connector name="AMQP" validateConnections="true" host="server2" fallbackAddresses="server3" doc:name="AMQP Connector" port="5672" mandatory="true" activeDeclarationsOnly="true">
<reconnect-forever frequency="30000"/>
</amqp:connector>
<flow name="rabbitFlow1" doc:name="rabbitFlow1" processingStrategy="synchronous">
<jms:inbound-endpoint doc:name="JMS" connector-ref="sonicMQConnectorSub" topic="testtopic"/>
<logger message="Message: #[message.payload]" level="INFO" doc:name="Logger"/>
<amqp:outbound-endpoint exchangeName="rabbitExchange" exchangeDurable="true" responseTimeout="10000" connector-ref="AMQP" doc:name="AMQP" exchangeType="fanout"/>
</flow>
Updated 04/22
Here is the exception trace when Mule is connecting to the 2nd broker. This is when I loose a message.
2014-04-22 09:49:29,453 - org.mule.exception.DefaultSystemExceptionStrategy - ERROR -
********************************************************************************
Message : Connection shutdown detected for: AMQP
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. Software caused connection abort: recv failed (java.net.SocketException)
java.net.SocketInputStream:-2 (null)
2. connection error; reason: java.net.SocketException: Software caused connection abort: recv failed (com.rabbitmq.client.ShutdownSignalException)
com.rabbitmq.client.impl.AMQConnection:715 (null)
3. Connection shutdown detected for: AMQP (org.mule.transport.ConnectException)
org.mule.transport.amqp.AmqpConnector$1:502 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/transport/ConnectException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.net.SocketException: Software caused connection abort: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
2014-04-22 09:49:29,453 - org.mule.exception.DefaultSystemExceptionStrategy - INFO - Exception caught is a ConnectException, attempting to reconnect...
2014-04-22 09:49:29,454 - org.mule.lifecycle.AbstractLifecycleManager - INFO - Stopping connector: AMQP
2014-04-22 09:49:29,454 - org.mule.lifecycle.AbstractLifecycleManager - INFO - Stopping: 'AMQP.dispatcher.1064499250'. Object is: AmqpMessageDispatcher
2014-04-22 09:49:29,454 - org.mule.lifecycle.AbstractLifecycleManager - INFO - Disposing: 'AMQP.dispatcher.1064499250'. Object is: AmqpMessageDispatcher
2014-04-22 09:49:29,455 - org.mule.transport.amqp.AmqpConnector - ERROR - clean connection shutdown; reason: Attempt to use closed connection
2014-04-22 09:49:29,461 - org.mule.transport.amqp.AmqpConnector - INFO - Connected: AmqpConnector
{
name=AMQP
lifecycle=stop
this=33c5919e
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=true
supportedProtocols=[amqp]
serviceOverrides=<none>
}
2014-04-22 09:49:29,461 - org.mule.transport.amqp.AmqpConnector - INFO - Starting: AmqpConnector
{
name=AMQP
lifecycle=stop
this=33c5919e
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=true
supportedProtocols=[amqp]
serviceOverrides=<none>
}
2014-04-22 09:49:29,461 - org.mule.lifecycle.AbstractLifecycleManager - INFO - Starting connector: AMQP
Updated 04/23 with the Exception received when JMS Transaction is added to AMQP outbound Endpoint:
Message : No active AMQP transaction found for endpoint: DefaultOutboundEndpoint{endpointUri=amqp://rabbitExchange, connector=AmqpConnector
{
name=AMQP
lifecycle=start
this=25ec1ff7
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=true
supportedProtocols=[amqp]
serviceOverrides=<none>
}
, name='endpoint.amqp.rabbitExchange', mep=ONE_WAY, properties={exchangeDurable=true, exchangeType=fanout}, transactionConfig=Transaction {factory=org.mule.transport.jms.JmsTransactionFactory#6491b172, action=ALWAYS_JOIN, timeout=30000}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=10000, endpointEncoding=UTF-8, disableTransportTransformer=false}
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Root Exception stack trace:
org.mule.transaction.IllegalTransactionStateException: No active AMQP transaction found for endpoint: DefaultOutboundEndpoint{endpointUri=amqp://rabbitExchange, connector=AmqpConnector
{
name=AMQP
lifecycle=start
this=25ec1ff7
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=true
supportedProtocols=[amqp]
serviceOverrides=<none>
}
, name='endpoint.amqp.rabbitExchange', mep=ONE_WAY, properties= {exchangeDurable=true, exchangeType=fanout}, transactionConfig=Transaction {factory=org.mule.transport.jms.JmsTransactionFactory#6491b172, action=ALWAYS_JOIN, timeout=30000}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=10000, endpointEncoding=UTF-8, disableTransportTransformer=false}
at org.mule.transport.amqp.AmqpMessageDispatcher.getEventChannel(AmqpMessageDispatcher.java:298)
at org.mule.transport.amqp.AmqpMessageDispatcher.doOutboundAction(AmqpMessageDispatcher.java:152)
at org.mule.transport.amqp.AmqpMessageDispatcher.doDispatch(AmqpMessageDispatcher.java:127)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
2014-04-23 10:52:03,178 - org.mule.transport.jms.JmsTransaction - WARN - Transaction rollback attempted, but no resource bound to org.mule.transport.jms.JmsTransaction#d4ac3d8f-caf6-11e3-bf9a-8b266a026dee [status=STATUS_MARKED_ROLLBACK, key=null, resource=null]
I see two options:
Make the JMS client a durable one and consume testtopic transactionally so if amqp:outbound-endpoint fails, the message will be redelivered.
Wrap the amqp:outbound-endpoint with until-successful to retry the outbound dispatches until the AMQP connector reconnects to RabbitMQ.