IgniteDataStreamer can't wirte data to ignite cluster - ignite

My ignite cluster is deploy on 2 machines. Each machine has 4 ignite process. like:
$jps
30678 CommandLineStartup
30723 CommandLineStartup
52461 Jps
30698 CommandLineStartup
30681 CommandLineStartup
When there is little data in the ignite cluster, I can use IgniteDataStreamer to write data into the ignite cluster. But if there have been more than 200G data in the ignite cluster, I can't use IgniteDataStreamer to write more data into the cluster. The memory and disk space are both enough. and the CPU load average is 2.09. The client log is :
[17:04:00,407][INFO][main][IgniteKernal%grid_recmd] Daemon mode: off
[17:04:00,407][INFO][main][IgniteKernal%grid_recmd] OS: Linux
[17:04:00,407][INFO][main][IgniteKernal%grid_recmd] OS user: mqq
[17:04:00,408][INFO][main][IgniteKernal%grid_recmd] Language runtime: Java Platform API Specification ver. 1.7
[17:04:00,408][INFO][main][IgniteKernal%grid_recmd] VM information: Java(TM) SE Runtime Environment 1.7.0_72-b14 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 24.72-b04
[17:04:00,409][INFO][main][IgniteKernal%grid_recmd] VM total memory: 2.9GB
[17:04:00,409][INFO][main][IgniteKernal%grid_recmd] Remote Management [restart: off, REST: on, JMX (remote: off)]
[17:04:00,409][INFO][main][IgniteKernal%grid_recmd] IGNITE_HOME=/data1/home/igni1.5
[17:04:00,410][INFO][main][IgniteKernal%grid_recmd] VM arguments: [-Xms3g, -Xmx3g, -XX:-UseGCOverheadLimit, -XX:+AggressiveOpts, -XX:MaxPermSize=256m, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+UseTLAB, -DIGNITE_QUIET=false]
[17:04:00,418][INFO][main][IgniteKernal%grid_recmd] Configured caches ['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache', 'cache_recmd']
[17:04:00,422][INFO][main][IgniteKernal%grid_recmd] 3-rd party licenses can be found at: /data1/home/igni1.5/libs/licenses
[17:04:00,518][INFO][main][IgniteKernal%grid_recmd] Non-loopback local IPs: 10.0.0.1
[17:04:00,519][INFO][main][IgniteKernal%grid_recmd] Enabled local MACs: 9017ACC1492D
[17:04:00,530][INFO][main][IgnitePluginProcessor] Configured plugins:
[17:04:00,530][INFO][main][IgnitePluginProcessor] ^-- None
[17:04:00,530][INFO][main][IgnitePluginProcessor]
[17:04:00,574][WARNING][main][TcpCommunicationSpi] Failure detection timeout will be ignored (one of SPI parameters has been set explicitly)
[17:04:00,627][INFO][main][TcpCommunicationSpi] IPC shared memory server endpoint started [port=48104, tokDir=/data1/home/igni1.5/work/ipc/shmem/e052dc07-8992-4834-87fd-abf84c0f644a-14472]
[17:04:00,628][INFO][main][TcpCommunicationSpi] Successfully bound shared memory communication to TCP port [port=48104, locHost=0.0.0.0/0.0.0.0]
[17:04:00,652][INFO][main][TcpCommunicationSpi] Successfully bound to TCP port [port=47106, locHost=0.0.0.0/0.0.0.0]
[17:04:00,685][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
[17:04:00,749][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival).
[17:04:00,755][WARNING][main][NoopSwapSpaceSpi] Swap space is disabled. To enable use FileSwapSpaceSpi.
[17:04:00,757][INFO][main][IgniteKernal%grid_recmd] Security status [authentication=off, tls/ssl=off]
[17:04:00,861][WARNING][main][GridCacheProcessor] Deployment mode for cache is not CONTINUOUS or SHARED (it is recommended that you change deployment mode and restart): ISOLATED
[17:04:01,040][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11217]
[17:04:01,138][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:04:01,139][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:04:01,140][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:04:01,141][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,179][SEVERE][main][IgniteKernal%grid_recmd] Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:660)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:917)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:929)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Join process timed out, did not receive response for join request (consider increasing 'joinTimeout' configuration property) [joinTimeout=60000, sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1334)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,180][SEVERE][main][IgniteKernal%grid_recmd] Got exception while starting (will rollback startup routine).
class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1510)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:917)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:929)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:660)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
... 12 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Join process timed out, did not receive response for join request (consider increasing 'joinTimeout' configuration property) [joinTimeout=60000, sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1334)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,193][INFO][main][GridTcpRestProtocol] Command protocol successfully stopped: TCP binary
[17:05:01,201][WARNING][grid-nio-worker-0-#58%grid_recmd%][TcpCommunicationSpi] Local node is not started or fully initialized [isStopping=true]
[17:05:01,201][WARNING][grid-nio-worker-1-#59%grid_recmd%][TcpCommunicationSpi] Local node is not started or fully initialized [isStopping=true]
[17:05:01,843][SEVERE][tcp-client-disco-sock-reader-#3%grid_recmd%][TcpDiscoverySpi] Failed to read message [sock=Socket[addr=/10.0.0.1,port=47500,localport=35836], locNodeId=e052dc07-8992-4834-87fd-abf84c0f644a, rmtNodeId=198bc1a5-1b1d-4b65-bccb-b2eb5ad1ec9b]
class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: sun.misc.Launcher$AppClassLoader#3415ddf5
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(JdkMarshaller.java:105)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:888)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.io.IOException: Stream closed.
at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:458)
at java.net.SocketInputStream.available(SocketInputStream.java:245)
at java.io.BufferedInputStream.read(BufferedInputStream.java:342)
at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3063)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2864)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1638)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.internal.util.IgniteUtils.readMap(IgniteUtils.java:4890)
at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:559)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.ArrayList.readObject(ArrayList.java:771)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.TreeMap.buildFromSorted(TreeMap.java:3302)
at java.util.TreeMap.readObject(TreeMap.java:3234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(JdkMarshaller.java:102)
... 2 more
[17:05:01,845][SEVERE][tcp-client-disco-sock-reader-#3%grid_recmd%][TcpDiscoverySpi] Connection failed [sock=Socket[addr=/10.0.0.1,port=47500,localport=35836], locNodeId=e052dc07-8992-4834-87fd-abf84c0f644a]
java.io.IOException: Stream closed.
at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:458)
at java.net.SocketInputStream.available(SocketInputStream.java:245)
at java.io.BufferedInputStream.read(BufferedInputStream.java:342)
at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3063)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2864)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1638)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.internal.util.IgniteUtils.readMap(IgniteUtils.java:4890)
at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:559)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.ArrayList.readObject(ArrayList.java:771)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.TreeMap.buildFromSorted(TreeMap.java:3302)
at java.util.TreeMap.readObject(TreeMap.java:3234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(JdkMarshaller.java:102)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:888)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,876][INFO][main][IgniteKernal%grid_recmd]
>>> +---------------------------------------------------------------------------------------+
>>> Ignite ver. 1.5.0-final#20151229-sha1:f1f8cda2f3f62231f42a59951bf34c39577c1bec stopped OK
>>> +---------------------------------------------------------------------------------------+
>>> Grid name: grid_recmd
>>> Grid uptime: 00:01:02:580
class org.apache.ignite.IgniteException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:882)
at org.apache.ignite.Ignition.start(Ignition.java:350)
at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1510)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:917)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:929)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
at org.apache.ignite.Ignition.start(Ignition.java:347)
... 2 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:660)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
... 12 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Join process timed out, did not receive response for join request (consider increasing 'joinTimeout' configuration property) [joinTimeout=60000, sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1334)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
the ignite process's JVM_OPTS is :
export JVM_OPTS="-Xms5g -Xmx5g -Xmn2g -server -XX:-UseGCOverheadLimit -XX:+AggressiveOpts -XX:MaxPermSize=256m -XX:+UseP
arNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB"
the srv.xml is
<bean id="cacheA" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="cache_recmd"/>
<property name="cacheMode" value="PARTITIONED"/><!-- partition -->
<property name="affinity">
<bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
<property name="hashIdResolver" >
<bean class="org.apache.ignite.cache.affinity.AffinityNodeAddressHashResolver"/>
</property>
<property name="partitions" value="280" />
</bean>
</property>
<property name="backups" value="0"/>
<property name="writeSynchronizationMode" value="PRIMARY_SYNC"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="atomicWriteOrderMode" value="PRIMARY"/>
<property name="memoryMode" value="ONHEAP_TIERED"/><!-- memory tire -->
<property name="startSize" value="#{30 * 1024 * 1024}"/>
<property name="offHeapMaxMemory" value="#{512 * 1024 * 1024}"/>
<property name="swapEnabled" value="true"/>
<property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxMemorySize" value="#{1024 * 1024 * 1024}"/>
</bean>
</property>
<property name="rebalanceMode" value="ASYNC"/>
<property name="rebalanceBatchSize" value="#{1024 * 1024}"/>
<property name="rebalanceThrottle" value="100"/>
</bean>
<bean id="igniA" class="org.apache.ignite.configuration.IgniteConfiguration" scope="singleton">
<property name="gridName" value="grid_recmd"/>
<property name="clientMode" value="false"/>
<property name="metricsLogFrequency" value="0"/><!-- 0 means disable metrics log -->
<property name="deploymentMode" value="ISOLATED"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="47500"/>
<property name="localPortRange" value="20"/>
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="shared" value="true"/>
<property name="addresses">
<list>
<value>10.0.0.1:47500..47520</value>
<value>10.0.0.2:47500..47520</value>
</list>
</property>
</bean>
</property>
<property name="joinTimeout" value="60000"/>
</bean>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="socketWriteTimeout" value="30000"/>
<property name="connectTimeout" value="50000"/>
</bean>
</property>
<property name="failureDetectionTimeout" value="60000"/>
<property name="swapSpaceSpi">
<bean class="org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi">
<property name="baseDirectory" value="/data1/home/igni1.5/my/swp"/>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<ref bean="cacheA"/>
</list>
</property>
<property name="includeEventTypes">
<list>
</list>
</property>
</bean>

Related

Ignite Local Node Binary Configuration Not Equal to Remote Node Configuration

I use the Ignite docker image to set up an ignite cluster on my local machine. Things work fine as long as I use the default configuration. When I tried overriding the config using the command
docker run -d --name my_ignite \
-p 11211:11211 -p 47100:47100 -p 47500:47500 -p 49112:49112 \
-e “OPTION_LIBS=ignite-indexing” \
-e "CONFIG_URI=file:///Users/abc/Documents/ignite_configs/ignite-config.xml" \
-v $(pwd):/Users/abc/Documents/ignite_configs apacheignite/ignite:2.7.0
Ignite starts up correctly but when I try to write to the cache I get the following exception.
My config file ignite-config.xml is
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns = "http://www.springframework.org/schema/beans"
xmlns:xsi = "http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation = "http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="binaryConfiguration">
<bean class="org.apache.ignite.configuration.BinaryConfiguration">
<property name="idMapper">
<bean class = "org.apache.ignite.binary.BinaryBasicIdMapper">
<property name = "lowerCase" value = "true"/>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
How should I fix this?
Stacktrace:
2020-05-20 14:34:49 [main] WARN o.a.i.i.p.c.d.d.t.PartitionsEvictManager - Logging at INFO level without checking if INFO level is enabled: Evict partition permits=4
2020-05-20 14:34:50 [tcp-client-disco-msg-worker-#4%fn_Instance_979931923%] ERROR ROOT - Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: GridWorker [name=tcp-client-disco-msg-worker, igniteInstanceName=svexecfn_Instance_979931923, finished=true, heartbeatTs=1590010490223]]]
org.apache.ignite.IgniteException: GridWorker [name=tcp-client-disco-msg-worker, igniteInstanceName=svexecfn_Instance_979931923, finished=true, heartbeatTs=1590010490223]
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at org.apache.ignite.internal.worker.WorkersRegistry.onStopped(WorkersRegistry.java:169)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:153)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:304)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
2020-05-20 14:34:50 [main] ERROR o.a.i.i.IgniteKernal%svexecfn_Instance_979931923 - Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]
org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1#7ed9ae94], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:603)
at org.apache.ignite.Ignition.start(Ignition.java:323)
...
Caused by: org.apache.ignite.spi.IgniteSpiException: Local node's binary configuration is not equal to remote node's binary configuration [locNodeId=4fc2317b-a7b5-43e0-bc37-20747fab2d73, rmtNodeId=a7ce1554-64cb-4de1-a71f-5c3ddb252ffa, locBinaryCfg=null, rmtBinaryCfg={globIdMapper=org.apache.ignite.binary.BinaryBasicIdMapper, compactFooter=true, globSerializer=null}]
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1888)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:304)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Take a look at the binary configuration of all your nodes:
https://apacheignite.readme.io/docs/binary-marshaller#configuring-binary-objects
you have:
Local node's binary configuration is not equal to remote node's binary configuration [locNodeId=4fc2317b-a7b5-43e0-bc37-20747fab2d73, rmtNodeId=a7ce1554-64cb-4de1-a71f-5c3ddb252ffa, locBinaryCfg=null, rmtBinaryCfg={globIdMapper=org.apache.ignite.binary.BinaryBasicIdMapper, compactFooter=true, globSerializer=null}]
which means you are inserting from a node w/a different binary config.
Take a look at the binary config of 4fc2317b-a7b5-43e0-bc37-20747fab2d73, it should be the same as the remote node.
<bean class="org.apache.ignite.configuration.BinaryConfiguration">
<property name="idMapper">
<bean class = "org.apache.ignite.binary.BinaryBasicIdMapper">
<property name = "lowerCase" value = "true"/>
</bean>
</property>
</bean>
This is binary configuration ^ it should be the same in your server node's ignite_configs/ignite-config.xml file

ERROR - WSO2 with rabbitMQ

In proxy i have this to send to rabbitMQ:
<property name="messageType" value="text/xml" scope="default"/>
<property name="contentType" value="text/xml" scope="default"/>
<property name="rabbitmq.attributes.app.id"
value="RMIS"
scope="axis2"
type="STRING"/>
<property name="rabbitmq.attributes.type"
value="BRANCH"
scope="axis2"
type="STRING"/>
<property name="rabbitmq.attributes.user.id"
value="RMIS"
scope="axis2"
type="STRING"/>
<property name="MESSAGE_ID" expression="get-property('MessageID')"/>
<script language="js">var messageID = mc.getProperty('MESSAGE_ID').substring(9,45);
mc.setProperty("RABBIT_MESSAGE_ID", messageID);</script>
<property name="rabbitmq.attributes.custom.message.id"
expression="get-property('RABBIT_MESSAGE_ID')"
scope="axis2"/>
<clone continueParent="true">
<target sequence="writeToFile"/>
</clone>
<send>
<endpoint key="rabbit"/>
</send>
in ENDPOINT:
<endpoint xmlns="http://ws.apache.org/ns/synapse" name="rabbit">
<address uri="rabbitmq://writeToQueue?rabbitmq.server.host.name=localhost&rabbitmq.server.port=5672&rabbitmq.server.user.name=guest&rabbitmq.server.password=guest&rabbitmq.queue.name=queque" format="soap11">
<suspendOnFailure>
<progressionFactor>1.0</progressionFactor>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
<retryDelay>3000</retryDelay>
</markForSuspension>
<timeout>
<responseAction>fault</responseAction>
</timeout>
</address>
</endpoint>
but I catch an error, when i send it to rabbitMQ Server:
TID[-1234] [ESB] [2016-06-03 16:32:06,479] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Unexpected error during sending message out
org.apache.axis2.transport.rabbitmq.RabbitMQMessage.(RabbitMQMessage.java:43)
org.apache.axis2.transport.rabbitmq.RabbitMQSender.sendOverAMQP(RabbitMQSender.java:84)
org.apache.axis2.transport.rabbitmq.RabbitMQSender.sendMessage(RabbitMQSender.java:72)
org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.send(DynamicAxisOperation.java:185)
org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.executeImpl(DynamicAxisOperation.java:167)
org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:542)
org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:79)
org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:461)
org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:372)
org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:65)
org.apache.synapse.endpoints.IndirectEndpoint.send(IndirectEndpoint.java:55)
org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:105)
org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:149)
org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:185)
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:395)
org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:142)
org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)
The log in WSO2 ESB there:
And i don't understand, why can not contact with rabbitMQ. In rabbitMQ also, i don't catch any error.
ERROR_DETAIL : org.apache.synapse.SynapseException: Unexpected error during sending message out at org.apache.synapse.core.axis2.Axis2Sender.handleException(Axis2Sender.java:247) at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:91) at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:461) at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:372) at org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:65) at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:105) at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81) at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48) at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:149) at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:185) at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:395) at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:142) at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.NullPointerException at org.apache.axis2.transport.rabbitmq.RabbitMQMessage.<init>(RabbitMQMessage.java:43) at org.apache.axis2.transport.rabbitmq.RabbitMQSender.sendOverAMQP(RabbitMQSender.java:84) at org.apache.axis2.transport.rabbitmq.RabbitMQSender.sendMessage(RabbitMQSender.java:72) at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112) at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.send(DynamicAxisOperation.java:185) at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.executeImpl(DynamicAxisOperation.java:167) at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149) at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:542) at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:79) ... 15 more
new stackTrace:
`[2016-06-27 17:07:00,380] ERROR - Axis2Sender Unexpected error during sending message out
java.lang.IllegalStateException: Invalid configuration: 'routingKey' must be non-null.
at com.rabbitmq.client.impl.AMQImpl$Basic$Publish.(AMQImpl.java:2278)
at com.rabbitmq.client.AMQP$Basic$Publish$Builder.build(AMQP.java:1219)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:647)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:630)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:621)
at org.apache.axis2.transport.rabbitmq.RabbitMQMessageSender.send(RabbitMQMessageSender.java:252)
at org.apache.axis2.transport.rabbitmq.RabbitMQSender.sendOverAMQP(RabbitMQSender.java:85)
at org.apache.axis2.transport.rabbitmq.RabbitMQSender.sendMessage(RabbitMQSender.java:72)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:297)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:542)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:79)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:461)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:372)
at org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:65)
at org.apache.synapse.endpoints.IndirectEndpoint.send(IndirectEndpoint.java:55)
at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:105)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:149)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:185)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:395)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:142)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
[2016-06-27 17:07:00,386] WARN - FaultHandler ERROR_CODE : 0
[2016-06-27 17:07:00,386] WARN - FaultHandler ERROR_MESSAGE : Unexpected error during sending message out
[2016-06-27 17:07:00,387] WARN - FaultHandler ERROR_DETAIL : org.apache.synapse.SynapseException: Unexpected error during sending message out
at org.apache.synapse.core.axis2.Axis2Sender.handleException(Axis2Sender.java:247)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:91)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:461)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:372)
at org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:65)
at org.apache.synapse.endpoints.IndirectEndpoint.send(IndirectEndpoint.java:55)
at org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:105)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:149)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:185)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:395)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:142)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Invalid configuration: 'routingKey' must be non-null.
at com.rabbitmq.client.impl.AMQImpl$Basic$Publish.(AMQImpl.java:2278)
at com.rabbitmq.client.AMQP$Basic$Publish$Builder.build(AMQP.java:1219)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:647)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:630)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:621)
at org.apache.axis2.transport.rabbitmq.RabbitMQMessageSender.send(RabbitMQMessageSender.java:252)
at org.apache.axis2.transport.rabbitmq.RabbitMQSender.sendOverAMQP(RabbitMQSender.java:85)
at org.apache.axis2.transport.rabbitmq.RabbitMQSender.sendMessage(RabbitMQSender.java:72)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:297)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:542)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:79)
... 16 more
[2016-06-27 17:07:00,388] WARN - FaultHandler ERROR_EXCEPTION : org.apache.synapse.SynapseException: Unexpected error during sending message out
[2016-06-27 17:07:00,388] WARN - FaultHandler FaultHandler : Endpoint [rabbit]
[2016-06-27 17:07:00,388] WARN - EndpointContext Endpoint : rabbit will be marked SUSPENDED as it failed`
Add following properties to insequence of Proxy.
<property name="OUT_ONLY" value="true"/>
<property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
Then your configurations should be like follow:
<property name="messageType" value="text/xml" scope="default"/>
<property name="contentType" value="text/xml" scope="default"/>
<property name="OUT_ONLY" value="true"/>
<property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
<property name="rabbitmq.attributes.app.id"
value="RMIS"
scope="axis2"
type="STRING"/>
<property name="rabbitmq.attributes.type"
value="BRANCH"
scope="axis2"
type="STRING"/>
<property name="rabbitmq.attributes.user.id"
value="RMIS"
scope="axis2"
type="STRING"/>
<property name="MESSAGE_ID" expression="get-property('MessageID')"/>
<script language="js">var messageID = mc.getProperty('MESSAGE_ID').substring(9,45);
mc.setProperty("RABBIT_MESSAGE_ID", messageID);</script>
<property name="rabbitmq.attributes.custom.message.id"
expression="get-property('RABBIT_MESSAGE_ID')"
scope="axis2"/>
<clone continueParent="true">
<target sequence="writeToFile"/>
</clone>
<send>
<endpoint key="rabbit"/>
</send>
According to new stackTrace, it is mention about "'routingKey' must be non-null."
Please add "rabbitmq.exchange.name" parameter and relevant value for that.
Ex:
<endpoint>
<address
uri="rabbitmq:/AMQPProxy?rabbitmq.server.host.name=192.168.0.3&rabbitmq.server.port=5672&rabbitmq.server.user.name=user&rabbitmq.server.password=abc123&rabbitmq.queue.name=queue2&rabbitmq.exchange.name=exchange2"/>
</endpoint>
If you need further clarification, please refer WSO2 docs: https://docs.wso2.com/display/ESB490/RabbitMQ+AMQP+Transport

C3P0ProxyConnection Class Not Found Error in Mule

I am working on a Mule flow which receives xml over a JMS queue and stores that xml as a clob value in an Oracle table. I am using C3P0 ComboPoolDataSource to manage connection pooling. I am getting a java.lang.NoClassDefFoundError: com/mchange/v2/c3p0/C3P0ProxyConnection error when I try to set the clob using below method call
getLobHandler().getLobCreator().setCloBAsString(pstmt,8,"Test Message");
I have checked the mule logs and it does say that it loaded the c3p0-0.9.1.2.jar
INFO 2013-10-31 19:29:06,054 [WrapperListener_start_runner] org.mule.module.launcher.MuleApplicationClassLoader: [TestWorkflow] Loading the following jar
s:
=============================
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/activemq-core-5.5.0.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/c3p0-0.9.1.2.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/geronimo-jms_1.1_spec-1.1.1.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/jdom2-2.0.5.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/log4j-1.2.16.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/mchange-commons-0.2.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/ojdbc14.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/slf4j-api-1.5.11.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/slf4j-nop-1.5.11.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/spring.jar
file:/C:/Test_mule/mule-standalone-3.4.0/apps/TestWorkflow/lib/test-notifier.jar
Below is my spring application context configuration.
<?xml version="1.0" encoding="UTF-8"?>
-<beans xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd" xmlns:util="http://www.springframework.org/schema/util" xmlns:context="http://www.springframework.org/schema/context" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.springframework.org/schema/beans"> -<bean class="com.zanaco.mule.workflow.NotificationListener" name="notifcationListener" id="notifcationListener"> <property name="oracleDAO" ref="oracleDAO"/> </bean>
<!-- Configurer that replaces ${...} placeholders with values from properties files -->
<!-- (in this case, mail and JDBC related properties) -->
<!-- <bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> <property name="locations"> <list> <value>classpath:jdbc.properties</value> </list> </property> </bean> -->
<!-- c3p0 data source added to manage Connection pooling and to handle deadlock scenario and bad performance of Apache DBCP -->
-<bean class="com.mchange.v2.c3p0.ComboPooledDataSource" id="dataSource" scope="singleton" destroy-method="close"> <property name="driverClass" value="oracle.jdbc.driver.OracleDriver"/> <property name="jdbcUrl" value="jdbc:oracle:thin:#xxxx:1521:fcdb"/> <property name="user" value="XXXXX"/> <property name="password" value="xxxxx"/> <property name="maxIdleTime" value="1000000"/>
<!-- <property name="debugUnreturnedConnectionStackTraces" value="${debugUnreturnedConnectionStackTraces}" /> <property name="unreturnedConnectionTimeout" value="${unreturnedConnectionTimeout}" /> -->
</bean>
<!-- LobHandler for well-behaved JDBC drivers -->
<!-- (simply delegating to corresponding PreparedStatement and ResultSet methods) -->
<bean class="org.springframework.jdbc.support.lob.DefaultLobHandler" id="defaultLobHandler" lazy-init="false"/>
<!-- LobHandler for Oracle JDBC drivers -->
<!-- (refers to the NativeJdbcExtractor above to get access to native OracleConnections) -->
-<bean class="org.springframework.jdbc.support.lob.OracleLobHandler" id="oracleLobHandler" lazy-init="false"> <property name="nativeJdbcExtractor" ref="nativeJdbcExtractor"/> </bean>
<!-- NativeJdbcExtractor for the C3P0 connection pool above -->
<!-- (just needed for oracleLobHandler) -->
<bean class="org.springframework.jdbc.support.nativejdbc.C3P0NativeJdbcExtractor" id="nativeJdbcExtractor" lazy-init="false"/> -<bean class="com.zanaco.db.OracleDAO" name="oracleDAO" id="oracleDAO"> -<property name="dataSource"> <ref bean="dataSource"/> </property> -<property name="lobHandler"> <ref bean="oracleLobHandler"/> </property> </bean> </beans>
and below is the log trace:
ERROR [[TestWorkflow].ReceiveObject.stage1.02] (DispatchingLogger.java:341) - Work caused exception on 'workCompleted'. Work being executed was: org.mule.
processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker#22b429
Exception in thread "[TestWorkflow].ReceiveObject.stage1.02" org.mule.api.MuleRuntimeException: An exception occurred while invoking message processor "De
faultMessageProcessorChain '(inner iterating chain) of 'ReceiveObject' processor chain'
[
DefaultJavaComponent{ReceiveObject.component.7097325},
DefaultOutboundEndpoint{endpointUri=smtp://test-abc:<password>#mail.abc.com, connector=SmtpConnector
{
name=smtpConnector
lifecycle=start
this=fc6af5
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true
connected=true
supportedProtocols=[smtp]
serviceOverrides=<none>
}
, name='endpoint.smtp.test.abc.abc.com.25', mep=ONE_WAY, properties={}, transactionConfig=Transaction{factory=null, action=INDIFFERENT, tim
eout=0}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=50000, endpointEncoding=UTF-8, disableTransportTransformer=false},
org.mule.routing.requestreply.AsyncReplyToPropertyRequestReplyReplier
]" for asynchronously.
at org.mule.processor.AsyncWorkListener.handleWorkException(AsyncWorkListener.java:74)
at org.mule.processor.AsyncWorkListener.workCompleted(AsyncWorkListener.java:50)
at org.mule.work.WorkerContext.run(WorkerContext.java:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NoClassDefFoundError: com/mchange/v2/c3p0/C3P0ProxyConnection
at org.springframework.jdbc.support.nativejdbc.C3P0NativeJdbcExtractor.doGetNativeConnection(C3P0NativeJdbcExtractor.java:97)
at org.springframework.jdbc.support.nativejdbc.NativeJdbcExtractorAdapter.getNativeConnection(NativeJdbcExtractorAdapter.java:99)
at org.springframework.jdbc.support.nativejdbc.NativeJdbcExtractorAdapter.getNativeConnectionFromStatement(NativeJdbcExtractorAdapter.java:135)
at org.springframework.jdbc.support.lob.OracleLobHandler$OracleLobCreator.getOracleConnection(OracleLobHandler.java:533)
at org.springframework.jdbc.support.lob.OracleLobHandler$OracleLobCreator.createLob(OracleLobHandler.java:493)
at org.springframework.jdbc.support.lob.OracleLobHandler$OracleLobCreator.setClobAsString(OracleLobHandler.java:420)
at com.test.db.OracleDAO.saveTransaction(OracleDAO.java:51)
at com.test.mule.workflow.NotificationListener.onCall(NotificationListener.java:53)
at org.mule.model.resolvers.CallableEntryPointResolver.invoke(CallableEntryPointResolver.java:50)
at org.mule.model.resolvers.DefaultEntryPointResolverSet.invoke(DefaultEntryPointResolverSet.java:39)
at org.mule.component.DefaultComponentLifecycleAdapter.invoke(DefaultComponentLifecycleAdapter.java:343)
at org.mule.component.AbstractJavaComponent.invokeComponentInstance(AbstractJavaComponent.java:86)
at org.mule.component.AbstractJavaComponent.doInvoke(AbstractJavaComponent.java:77)
at org.mule.component.AbstractComponent.invokeInternal(AbstractComponent.java:126)
at org.mule.component.AbstractComponent.access$000(AbstractComponent.java:61)
at org.mule.component.AbstractComponent$1$1.process(AbstractComponent.java:242)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:61)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:47)
at org.mule.processor.chain.DefaultMessageProcessorChain.doProcess(DefaultMessageProcessorChain.java:95)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:70)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.doProcess(InterceptingChainLifecycleWrapper.java:54)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:70)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.access$001(InterceptingChainLifecycleWrapper.java:26)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper$1.process(InterceptingChainLifecycleWrapper.java:70)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:61)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:47)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.process(InterceptingChainLifecycleWrapper.java:65)
at org.mule.component.AbstractComponent.process(AbstractComponent.java:160)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:61)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:47)
at org.mule.processor.chain.DefaultMessageProcessorChain.doProcess(DefaultMessageProcessorChain.java:95)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:70)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:47)
at org.mule.processor.AbstractInterceptingMessageProcessorBase.processNext(AbstractInterceptingMessageProcessorBase.java:106)
at org.mule.interceptor.AbstractEnvelopeInterceptor.process(AbstractEnvelopeInterceptor.java:55)
at org.mule.processor.AsyncInterceptingMessageProcessor.processNextTimed(AsyncInterceptingMessageProcessor.java:122)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker$1.process(AsyncInterceptingMessageProcessor.java:192)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker$1.process(AsyncInterceptingMessageProcessor.java:185)
at org.mule.execution.ExecuteCallbackInterceptor.execute(ExecuteCallbackInterceptor.java:20)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:34)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:18)
at org.mule.execution.BeginAndResolveTransactionInterceptor.execute(BeginAndResolveTransactionInterceptor.java:58)
at org.mule.execution.ResolvePreviousTransactionInterceptor.execute(ResolvePreviousTransactionInterceptor.java:48)
at org.mule.execution.SuspendXaTransactionInterceptor.execute(SuspendXaTransactionInterceptor.java:54)
at org.mule.execution.ValidateTransactionalStateInterceptor.execute(ValidateTransactionalStateInterceptor.java:44)
at org.mule.execution.IsolateCurrentTransactionInterceptor.execute(IsolateCurrentTransactionInterceptor.java:44)
at org.mule.execution.ExternalTransactionInterceptor.execute(ExternalTransactionInterceptor.java:52)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:32)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:17)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:113)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:34)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker.doRun(AsyncInterceptingMessageProcessor.java:184)
at org.mule.work.AbstractMuleEventWork.run(AbstractMuleEventWork.java:43)
at org.mule.work.WorkerContext.run(WorkerContext.java:311)
... 3 more
Caused by: java.lang.ClassNotFoundException: com.mchange.v2.c3p0.C3P0ProxyConnection
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 61 more
It's as if the classloader into which com.test.db.OracleDAO.saveTransaction() executes is not the classloader of the TestWorkflow application.
Do you have several applications?
If not, can you try adding c3p0-0.9.1.2.jar to $MULE_HOME/lib/user to see if it helps.

system-properties In standalone-full.xml

Jboss version: jboss-as-7.1.0.Final
can we add
<system-properties>
<property name="x" value="/opt/db/common"/>
</system-properties>
In standalone-full.xml?
I am getting below error:
20:54:35,726 INFO [org.jboss.modules] JBoss Modules version 1.1.1.GA
20:54:35,884 INFO [org.jboss.msc] JBoss MSC version 1.0.2.GA
20:54:35,942 INFO [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
20:54:36,575 INFO [org.jboss.as] JBAS015950: JBoss AS 7.1.0.Final "Thunder" stopped in 5ms
20:54:36,569 ERROR [org.jboss.as.controller] JBAS014601: Error booting the container: java.lang.RuntimeException: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014676: Failed to parse configuration
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:161) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_23]
Caused by: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014676: Failed to parse configuration
at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:125) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:187) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.ServerService.boot(ServerService.java:261) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:155) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
... 1 more
Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[479,1]
Message: JBAS014789: Unexpected element '{urn:jboss:domain:1.1}system-properties' encountered
at org.jboss.as.controller.parsing.ParseUtils.unexpectedElement(ParseUtils.java:85) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.readServerElement_1_1(StandaloneXml.java:350) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:126) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:100) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
at org.jboss.staxmapper.XMLMapperImpl.parseDocument(XMLMapperImpl.java:69) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:117) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
... 4 more
at [row,col]:[479,1] I am having below entry:
<system-properties>
<property name="x" value="/opt/db/common"/>
<property name="y" value="/opt/db/engine/work/q22222-161-1/engine/info/exye/2.0/files/config.xml"/>
<property name="z" value="/opt/db/engine/work/q22222-161-1/engine/info/exye/2.0"/>
<property name="m" value="/opt/db/engine/work/q22222-161-1/engine/info/exye/2.0/log"/>
<property name="n" value="Member1"/>
<property name="log4j.ignoreTCL" value="true"/>
<property name="org.apache.tomcat.util.http.Parameters.MAX_COUNT" value="5000"/>
</system-properties>
IIRC it's been fixed upstream, but some versions require the <system-properties/> to be directly under the <extensions/> tag.
Try something like:
<?xml version='1.0' encoding='UTF-8'?>
<server xmlns="urn:jboss:domain:1.1">
<extensions>
...
</extensions>
<system-properties>
<property name="x" value="/opt/db/common"/>
<property name="y" value="/opt/db/engine/work/q22222-161-1/engine/info/exye/2.0/files/config.xml"/>
<property name="z" value="/opt/db/engine/work/q22222-161-1/engine/info/exye/2.0"/>
<property name="m" value="/opt/db/engine/work/q22222-161-1/engine/info/exye/2.0/log"/>
<property name="n" value="Member1"/>
<property name="log4j.ignoreTCL" value="true"/>
<property name="org.apache.tomcat.util.http.Parameters.MAX_COUNT" value="5000"/>
</system-properties>
...
</server>

Using WSO2 Identity Server with OpenLDAP

While trying to use WSO2 Identity Server with OpenLDAP after following the steps on:
http://wso2.org/project/solutions/identity/3.2.3/docs/user-core/admin_guide.html#LDAP
I get this message, when trying to authenticate:
TID: [0] [WSO2 Identity Server] [2012-07-30 13:30:34,088] WARN {org.wso2.carbon.core.services.authentication.AuthenticationAdmin} - None active tenant trying to sign in.example.com {org.wso2.carbon.core.services.authentication.AuthenticationAdmin}
TID: [0] [WSO2 Identity Server] [2012-07-30 13:30:34,088] ERROR {org.wso2.carbon.core.services.authentication.AuthenticationAdmin} - Your account is not active. Please activate your account first. {org.wso2.carbon.core.services.authentication.AuthenticationAdmin}
org.wso2.carbon.core.common.AuthenticationException: Your account is not active. Please activate your account first.
at org.wso2.carbon.core.services.authentication.AuthenticationAdmin.login(AuthenticationAdmin.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:181)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
at org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:199)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:641)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:90)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:67)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.wso2.carbon.bridge.BridgeServlet.service(BridgeServlet.java:164)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.wso2.carbon.server.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:154)
at org.wso2.carbon.server.TomcatServer$1.invoke(TomcatServer.java:254)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:399)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:396)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:356)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1534)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
TID: [] [WSO2 Identity Server] [2012-07-30 13:30:34,100] ERROR {org.wso2.carbon.authenticator.proxy.AuthenticationAdminClient} - Error occurred while logging in {org.wso2.carbon.authenticator.proxy.AuthenticationAdminClient}
org.apache.axis2.AxisFault: Your account is not active. Please activate your account first.
at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:375)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:421)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at org.wso2.carbon.authenticator.stub.AuthenticationAdminStub.login(AuthenticationAdminStub.java:659)
at org.wso2.carbon.authenticator.proxy.AuthenticationAdminClient.login(AuthenticationAdminClient.java:64)
at org.wso2.carbon.ui.DefaultCarbonAuthenticator.authenticate(DefaultCarbonAuthenticator.java:142)
at org.wso2.carbon.ui.DefaultCarbonAuthenticator.authenticate(DefaultCarbonAuthenticator.java:85)
at org.wso2.carbon.ui.CarbonSecuredHttpContext.handleSecurity(CarbonSecuredHttpContext.java:387)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:86)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:67)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.wso2.carbon.bridge.BridgeServlet.service(BridgeServlet.java:164)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.wso2.carbon.server.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:154)
at org.wso2.carbon.server.TomcatServer$1.invoke(TomcatServer.java:254)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:399)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:396)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:356)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1534)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
TID: [] [WSO2 Identity Server] [2012-07-30 13:30:34,105] ERROR {org.wso2.carbon.ui.CarbonSecuredHttpContext} - error occurred while login {org.wso2.carbon.ui.CarbonSecuredHttpContext}
org.wso2.carbon.core.common.AuthenticationException: Exception occured
at org.wso2.carbon.ui.DefaultCarbonAuthenticator.authenticate(DefaultCarbonAuthenticator.java:87)
at org.wso2.carbon.ui.CarbonSecuredHttpContext.handleSecurity(CarbonSecuredHttpContext.java:387)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.handleRequest(ServletRegistration.java:86)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:111)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:67)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.wso2.carbon.bridge.BridgeServlet.service(BridgeServlet.java:164)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.wso2.carbon.server.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:154)
at org.wso2.carbon.server.TomcatServer$1.invoke(TomcatServer.java:254)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:399)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:396)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:356)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1534)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.axis2.AxisFault: Exception occured
at org.wso2.carbon.ui.DefaultCarbonAuthenticator.authenticate(DefaultCarbonAuthenticator.java:150)
at org.wso2.carbon.ui.DefaultCarbonAuthenticator.authenticate(DefaultCarbonAuthenticator.java:85)
... 25 more
Caused by: org.wso2.carbon.core.common.AuthenticationException: org.apache.axis2.AxisFault: Your account is not active. Please activate your account first.
at org.wso2.carbon.authenticator.proxy.AuthenticationAdminClient.login(AuthenticationAdminClient.java:70)
at org.wso2.carbon.ui.DefaultCarbonAuthenticator.authenticate(DefaultCarbonAuthenticator.java:142)
... 26 more
Caused by: org.apache.axis2.AxisFault: Your account is not active. Please activate your account first.
at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:375)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:421)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at org.wso2.carbon.authenticator.stub.AuthenticationAdminStub.login(AuthenticationAdminStub.java:659)
at org.wso2.carbon.authenticator.proxy.AuthenticationAdminClient.login(AuthenticationAdminClient.java:64)
... 27 more
My user-mgt.xml is:
<UserManager>
<Realm>
<Configuration>
<AdminRole>admin</AdminRole>
<AdminUser>
<UserName>admin#example.com</UserName>
<Password>admin</Password>
</AdminUser>
<EveryOneRoleName>everyone</EveryOneRoleName> <!-- By default users in this role sees the registry root -->
<ReadOnly>true</ReadOnly>
<MaxUserNameListLength>500</MaxUserNameListLength>
<Property name="url">jdbc:h2:repository/database/WSO2CARBON_DB</Property>
<Property name="userName">wso2carbon</Property>
<Property name="password">wso2carbon</Property>
<Property name="driverName">org.h2.Driver</Property>
<Property name="maxActive">50</Property>
<Property name="maxWait">60000</Property>
<Property name="minIdle">5</Property>
</Configuration>
<UserStoreManager class="org.wso2.carbon.user.core.ldap.LDAPUserStoreManager">
<Property name="ReadOnly">true</Property>
<Property name="MaxUserNameListLength">100</Property>
<Property name="ConnectionURL">ldap://192.168.81.128:389</Property>
<Property name="ConnectionName">uid=admin#example.com,ou=users,dc=maxcrc,dc=com</Property>
<Property name="ConnectionPassword">admin</Property>
<Property name="UserSearchBase">ou=users,dc=maxcrc,dc=com</Property>
<Property name="UserNameListFilter">(objectClass=person)</Property>
<Property name="UserNameAttribute">uid</Property>
<Property name="ReadLDAPGroups">false</Property>
<Property name="GroupSearchBase">ou=system,dc=maxcrc,dc=com</Property>
<Property name="GroupSearchFilter">(objectClass=groupOfNames)</Property>
<Property name="GroupNameAttribute">cn</Property>
<Property name="MembershipAttribute">member</Property>
</UserStoreManager>
<AuthorizationManager class="org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager"></AuthorizationManager>
</Realm>
</UserManager>
Can you give any suggestions as to where the error might be?
The only user on my OpenLDAP installation is admin#example.com with password admin
Since you are trying to connect to the OpenLDAP in read only mode use the LDAPUserStoreManager instead the ApacheDSUserStoreManager. When you do the configurations do not comment out or remove configuration elements, cause that would would cause for NULL POINTER EXCEPTIONS.