Ignite Local Node Binary Configuration Not Equal to Remote Node Configuration - ignite

I use the Ignite docker image to set up an ignite cluster on my local machine. Things work fine as long as I use the default configuration. When I tried overriding the config using the command
docker run -d --name my_ignite \
-p 11211:11211 -p 47100:47100 -p 47500:47500 -p 49112:49112 \
-e “OPTION_LIBS=ignite-indexing” \
-e "CONFIG_URI=file:///Users/abc/Documents/ignite_configs/ignite-config.xml" \
-v $(pwd):/Users/abc/Documents/ignite_configs apacheignite/ignite:2.7.0
Ignite starts up correctly but when I try to write to the cache I get the following exception.
My config file ignite-config.xml is
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns = "http://www.springframework.org/schema/beans"
xmlns:xsi = "http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation = "http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="binaryConfiguration">
<bean class="org.apache.ignite.configuration.BinaryConfiguration">
<property name="idMapper">
<bean class = "org.apache.ignite.binary.BinaryBasicIdMapper">
<property name = "lowerCase" value = "true"/>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
How should I fix this?
Stacktrace:
2020-05-20 14:34:49 [main] WARN o.a.i.i.p.c.d.d.t.PartitionsEvictManager - Logging at INFO level without checking if INFO level is enabled: Evict partition permits=4
2020-05-20 14:34:50 [tcp-client-disco-msg-worker-#4%fn_Instance_979931923%] ERROR ROOT - Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: GridWorker [name=tcp-client-disco-msg-worker, igniteInstanceName=svexecfn_Instance_979931923, finished=true, heartbeatTs=1590010490223]]]
org.apache.ignite.IgniteException: GridWorker [name=tcp-client-disco-msg-worker, igniteInstanceName=svexecfn_Instance_979931923, finished=true, heartbeatTs=1590010490223]
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at org.apache.ignite.internal.worker.WorkersRegistry.onStopped(WorkersRegistry.java:169)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:153)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:304)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
2020-05-20 14:34:50 [main] ERROR o.a.i.i.IgniteKernal%svexecfn_Instance_979931923 - Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]
org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1#7ed9ae94], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:603)
at org.apache.ignite.Ignition.start(Ignition.java:323)
...
Caused by: org.apache.ignite.spi.IgniteSpiException: Local node's binary configuration is not equal to remote node's binary configuration [locNodeId=4fc2317b-a7b5-43e0-bc37-20747fab2d73, rmtNodeId=a7ce1554-64cb-4de1-a71f-5c3ddb252ffa, locBinaryCfg=null, rmtBinaryCfg={globIdMapper=org.apache.ignite.binary.BinaryBasicIdMapper, compactFooter=true, globSerializer=null}]
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1888)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:304)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Take a look at the binary configuration of all your nodes:
https://apacheignite.readme.io/docs/binary-marshaller#configuring-binary-objects
you have:
Local node's binary configuration is not equal to remote node's binary configuration [locNodeId=4fc2317b-a7b5-43e0-bc37-20747fab2d73, rmtNodeId=a7ce1554-64cb-4de1-a71f-5c3ddb252ffa, locBinaryCfg=null, rmtBinaryCfg={globIdMapper=org.apache.ignite.binary.BinaryBasicIdMapper, compactFooter=true, globSerializer=null}]
which means you are inserting from a node w/a different binary config.
Take a look at the binary config of 4fc2317b-a7b5-43e0-bc37-20747fab2d73, it should be the same as the remote node.

<bean class="org.apache.ignite.configuration.BinaryConfiguration">
<property name="idMapper">
<bean class = "org.apache.ignite.binary.BinaryBasicIdMapper">
<property name = "lowerCase" value = "true"/>
</bean>
</property>
</bean>
This is binary configuration ^ it should be the same in your server node's ignite_configs/ignite-config.xml file

Related

Plugin, authenticateNode NotSerializableException

I have troubles implementing an authentication plugin. A NotSerializableException is raised on the instance of my AuthPluginProcessor class. Disabling the authentication with enabled returning false makes the node start.
I think I am missing something (I tried adding parameters to authenticateNode without success), below is the minimum code reproducing the error.
AuthPluginProcessor :
public class AuthPluginProcessor implements IgnitePlugin, GridSecurityProcessor, Serializable
{
#Override
public SecurityContext authenticateNode(ClusterNode clusterNode, SecurityCredentials securityCredentials) throws IgniteCheckedException
{
return new SecurityContext() {
#Override
public SecuritySubject subject() { return null; }
#Override
public boolean taskOperationAllowed(String s, SecurityPermission securityPermission) { return true; }
#Override
public boolean cacheOperationAllowed(String s, SecurityPermission securityPermission) { return true; }
#Override
public boolean serviceOperationAllowed(String s, SecurityPermission securityPermission) { return true; }
#Override
public boolean systemOperationAllowed(SecurityPermission securityPermission) { return true; }
};
}
#Override
public boolean isGlobalNodeAuthentication() { return true; }
#Override
public boolean enabled() { return true; }
// others empty methods ...
}
Here is most of the log file :
[16:38:10,598][INFO][main][IgniteKernal] PID: 10152
[16:38:10,598][INFO][main][IgniteKernal] Language runtime: Java Platform API Specification ver. 1.8
[16:38:10,599][INFO][main][IgniteKernal] VM information: Java(TM) SE Runtime Environment 1.8.0_172-b11 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.172-b11
[16:38:10,602][INFO][main][IgniteKernal] VM total memory: 0.96GB
[16:38:10,602][INFO][main][IgniteKernal] Remote Management [restart: on, REST: on, JMX (remote: on, port: 49186, auth: off, ssl: off)]
[16:38:10,603][INFO][main][IgniteKernal] Logger: JavaLogger [quiet=false, config=null]
[16:38:10,603][INFO][main][IgniteKernal] IGNITE_HOME=D:\db\Ignite2.4
[16:38:10,605][INFO][main][IgniteKernal] VM arguments: [-Xms1g, -Xmx1g, -XX:+AggressiveOpts, -XX:MaxMetaspaceSize=256m, -DIGNITE_QUIET=false, -DIGNITE_SUCCESS_FILE=D:\db\Ignite2.4\work\ignite_success_18756fda-d6de-4b6f-9ff2-5dc0bbe5f8b7, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=49186, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -DIGNITE_HOME=D:\db\Ignite2.4, -DIGNITE_PROG_NAME=ignite.bat]
[16:38:10,606][INFO][main][IgniteKernal] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemCacheMemorySize property to change the setting.
[16:38:10,613][INFO][main][IgniteKernal] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
[16:38:10,613][WARNING][main][IgniteKernal] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons)
[16:38:10,617][WARNING][pub-#19][GridDiagnostic] This operating system has been tested less rigorously: Windows 10 10.0 amd64. Our team will appreciate the feedback if you experience any problems running ignite in this environment.
[16:38:10,620][INFO][main][IgniteKernal] 3-rd party licenses can be found at: D:\db\Ignite2.4\libs\licenses
[16:38:10,683][INFO][main][IgnitePluginProcessor] Configured plugins:
[16:38:10,684][INFO][main][IgnitePluginProcessor] ^-- AuthPlugin 0.1.0
[16:38:10,686][INFO][main][IgnitePluginProcessor] ^-- null
[16:38:10,686][INFO][main][IgnitePluginProcessor]
[16:38:10,725][INFO][main][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=47101, locHost=0.0.0.0/0.0.0.0, selectorsCnt=4, selectorSpins=0, pairedConn=false]
[16:38:11,230][WARNING][main][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
[16:38:11,242][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
[16:38:11,263][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival).
[16:38:11,265][INFO][main][IgniteKernal] Security status [authentication=on, tls/ssl=off]
[16:38:11,314][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47501, localHost=0.0.0.0/0.0.0.0, locNodeId=8acfdf1f-1371-443a-a6b1-38189fcee66f]
[16:38:12,331][INFO][main][PdsFoldersResolver] Unable to acquire lock to file [D:\db\Ignite2.4\work\db\node00-3413bf9c-56ea-4041-ae02-6967e6118354], reason: Failed to acquire file lock during 1 sec, (locked by [4c07699d-cb8f-4e20-9531-9154eb4e8a12][]): D:\db\Ignite2.4\work\db\node00-3413bf9c-56ea-4041-ae02-6967e6118354\lock
[16:38:12,364][INFO][main][PdsFoldersResolver] Successfully locked persistence storage folder [D:\db\Ignite2.4\work\db\node01-e607d470-4b11-42c1-b054-2a60a5ddaa02]
[16:38:12,364][INFO][main][PdsFoldersResolver] Consistent ID used for local node is [e607d470-4b11-42c1-b054-2a60a5ddaa02] according to persistence data storage folders
[16:38:12,365][INFO][main][CacheObjectBinaryProcessorImpl] Resolved directory for serialized binary metadata: D:\db\Ignite2.4\work\binary_meta\node01-e607d470-4b11-42c1-b054-2a60a5ddaa02
[16:38:12,530][INFO][main][FilePageStoreManager] Resolved page store work directory: D:\db\Ignite2.4\work\db\node01-e607d470-4b11-42c1-b054-2a60a5ddaa02
[16:38:12,531][INFO][main][FileWriteAheadLogManager] Resolved write ahead log work directory: D:\db\Ignite2.4\work\db\wal\node01-e607d470-4b11-42c1-b054-2a60a5ddaa02
[16:38:12,531][INFO][main][FileWriteAheadLogManager] Resolved write ahead log archive directory: D:\db\Ignite2.4\work\db\wal\archive\node01-e607d470-4b11-42c1-b054-2a60a5ddaa02
[16:38:12,543][INFO][main][FileWriteAheadLogManager] Started write-ahead log manager [mode=LOG_ONLY]
[16:38:12,552][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=null, endMarker=null]
[16:38:12,565][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=100,0 MiB, pages=24936, tableSize=1,4 MiB, checkpointBuffer=100,0 MiB]
[16:38:12,566][INFO][main][GridCacheDatabaseSharedManager] Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
[16:38:12,572][INFO][main][GridCacheDatabaseSharedManager] Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
[16:38:12,574][INFO][main][GridCacheDatabaseSharedManager] Finished applying WAL changes [updatesApplied=0, time=0ms]
[16:38:12,633][INFO][main][ClientListenerProcessor] Client connector processor has started on TCP port 10801
[16:38:12,665][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11212]
[16:38:13,969][INFO][main][IgniteKernal] Non-loopback local IPs: 124.0.0.9, fe80:0:0:0:15be:bcfe:4ff6:1422%eth10, fe80:0:0:0:49ae:87cd:b8a4:a6bb%eth2, fe80:0:0:0:7943:2fb2:1da3:e69b%eth1, fe80:0:0:0:859b:b174:7412:4eb7%wlan1, fe80:0:0:0:d1fb:422:9dfa:aada%eth17, fe80:0:0:0:dd7d:8d79:7438:1db7%eth0, fe80:0:0:0:e593:8733:6979:aa7f%wlan0
[16:38:13,969][INFO][main][IgniteKernal] Enabled local MACs: 00090FFE0001, 00FF20F9E686, 025041000001, 184F32F210F1, 1A4F32F210F1, 54587804FD10, 54C6D294231E, 54CCB184D71B, 64006A8BA7BF
[16:38:13,995][SEVERE][main][IgniteKernal] Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.internal.IgniteKernal$5#3f390d63], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:892)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1669)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:983)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1973)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1716)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1144)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1062)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:948)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:847)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:717)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:686)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:302)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to authenticate local node (will shutdown local node).
at org.apache.ignite.spi.discovery.tcp.ServerImpl.localAuthentication(ServerImpl.java:991)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:862)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:364)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1930)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
... 13 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to serialize object: test.ignite.AuthPluginProcessor$1#1af687fe
at org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:103)
at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:70)
at org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:117)
at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9984)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.localAuthentication(ServerImpl.java:985)
... 17 more
Caused by: java.io.NotSerializableException: test.ignite.AuthPluginProcessor$1
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:98)
... 22 more
Here is the configuration file :
<bean abstract="true" id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Set to true to enable distributed class loading for examples, default is false. -->
<property name="peerClassLoadingEnabled" value="true"/>
<!-- Enable task execution events for examples. -->
<property name="includeEventTypes">
<list>
<!--Task execution events-->
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_TIMEDOUT"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_SESSION_ATTR_SET"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_REDUCED"/>
<!--Cache events-->
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_READ"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED"/>
</list>
</property>
<property name="pluginConfigurations">
<bean class="test.ignite.AuthPluginConfiguration"/>
</property>
<!--<property name="binaryConfiguration">
<bean class="org.apache.ignite.configuration.BinaryConfiguration">
<property name="compactFooter" value="false"/>
<property name="idMapper">
<bean class="org.apache.ignite.binary.BinaryBasicIdMapper">
<property name="lowerCase" value="true"/>
</bean>
</property>
</bean>
</property>-->
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery. For information on all options refer
to our documentation: http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>124.0.0.127</value>
<value>124.0.0.125</value>
<value>124.0.0.9</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
</bean>
</property>
</bean>
</property>
</bean>
Your SecurityContext doesn't implement Serializable likely.
Move to named inner class => add implements.

Apache Ignite does not distribute an executing closure to a newly spawned node in the compute grid

Apache Ignite Version is: 2.1.0
I am using the default configuration for client & servers. The following is the client configuration. The server configuration does not have the "clientMode" property.
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean abstract="true" id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Set to true to enable distributed class loading for examples, default is false. -->
<property name="peerClassLoadingEnabled" value="true"/>
<property name="clientMode" value="true"/>
<!-- Enable task execution events for examples. -->
<property name="includeEventTypes">
<list>
<!--Task execution events-->
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_TIMEDOUT"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_SESSION_ATTR_SET"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_REDUCED"/>
<!--Cache events -->
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_READ"/>
<util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED"/>
</list>
</property>
<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!--
Ignite provides several options for automatic discovery that can be used
instead os static IP based discovery. For information on all options refer
to our documentation: http://apacheignite.readme.io/docs/cluster-config
-->
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<!-- <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder"> -->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>xxx.1y4.1zz.91:47500..47509</value>
<value>xxx.1y4.1zz.92:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
The closure gets executed over the server nodes in the grid as expected.
When we add a new node by executing the below command to the grid during the execution of closure
.\ignite.bat ..\examples\config\example-ignite.xml
The existing nodes acknowledge the addition of the new node in the grid but the closure is not distributed to the newly added node.
Is there any configuration available to enable execution of closure to a node added during the execution of the closure?
Edit 1:
Below is the IgniteClosure implementation class:
public class SimpleInterestClosure implements IgniteClosure<SimpleInterestParam, AccruedSimpleInterest> {
private static final long serialVersionUID = -5542687183747797356L;
private static final BigInteger HUNDRED = new BigInteger("100".getBytes());
private static Logger log = Logger.getLogger("SimpleInterestClosure");
#Override
public AccruedSimpleInterest apply(SimpleInterestParam e) {
BigInteger si = e.getPrincipal().multiply(new BigInteger(e.getDurationInYears().toString().getBytes())).
multiply(new BigInteger(e.getInterestRate().toString().getBytes())).divide(SimpleInterestClosure.HUNDRED);
log.info("Calculated SI for id=" + e.getId());
return new AccruedSimpleInterest(e, si);
}
}
Edit 2:
Below is the method which invokes the IgniteClosure implementation
public void method() throws IgniteException, IOException {
Factory<SimpleInterestClosure> siClosureFactory = FactoryBuilder.singletonfactoryOf( new SimpleInterestClosure());
ClassPathResource ress = new ClassPathResource("example-ignite.xml");
File file = new File(ress.getPath());
try (Ignite ignite = Ignition.start(file.getPath())) {
log.info("Started Ignite Cluster");
IgniteFuture<Collection<AccruedSimpleInterest>> igniteFuture = ignite.compute()
.applyAsync(siClosureFactory.create(), createParamCollection());
Collection<AccruedSimpleInterest> res = igniteFuture.get();
}
}
This sounds like you're looking for job stealing: http://apacheignite.readme.io/docs/load-balancing#job-stealing
Although it currently has a bug that may be an issue in this particular case: http://issues.apache.org/jira/browse/IGNITE-1267

IgniteDataStreamer can't wirte data to ignite cluster

My ignite cluster is deploy on 2 machines. Each machine has 4 ignite process. like:
$jps
30678 CommandLineStartup
30723 CommandLineStartup
52461 Jps
30698 CommandLineStartup
30681 CommandLineStartup
When there is little data in the ignite cluster, I can use IgniteDataStreamer to write data into the ignite cluster. But if there have been more than 200G data in the ignite cluster, I can't use IgniteDataStreamer to write more data into the cluster. The memory and disk space are both enough. and the CPU load average is 2.09. The client log is :
[17:04:00,407][INFO][main][IgniteKernal%grid_recmd] Daemon mode: off
[17:04:00,407][INFO][main][IgniteKernal%grid_recmd] OS: Linux
[17:04:00,407][INFO][main][IgniteKernal%grid_recmd] OS user: mqq
[17:04:00,408][INFO][main][IgniteKernal%grid_recmd] Language runtime: Java Platform API Specification ver. 1.7
[17:04:00,408][INFO][main][IgniteKernal%grid_recmd] VM information: Java(TM) SE Runtime Environment 1.7.0_72-b14 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 24.72-b04
[17:04:00,409][INFO][main][IgniteKernal%grid_recmd] VM total memory: 2.9GB
[17:04:00,409][INFO][main][IgniteKernal%grid_recmd] Remote Management [restart: off, REST: on, JMX (remote: off)]
[17:04:00,409][INFO][main][IgniteKernal%grid_recmd] IGNITE_HOME=/data1/home/igni1.5
[17:04:00,410][INFO][main][IgniteKernal%grid_recmd] VM arguments: [-Xms3g, -Xmx3g, -XX:-UseGCOverheadLimit, -XX:+AggressiveOpts, -XX:MaxPermSize=256m, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+UseTLAB, -DIGNITE_QUIET=false]
[17:04:00,418][INFO][main][IgniteKernal%grid_recmd] Configured caches ['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache', 'cache_recmd']
[17:04:00,422][INFO][main][IgniteKernal%grid_recmd] 3-rd party licenses can be found at: /data1/home/igni1.5/libs/licenses
[17:04:00,518][INFO][main][IgniteKernal%grid_recmd] Non-loopback local IPs: 10.0.0.1
[17:04:00,519][INFO][main][IgniteKernal%grid_recmd] Enabled local MACs: 9017ACC1492D
[17:04:00,530][INFO][main][IgnitePluginProcessor] Configured plugins:
[17:04:00,530][INFO][main][IgnitePluginProcessor] ^-- None
[17:04:00,530][INFO][main][IgnitePluginProcessor]
[17:04:00,574][WARNING][main][TcpCommunicationSpi] Failure detection timeout will be ignored (one of SPI parameters has been set explicitly)
[17:04:00,627][INFO][main][TcpCommunicationSpi] IPC shared memory server endpoint started [port=48104, tokDir=/data1/home/igni1.5/work/ipc/shmem/e052dc07-8992-4834-87fd-abf84c0f644a-14472]
[17:04:00,628][INFO][main][TcpCommunicationSpi] Successfully bound shared memory communication to TCP port [port=48104, locHost=0.0.0.0/0.0.0.0]
[17:04:00,652][INFO][main][TcpCommunicationSpi] Successfully bound to TCP port [port=47106, locHost=0.0.0.0/0.0.0.0]
[17:04:00,685][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
[17:04:00,749][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival).
[17:04:00,755][WARNING][main][NoopSwapSpaceSpi] Swap space is disabled. To enable use FileSwapSpaceSpi.
[17:04:00,757][INFO][main][IgniteKernal%grid_recmd] Security status [authentication=off, tls/ssl=off]
[17:04:00,861][WARNING][main][GridCacheProcessor] Deployment mode for cache is not CONTINUOUS or SHARED (it is recommended that you change deployment mode and restart): ISOLATED
[17:04:01,040][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11217]
[17:04:01,138][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:04:01,139][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:04:01,140][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:04:01,141][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] Exception on joining: Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1266)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1241)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:579)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:505)
at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:118)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1508)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1327)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,179][SEVERE][main][IgniteKernal%grid_recmd] Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:660)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:917)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:929)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Join process timed out, did not receive response for join request (consider increasing 'joinTimeout' configuration property) [joinTimeout=60000, sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1334)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,180][SEVERE][main][IgniteKernal%grid_recmd] Got exception while starting (will rollback startup routine).
class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1510)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:917)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:929)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
at org.apache.ignite.Ignition.start(Ignition.java:347)
at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:660)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
... 12 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Join process timed out, did not receive response for join request (consider increasing 'joinTimeout' configuration property) [joinTimeout=60000, sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1334)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,193][INFO][main][GridTcpRestProtocol] Command protocol successfully stopped: TCP binary
[17:05:01,201][WARNING][grid-nio-worker-0-#58%grid_recmd%][TcpCommunicationSpi] Local node is not started or fully initialized [isStopping=true]
[17:05:01,201][WARNING][grid-nio-worker-1-#59%grid_recmd%][TcpCommunicationSpi] Local node is not started or fully initialized [isStopping=true]
[17:05:01,843][SEVERE][tcp-client-disco-sock-reader-#3%grid_recmd%][TcpDiscoverySpi] Failed to read message [sock=Socket[addr=/10.0.0.1,port=47500,localport=35836], locNodeId=e052dc07-8992-4834-87fd-abf84c0f644a, rmtNodeId=198bc1a5-1b1d-4b65-bccb-b2eb5ad1ec9b]
class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: sun.misc.Launcher$AppClassLoader#3415ddf5
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(JdkMarshaller.java:105)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:888)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.io.IOException: Stream closed.
at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:458)
at java.net.SocketInputStream.available(SocketInputStream.java:245)
at java.io.BufferedInputStream.read(BufferedInputStream.java:342)
at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3063)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2864)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1638)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.internal.util.IgniteUtils.readMap(IgniteUtils.java:4890)
at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:559)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.ArrayList.readObject(ArrayList.java:771)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.TreeMap.buildFromSorted(TreeMap.java:3302)
at java.util.TreeMap.readObject(TreeMap.java:3234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(JdkMarshaller.java:102)
... 2 more
[17:05:01,845][SEVERE][tcp-client-disco-sock-reader-#3%grid_recmd%][TcpDiscoverySpi] Connection failed [sock=Socket[addr=/10.0.0.1,port=47500,localport=35836], locNodeId=e052dc07-8992-4834-87fd-abf84c0f644a]
java.io.IOException: Stream closed.
at java.net.AbstractPlainSocketImpl.available(AbstractPlainSocketImpl.java:458)
at java.net.SocketInputStream.available(SocketInputStream.java:245)
at java.io.BufferedInputStream.read(BufferedInputStream.java:342)
at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3063)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2864)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1638)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.internal.util.IgniteUtils.readMap(IgniteUtils.java:4890)
at org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:559)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.ArrayList.readObject(ArrayList.java:771)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at java.util.TreeMap.buildFromSorted(TreeMap.java:3302)
at java.util.TreeMap.readObject(TreeMap.java:3234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal(JdkMarshaller.java:102)
at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketReader.body(ClientImpl.java:888)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[17:05:01,876][INFO][main][IgniteKernal%grid_recmd]
>>> +---------------------------------------------------------------------------------------+
>>> Ignite ver. 1.5.0-final#20151229-sha1:f1f8cda2f3f62231f42a59951bf34c39577c1bec stopped OK
>>> +---------------------------------------------------------------------------------------+
>>> Grid name: grid_recmd
>>> Grid uptime: 00:01:02:580
class org.apache.ignite.IgniteException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:882)
at org.apache.ignite.Ignition.start(Ignition.java:350)
at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30)
at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1510)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:917)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1547)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:929)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554)
at org.apache.ignite.Ignition.start(Ignition.java:347)
... 2 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:258)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:660)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1505)
... 12 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Join process timed out, did not receive response for join request (consider increasing 'joinTimeout' configuration property) [joinTimeout=60000, sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]]
at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1334)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
the ignite process's JVM_OPTS is :
export JVM_OPTS="-Xms5g -Xmx5g -Xmn2g -server -XX:-UseGCOverheadLimit -XX:+AggressiveOpts -XX:MaxPermSize=256m -XX:+UseP
arNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB"
the srv.xml is
<bean id="cacheA" class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="cache_recmd"/>
<property name="cacheMode" value="PARTITIONED"/><!-- partition -->
<property name="affinity">
<bean class="org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction">
<property name="hashIdResolver" >
<bean class="org.apache.ignite.cache.affinity.AffinityNodeAddressHashResolver"/>
</property>
<property name="partitions" value="280" />
</bean>
</property>
<property name="backups" value="0"/>
<property name="writeSynchronizationMode" value="PRIMARY_SYNC"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="atomicWriteOrderMode" value="PRIMARY"/>
<property name="memoryMode" value="ONHEAP_TIERED"/><!-- memory tire -->
<property name="startSize" value="#{30 * 1024 * 1024}"/>
<property name="offHeapMaxMemory" value="#{512 * 1024 * 1024}"/>
<property name="swapEnabled" value="true"/>
<property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxMemorySize" value="#{1024 * 1024 * 1024}"/>
</bean>
</property>
<property name="rebalanceMode" value="ASYNC"/>
<property name="rebalanceBatchSize" value="#{1024 * 1024}"/>
<property name="rebalanceThrottle" value="100"/>
</bean>
<bean id="igniA" class="org.apache.ignite.configuration.IgniteConfiguration" scope="singleton">
<property name="gridName" value="grid_recmd"/>
<property name="clientMode" value="false"/>
<property name="metricsLogFrequency" value="0"/><!-- 0 means disable metrics log -->
<property name="deploymentMode" value="ISOLATED"/>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="localPort" value="47500"/>
<property name="localPortRange" value="20"/>
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="shared" value="true"/>
<property name="addresses">
<list>
<value>10.0.0.1:47500..47520</value>
<value>10.0.0.2:47500..47520</value>
</list>
</property>
</bean>
</property>
<property name="joinTimeout" value="60000"/>
</bean>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="socketWriteTimeout" value="30000"/>
<property name="connectTimeout" value="50000"/>
</bean>
</property>
<property name="failureDetectionTimeout" value="60000"/>
<property name="swapSpaceSpi">
<bean class="org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi">
<property name="baseDirectory" value="/data1/home/igni1.5/my/swp"/>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<ref bean="cacheA"/>
</list>
</property>
<property name="includeEventTypes">
<list>
</list>
</property>
</bean>

Activemq Shutdown fails and then kills process

I am implementing replicated leveldb activemq setup. I have 3 instance of activemq running on same box. I am changing their rmiPort, amqpport and openwire port in config file.
config like lookslie this:
<?xml version="1.0" encoding="UTF-8"?><beans xmlns=" http://www.springframework.org /schema/beans" xmlns:amq=" http://activemq.apache.org/schema/core" xmlns:xsi=" http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="activemq_8200" dataDirectory="${activemq.data}">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry producerFlowControl="false" topic=">">
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see:
http://activemq.apache.org/slow-consumer-handling.html
-->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
<policyEntry producerFlowControl="false" queue=">">
<deadLetterStrategy>
<!--
Use the prifix 'DLQ.' for the destination name, and make the DLQ a queue rather than a topic
-->
<individualDeadLetterStrategy queuePrefix="DLQ." useQueueForQueueMessages="true"/>
</deadLetterStrategy>
<!-- Use VM cursor for better latency
For more information, see:
http://activemq.apache.org/message-cursors.html
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
-->
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceAdapter>
<replicatedLevelDB bind="tcp://0.0.0.0:0" directory="${activemq.data}/leveldb" replicas="3" zkAddress="gwxdev05.northamerica.cerner.net:2181,gwxdev05.northamerica.cerner.net:2182,gwxdev05.northamerica.cerner.net:2183" zkPassword="password" zkPath="/opt/gwx/activemqdata"/>
</persistenceAdapter>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="256 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="128 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:${openwirePort}?maximumConnections=1000&wireformat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:${amqpPort}?maximumConnections=1000&wireformat.maxFrameSize=104857600"/>
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook"/>
</shutdownHooks>
</broker>
<import resource="jetty.xml"/>
my instance file looks like this:
ACTIVEMQ_BASE=`cd "$ACTIVEMQ_BASE" && pwd`
## Add system properties for this instance here (if needed), e.g
#export ACTIVEMQ_OPTS_MEMORY="-Xms256M -Xmx1G"
#export ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS_MEMORY
-Dorg.apache.activemq.UseDedicatedTaskRunner=true
-Djava.util.logging.config.file=logging.properties"
export ACTIVEMQ_SUNJMX_CONTROL="-Dactivemq.jmx.url=service:jmx:rmi:///jndi/rmi://127.0.0.1:8100/jmxrmi"
#
ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=8100 "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.authenticate=false"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"
export ACTIVEMQ_SUNJMX_START=$ACTIVEMQ_SUNJMX_START
export ACTIVEMQ_HOME=/opt/gwx/apache-activemq-5.10-SNAPSHOT
export ACTIVEMQ_BASE=$ACTIVEMQ_BASE
export JAVA_HOME=/opt/gwx/apache-activemq-5.10-SNAPSHOT/jdk1.7.0_25
${ACTIVEMQ_HOME}/bin/activemq "$#"
Here is exception I get:
Connecting to pid: 2410
INFO: failed to resolve jmxUrl for pid:2410, using default JMX url Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
ERROR: java.lang.RuntimeException: Failed to execute stop task. Reason: java.io.IOException: Failed to retrieve RMIServer stub:
javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: java.net.ConnectException: Connection refused] java.lang.RuntimeException: Failed to execute stop task. Reason: java.io.IOException: Failed to retrieve RMIServer stub:
javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: java.net.ConnectException: Connection refused]
at
I checked firewall. It not issue.
Any idea what might be causing this issue.
Activemq version 5.10 snapshot
Java 1.7
OS linux 6.4
If it still helps someone:
When I had in bin/env this
export ACTIVEMQ_SUNJMX_CONTROL="-Dactivemq.jmx.url=service:jmx:rmi:///jndi/rmi://127.0.0.1:8100/jmxrmi"
#
ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=8100 "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.authenticate=false"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"
, it did not work, process failed even without logging.
I don't know why, but this method works(for AMQ 5.13): u add to xml this:
<managementContext>
<managementContext connectorPort="1099"/>
</managementContext>
and nothing like ACTIVEMQ_SUNJMX_START in env file

ActiveMQ as local JNDI tomcat ressource

i'm trying to set up ActiveMQ as Tomcat ressource with local JNDI. But when i add the config-file to the
Broker URI "brokerConfig=xbean:activemq.xml" the broker isn't starting up without any error message.
it just keeps telling me:
Mrz 30, 2012 10:23:19 AM org.springframework.jms.listener.DefaultMessageListenerContainer refreshConnectionUntilSuccessful
Warnung: Could not refresh JMS Connection for destination 'FOO.QUEUE' - retrying in 5000 ms. Cause: Could not create Transport. Reason: java.io.IOException: Could not load xbean factory:java.lang.NoClassDefFoundError: Could not initialize class org.apache.activemq.xbean.XBeanBrokerFactory
I used the default config from http://svn.apache.org/repos/asf/activemq/trunk/assembly/src/release/conf/activemq.xml and is placed in the root of my src folder.
i'm using "activemq-all_5.4.3.jar"
My web.xml in "WebContent\META-INF"
<resource-ref>
<description>JMS Connection</description>
<res-ref-name>jms/ConnectionFactory</res-ref-name>
<res-type>org.apache.activemq.ActiveMQConnectionFactory</res-type>
<res-auth>Container</res-auth>
<res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
<resource-ref>
<res-ref-name>jms/FooQueue</res-ref-name>
<res-type>javax.jms.Queue</res-type>
<res-auth>Container</res-auth>
</resource-ref>
My applicationContext.xml in "WebContent\WEB-INF"
<jee:jndi-lookup id="fooQueue"
jndi-name="java:comp/env/jms/FooQueue"
cache="true"
resource-ref="true"
lookup-on-startup="true"
expected-type="org.apache.activemq.command.ActiveMQQueue"
proxy-interface="javax.jms.Queue" />
<bean id="singleConnectionFactory"
class="org.springframework.jms.connection.SingleConnectionFactory"
p:targetConnectionFactory-ref="connectionFactory"/>
<bean id="jmsTemplate"
class="org.springframework.jms.core.JmsTemplate"
p:connectionFactory-ref="singleConnectionFactory"
p:defaultDestination-ref="fooQueue"/>
<bean id="messageSenderService"
class="by2.server.JmsMessageSenderService"
p:jmsTemplate-ref="jmsTemplate" />
<bean id="jmsMessageDelegate"
class="by2.server.JmsMessageDelegate" />
<bean id="myMessageListener"
class="org.springframework.jms.listener.adapter.MessageListenerAdapter"
p:delegate-ref="jmsMessageDelegate"
p:defaultListenerMethod="handleMessage">
</bean>
<jms:listener-container
container-type="default"
connection-factory="singleConnectionFactory"
acknowledge="auto">
<jms:listener destination="FOO.QUEUE" ref="myMessageListener" />
</jms:listener-container>
My context.xml in "WebContent\META-INF"
<Context reloadable="true">
<Resource auth="Container" name="jms/ConnectionFactory"
type="org.apache.activemq.ActiveMQConnectionFactory" description="JMS Connection Factory"
factory="org.apache.activemq.jndi.JNDIReferenceFactory" brokerURL="vm://localhost?brokerConfig=xbean:activemq.xml"
brokerName="FooBroker" />
<Resource auth="Container" name="jms/FooQueue"
type="org.apache.activemq.command.ActiveMQQueue" description="JMS queue"
factory="org.apache.activemq.jndi.JNDIReferenceFactory" physicalName="FOO.QUEUE" />
</Context>
For me it looks like a classpath error.
Did you have the xbean-spring-x.x.jar in your class path?
If not copy this file also from activemq distribution and put it in your app server classpath.