Related
I am running my service in local environment and trying to connect to remote node but it showing error failed to send message to remote node.
I want to run my service in local environment and connect it to remote ignite node on different server.
my configuration is:
IgniteConfiguration igniteConfig = new IgniteConfiguration();
igniteConfig.setIgniteInstanceName("MasterCacheCluster");
igniteConfig.setPeerClassLoadingEnabled(true);
igniteConfig.setClientMode(true);
TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new
TcpDiscoveryVmIpFinder();
TcpCommunicationSpi communicationSpi = new
TcpCommunicationSpi();
ipFinder.setAddresses(Arrays.asList("server_address":47500..47509"));
discoverySpi.setIpFinder(ipFinder);
igniteConfig.setDiscoverySpi(discoverySpi);
DataStorageConfiguration dataCfg = new
DataStorageConfiguration();
DataRegionConfiguration rgnCfg = new
DataRegionConfiguration();
rgnCfg.setName("Sample_Cluster_Region");
rgnCfg.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
rgnCfg.setPersistenceEnabled(true);
rgnCfg.setMetricsEnabled(true);
dataCfg.setDataRegionConfigurations(rgnCfg);
Ignite ignite = Ignition.start(igniteConfig);
ignite.cluster().active(true);
System.out.println("Cluster Size: " +
ignite.cluster().nodes().size());
return ignite;
** server address is hidden due to privacy reason
[13:12:18,839][SEVERE][exchange-worker-#62%MasterCacheCluster%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=724fff2c-76c2-44e7-921f-b7c37dac7d15, consistentId=7c4ed309-0b9b-40ba-84a1-90384e0940ea, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.3.0.8, 127.0.0.1], sockAddrs=null, discPort=47500, order=1, intOrder=1, lastExchangeTime=1676878928401, loc=false, ver=2.14.0#20220929-sha1:951e8deb, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null, partsSizes=null, partHistCntrs=null, err=null, client=true, exchangeStartTime=1676878928573, finishMsg=null, super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=96f70bd7-cbfb-4a3e-900d-00a93b10d892, consistentId=96f70bd7-cbfb-4a3e-900d-00a93b10d892, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 172.16.16.50], sockAddrs=HashSet [/[0:0:0:0:0:0:0:1]:0, /127.0.0.1:0, LAPTOP-6AUCFF2I/172.16.16.50:0], discPort=0, order=2, intOrder=0, lastExchangeTime=1676878923997, loc=true, ver=2.14.0#20220929-sha1:951e8deb, isClient=true], topVer=2, msgTemplate=null, span=org.apache.ignite.internal.processors.tracing.NoopSpan#baed14f, nodeId8=96f70bd7, msg=null, type=NODE_JOINED, tstamp=1676878928556], nodeId=96f70bd7, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, order=1676878923547, nodeOrder=0, dataCenterId=0], super=GridCacheMessage [msgId=1, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], err=null, skipPrepare=false]]]]]
class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=724fff2c-76c2-44e7-921f-b7c37dac7d15, addrs=[/10.3.0.8:47100, /[0:0:0:0:0:0:0:1%lo]:47100, /127.0.0.1:47100]]
Your client tries to establish communication link to the server node with id=724fff2c-76c2-44e7-921f-b7c37dac7d15 after receiving it's address through discovery protocol. This exception basically implies that there's no connectivity between your local host and "server_address":47100. Every single node (including clients) should be visible to the rest of a cluster. My guess is you have some firewall rules or something like that.
Try running some tools to troubleshoot, you could start with.
nc -vz "server_address" 47100
It should be run from your laptop.
It's also worth mentioning that your server expose ipv6 addresses. It's recommended to use ipv4 at the moment. Add -Djava.net.preferIPv4Stack=true JVM param to the both client and server JVM start scripts.
Ignite version: 2.14.0
Node configuration: 2 Nodes running on same PC (IPV4) using localhost and 255 available ports:
TcpDiscoveryMulticastIpFinder ipFinder = new TcpDiscoveryMulticastIpFinder();
ipFinder.setAddresses(Collections.singletonList("127.0.0.1"));
Also 2 different working dirs, Threadpool 16, 2 caches (one atomic, one transactional)
What happens: Using ExecutorService i submit 8 threads to pool. Class run correctly (4 on each node) and execute tasks as expected.
But during execution raise, repeatedly and with some frequency, the following exception on both nodes: GRAVE: "Failed to process selector key".
The application generates a high computational load. A simple "for loop" with a sleep gives no error
Full stack follows:
GRAVE: Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=3, bytesRcvd=97567668, bytesSent=100128669, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-3, igniteInstanceName=TcpCommunicationSpi, finished=false, heartbeatTs=1675265761563, hashCode=2143442267, interrupted=false, runner=grid-nio-worker-tcp-comm-3-#26%TcpCommunicationSpi%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=1690656, resendCnt=0, rcvCnt=1696452, sentCnt=1691375, reserved=true, lastAck=1696448, nodeLeft=false, node=TcpDiscoveryNode [id=cd1ffdf0-b9b3-49ef-a9e3-db1676fad428, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.178.30,192.168.56.1:47500, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.178.30, 192.168.56.1], sockAddrs=HashSet [host.docker.internal/192.168.178.30:47500, /0:0:0:0:0:0:0:1:47500, WOPR/192.168.56.1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1675265584899, loc=false, ver=2.14.0#20220929-sha1:951e8deb, isClient=false], connected=true, connectCnt=69, queueLimit=4096, reserveCnt=101, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=1690656, resendCnt=0, rcvCnt=1696452, sentCnt=1691375, reserved=true, lastAck=1696448, nodeLeft=false, node=TcpDiscoveryNode [id=cd1ffdf0-b9b3-49ef-a9e3-db1676fad428, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.178.30,192.168.56.1:47500, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.178.30, 192.168.56.1], sockAddrs=HashSet [host.docker.internal/192.168.178.30:47500, /0:0:0:0:0:0:0:1:47500, WOPR/192.168.56.1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1675265584899, loc=false, ver=2.14.0#20220929-sha1:951e8deb, isClient=false], connected=true, connectCnt=69, queueLimit=4096, reserveCnt=101, pairedConnections=false], closeSocket=true, outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric#69a257d1, super=GridNioSessionImpl [locAddr=/0:0:0:0:0:0:0:1:47101, rmtAddr=/0:0:0:0:0:0:0:1:56361, createTime=1675265760336, closeTime=0, bytesSent=8479762, bytesRcvd=7459908, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1675265760336, lastSndTime=1675265761545, lastRcvTime=1675265761563, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser#b329ba4, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=true]]]
java.io.IOException: Connessione in corso interrotta forzatamente dall'host remoto
at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51)
at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:58)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite0(GridNioServer.java:1715)
at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1407)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2511)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at java.base/java.lang.Thread.run(Thread.java:834)
Expected: I read that it could be a configuration problem but I don't understand how to fix it.
The configuration seems simple but and even if the execution is without calculation errors i would like to avoid this exception.
NODE1
[2023-02-02 15:54:30] [AVVERTENZA] Client disconnected abruptly due to network connection loss or because the connection was left open on application shutdown. [cls=class o.a.i.i.util.nio.GridNioException, msg=Connessione in corso interrotta forzatamente dall'host remoto] - [org.apache.ignite.logger.java.JavaLogger warning:]
[2023-02-02 15:54:30] [AVVERTENZA] Unacknowledged messages queue size overflow, will attempt to reconnect [remoteAddr=/127.0.0.1:63660, queueLimit=4096] - [org.apache.ignite.logger.java.JavaLogger warning:]
[2023-02-02 15:54:30] [INFORMAZIONI] Accepted incoming communication connection [locAddr=/127.0.0.1:47101, rmtAddr=/127.0.0.1:63670] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:30] [INFORMAZIONI] Accepted incoming communication connection [locAddr=/127.0.0.1:47101, rmtAddr=/127.0.0.1:63671] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:30] [INFORMAZIONI] Received incoming connection when already connected to this node, rejecting [locNode=af74d5c9-3631-4fdf-b9f2-0babc853019f, rmtNode=8a378874-f3ae-4d0c-9733-a6b143097658] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:30] [INFORMAZIONI] Accepted incoming communication connection [locAddr=/127.0.0.1:47101, rmtAddr=/127.0.0.1:63672] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:30] [INFORMAZIONI] Received incoming connection when already connected to this node, rejecting [locNode=af74d5c9-3631-4fdf-b9f2-0babc853019f, rmtNode=8a378874-f3ae-4d0c-9733-a6b143097658] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:31] [INFORMAZIONI] Accepted incoming communication connection [locAddr=/127.0.0.1:47101, rmtAddr=/127.0.0.1:63673] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:31] [INFORMAZIONI] Received incoming connection when already connected to this node, rejecting [locNode=af74d5c9-3631-4fdf-b9f2-0babc853019f, rmtNode=8a378874-f3ae-4d0c-9733-a6b143097658] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:31] [GRAVE ] Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=2269317, bytesSent=3928093, bytesRcvd0=1909138, bytesSent0=720914, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=TcpCommunicationSpi, finished=false, heartbeatTs=1675349670621, hashCode=722948156, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#23%TcpCommunicationSpi%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=17152, resendCnt=870, rcvCnt=31061, sentCnt=18796, reserved=true, lastAck=31040, nodeLeft=false, node=TcpDiscoveryNode [id=8a378874-f3ae-4d0c-9733-a6b143097658, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.178.30,192.168.56.1:47500, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.178.30, 192.168.56.1], sockAddrs=HashSet [host.docker.internal/192.168.178.30:47500, /0:0:0:0:0:0:0:1:47500, WOPR/192.168.56.1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1675349650217, loc=false, ver=2.14.0#20220929-sha1:951e8deb, isClient=false], connected=true, connectCnt=7, queueLimit=4096, reserveCnt=9, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=17152, resendCnt=870, rcvCnt=31061, sentCnt=18796, reserved=true, lastAck=31040, nodeLeft=false, node=TcpDiscoveryNode [id=8a378874-f3ae-4d0c-9733-a6b143097658, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.178.30,192.168.56.1:47500, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.178.30, 192.168.56.1], sockAddrs=HashSet [host.docker.internal/192.168.178.30:47500, /0:0:0:0:0:0:0:1:47500, WOPR/192.168.56.1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1675349650217, loc=false, ver=2.14.0#20220929-sha1:951e8deb, isClient=false], connected=true, connectCnt=7, queueLimit=4096, reserveCnt=9, pairedConnections=false], closeSocket=true, outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric#69a257d1, super=GridNioSessionImpl [locAddr=/127.0.0.1:47101, rmtAddr=/127.0.0.1:63670, createTime=1675349670241, closeTime=0, bytesSent=720914, bytesRcvd=1909138, bytesSent0=720914, bytesRcvd0=1909138, sndSchedTime=1675349670241, lastSndTime=1675349670277, lastRcvTime=1675349670621, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser#16179752, directMode=true], GridConnectionBytesVerifyFilter], accepted=true, markedForClose=true]]] - [org.apache.ignite.logger.java.JavaLogger error:
java.io.IOException: Connessione in corso interrotta forzatamente dall'host remoto
NODE2
[2023-02-02 15:54:30] [INFORMAZIONI] Accepted incoming communication connection [locAddr=/127.0.0.1:47100, rmtAddr=/127.0.0.1:63669] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:30] [INFORMAZIONI] Received incoming connection from remote node while connecting to this node, rejecting [locNode=8a378874-f3ae-4d0c-9733-a6b143097658, locNodeOrder=1, rmtNode=af74d5c9-3631-4fdf-b9f2-0babc853019f, rmtNodeOrder=2] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:30] [INFORMAZIONI] Established outgoing communication connection [locAddr=/127.0.0.1:63670, rmtAddr=/127.0.0.1:47101] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:31] [INFORMAZIONI] Established outgoing communication connection [locAddr=/127.0.0.1:63676, rmtAddr=/127.0.0.1:47101] - [org.apache.ignite.logger.java.JavaLogger info:]
[2023-02-02 15:54:31] [INFORMAZIONI] TCP client created [client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1, bytesRcvd=84, bytesSent=56, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=TcpCommunicationSpi, finished=false, heartbeatTs=1675349671637, hashCode=762674116, interrupted=false, runner=grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%]]], writeBuf=java.nio.DirectByteBuffer[pos=9391 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=31061, resendCnt=753, rcvCnt=17160, sentCnt=31871, reserved=true, lastAck=17152, nodeLeft=false, node=TcpDiscoveryNode [id=af74d5c9-3631-4fdf-b9f2-0babc853019f, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.178.30,192.168.56.1:47501, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.178.30, 192.168.56.1], sockAddrs=HashSet [host.docker.internal/192.168.178.30:47501, /0:0:0:0:0:0:0:1:47501, WOPR/192.168.56.1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1675349650060, loc=false, ver=2.14.0#20220929-sha1:951e8deb, isClient=false], connected=false, connectCnt=8, queueLimit=4096, reserveCnt=9, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=31061, resendCnt=531, rcvCnt=17160, sentCnt=31871, reserved=true, lastAck=17152, nodeLeft=false, node=TcpDiscoveryNode [id=af74d5c9-3631-4fdf-b9f2-0babc853019f, consistentId=0:0:0:0:0:0:0:1,127.0.0.1,192.168.178.30,192.168.56.1:47501, addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.178.30, 192.168.56.1], sockAddrs=HashSet [host.docker.internal/192.168.178.30:47501, /0:0:0:0:0:0:0:1:47501, WOPR/192.168.56.1:47501, /127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1675349650060, loc=false, ver=2.14.0#20220929-sha1:951e8deb, isClient=false], connected=false, connectCnt=8, queueLimit=4096, reserveCnt=9, pairedConnections=false], closeSocket=true, outboundMessagesQueueSizeMetric=org.apache.ignite.internal.processors.metric.impl.LongAdderMetric#69a257d1, super=GridNioSessionImpl [locAddr=/127.0.0.1:63676, rmtAddr=/127.0.0.1:47101, createTime=1675349671637, closeTime=0, bytesSent=0, bytesRcvd=0, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1675349671637, lastSndTime=1675349671637, lastRcvTime=1675349671637, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser#544beb47, directMode=true], GridConnectionBytesVerifyFilter], accepted=false, markedForClose=false]], super=GridAbstractCommunicationClient [lastUsed=1675349671637, closed=false, connIdx=0]], duration=339ms] - [org.apache.ignite.logger.java.JavaLogger info:]
I configure the static IP:
TcpDiscoverySpi spi = new TcpDiscoverySpi();`TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();ipFinder.setAddresses(Arrays.asList("76.3.16.109", "76.3.16.110","76.3.16.111", "76.3.16.112", "76.3.16.113"));`
ignite log:
Failed to send message [node=TcpDiscoveryNode [id=2402793f-f484-4f3a-9213-82beeebfd09a, consistentId=76.3.16.110:23054, addrs=ArrayList [76.3.16.110], sockAddrs=HashSet [fl-76-3-16-110.dhcp.embarqhsd.net/76.3.16.110:23054], discPort=23054, order=15, intOrder=9, lastExchangeTime=1631517404103, loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=false], msg=GridQueryCancelRequest [qryReqId=3560], errMsg=Failed to send message (node left topology): TcpDiscoveryNode [id=2402793f-f484-4f3a-9213-82beeebfd09a, consistentId=76.3.16.110:23054, addrs=ArrayList [76.3.16.110], sockAddrs=HashSet [fl-76-3-16-110.dhcp.embarqhsd.net/76.3.16.110:23054], discPort=23054, order=15, intOrder=9, lastExchangeTime=1631517404103, loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=false]]
/etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.2.144.62 tools.cmc.rnd.huawei.com
/etc/networks
default 0.0.0.0 loopback 127.0.0.0 link-local 169.254.0.0
/etc/hostname
EulerOS
I don't know which configuration has a problem
There are no similar problems in other environments
Please look at it for me, thank you
[2021-10-28 19:42:20,560][WARN ][0][0][tcp-disco-sock-reader-[d5c103a9 115.0.77.41:39585]-#4-#800][][IgniteLoggerImp][74] Failed to shutdown socket: closing inbound before receiving peer's close_notify javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
at sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:735)
at sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:714)
at org.apache.ignite.internal.util.IgniteUtils.close(IgniteUtils.java:4232)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:7382)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
[2021-10-28 19:42:25,205][WARN ][0][0][jvm-pause-detector-worker][][IgniteLoggerImp][72] Possible too long JVM pause: 4503 milliseconds.
[2021-10-28 19:42:47,285][WARN ][0][0][fm.monitor.rebuild-105-1][ROOT][IgniteLoggerImp][72] Query produced big result set. [fetched=100000, duration=1250ms, type=MAP, distributedJoin=false, enforceJoinOrder=false, lazy=false, schema=alarmCache, sql='SELECT\n__Z0.MERGED __C0_0,\n__Z0.SPECIALALARMSTATUS __C0_1,\n__Z0.NATIVEMODN __C0_2,\n__Z0.SEVERITY __C0_3,\n__Z0.ACKED __C0_4,\n__Z0.CLEARED __C0_5,\n__Z0.MEDN __C0_6,\n__Z0.CSN __C0_7\nFROM "alarmCache".ALARMRECORD __Z0', plan=SELECT\n __Z0.MERGED AS __C0_0,\n __Z0.SPECIALALARMSTATUS AS __C0_1,\n __Z0.NATIVEMODN AS __C0_2,\n __Z0.SEVERITY AS __C0_3,\n __Z0.ACKED AS __C0_4,\n __Z0.CLEARED AS __C0_5,\n __Z0.MEDN AS __C0_6,\n __Z0.CSN AS __C0_7\nFROM "alarmCache".ALARMRECORD __Z0\n /* "alarmCache".ALARMRECORD._SCAN /\n / scanCount: 140023 /, node=TcpDiscoveryNode [id=4b9a0352-7d68-4a51-8f31-b4ae2405ee7c, consistentId=115.0.77.40:23054, addrs=ArrayList [115.0.77.40], sockAddrs=HashSet [EulerOS/115.0.77.40:23054], discPort=23054, order=1, intOrder=1, lastExchangeTime=1635415317696, loc=true, ver=2.11.0#20210911-sha1:8f3f07d3, isClient=false], reqId=35, segment=1]
[2021-10-28 19:42:47,287][WARN ][0][0][fm.monitor.rebuild-105-1][ROOT][IgniteLoggerImp][72] Query produced big result set. [fetched=100000, duration=1260ms, type=MAP, distributedJoin=false, enforceJoinOrder=false, lazy=false, schema=alarmCache, sql='SELECT\n__Z0.MERGED __C0_0,\n__Z0.SPECIALALARMSTATUS __C0_1,\n__Z0.NATIVEMODN __C0_2,\n__Z0.SEVERITY __C0_3,\n__Z0.ACKED __C0_4,\n__Z0.CLEARED __C0_5,\n__Z0.MEDN __C0_6,\n__Z0.CSN __C0_7\nFROM "alarmCache".ALARMRECORD __Z0', plan=SELECT\n __Z0.MERGED AS __C0_0,\n __Z0.SPECIALALARMSTATUS AS __C0_1,\n __Z0.NATIVEMODN AS __C0_2,\n __Z0.SEVERITY AS __C0_3,\n __Z0.ACKED AS __C0_4,\n __Z0.CLEARED AS __C0_5,\n __Z0.MEDN AS __C0_6,\n __Z0.CSN AS __C0_7\nFROM "alarmCache".ALARMRECORD __Z0\n / "alarmCache".ALARMRECORD._SCAN /\n / scanCount: 140688 /, node=TcpDiscoveryNode [id=4b9a0352-7d68-4a51-8f31-b4ae2405ee7c, consistentId=115.0.77.40:23054, addrs=ArrayList [115.0.77.40], sockAddrs=HashSet [EulerOS/115.0.77.40:23054], discPort=23054, order=1, intOrder=1, lastExchangeTime=1635415317696, loc=true, ver=2.11.0#20210911-sha1:8f3f07d3, isClient=false], reqId=35, segment=0]
[2021-10-28 19:42:47,453][WARN ][0][0][fm.monitor.rebuild-105-1][ROOT][IgniteLoggerImp][72] Query produced big result set. [fetched=140022, duration=1422ms, type=MAP, distributedJoin=false, enforceJoinOrder=false, lazy=false, schema=alarmCache, sql='SELECT\n__Z0.MERGED __C0_0,\n__Z0.SPECIALALARMSTATUS __C0_1,\n__Z0.NATIVEMODN __C0_2,\n__Z0.SEVERITY __C0_3,\n__Z0.ACKED __C0_4,\n__Z0.CLEARED __C0_5,\n__Z0.MEDN __C0_6,\n__Z0.CSN __C0_7\nFROM "alarmCache".ALARMRECORD __Z0', plan=SELECT\n __Z0.MERGED AS __C0_0,\n __Z0.SPECIALALARMSTATUS AS __C0_1,\n __Z0.NATIVEMODN AS __C0_2,\n __Z0.SEVERITY AS __C0_3,\n __Z0.ACKED AS __C0_4,\n __Z0.CLEARED AS __C0_5,\n __Z0.MEDN AS __C0_6,\n __Z0.CSN AS __C0_7\nFROM "alarmCache".ALARMRECORD __Z0\n / "alarmCache".ALARMRECORD._SCAN /\n / scanCount: 140023 /, node=TcpDiscoveryNode [id=4b9a0352-7d68-4a51-8f31-b4ae2405ee7c, consistentId=115.0.77.40:23054, addrs=ArrayList [115.0.77.40], sockAddrs=HashSet [EulerOS/115.0.77.40:23054], discPort=23054, order=1, intOrder=1, lastExchangeTime=1635415317696, loc=true, ver=2.11.0#20210911-sha1:8f3f07d3, isClient=false], reqId=35, segment=1]
[2021-10-28 19:42:47,461][WARN ][0][0][fm.monitor.rebuild-105-1][ROOT][IgniteLoggerImp][72] Query produced big result set. [fetched=140687, duration=1432ms, type=MAP, distributedJoin=false, enforceJoinOrder=false, lazy=false, schema=alarmCache, sql='SELECT\n__Z0.MERGED __C0_0,\n__Z0.SPECIALALARMSTATUS __C0_1,\n__Z0.NATIVEMODN __C0_2,\n__Z0.SEVERITY __C0_3,\n__Z0.ACKED __C0_4,\n__Z0.CLEARED __C0_5,\n__Z0.MEDN __C0_6,\n__Z0.CSN __C0_7\nFROM "alarmCache".ALARMRECORD __Z0', plan=SELECT\n __Z0.MERGED AS __C0_0,\n __Z0.SPECIALALARMSTATUS AS __C0_1,\n __Z0.NATIVEMODN AS __C0_2,\n __Z0.SEVERITY AS __C0_3,\n __Z0.ACKED AS __C0_4,\n __Z0.CLEARED AS __C0_5,\n __Z0.MEDN AS __C0_6,\n __Z0.CSN AS __C0_7\nFROM "alarmCache".ALARMRECORD __Z0\n / "alarmCache".ALARMRECORD._SCAN /\n / scanCount: 140688 */, node=TcpDiscoveryNode [id=4b9a0352-7d68-4a51-8f31-b4ae2405ee7c, consistentId=115.0.77.40:23054, addrs=ArrayList [115.0.77.40], sockAddrs=HashSet [EulerOS/115.0.77.40:23054], discPort=23054, order=1, intOrder=1, lastExchangeTime=1635415317696, loc=true, ver=2.11.0#20210911-sha1:8f3f07d3, isClient=false], reqId=35, segment=0]
[2021-10-28 19:43:04,759][WARN ][0][0][jvm-pause-detector-worker][ROOT][IgniteLoggerImp][72] Possible too long JVM pause: 12688 milliseconds.
[2021-10-28 19:43:04,775][WARN ][0][0][tcp-disco-msg-worker-[crd]-#2-#55][][IgniteLoggerImp][72] Failed to send message to next node [msg=TcpDiscoveryNodeAddedMessage [node=TcpDiscoveryNode [id=d5c103a9-4b8a-4430-bab1-fa63ad8066e7, consistentId=115.0.77.41:23054, addrs=ArrayList [115.0.77.41], sockAddrs=HashSet [EulerOS/115.0.77.40:23054, 115.0.77.41/115.0.77.41:23054], discPort=23054, order=0, intOrder=2, lastExchangeTime=1635421284474, loc=false, ver=2.11.0#20210911-sha1:8f3f07d3, isClient=false], dataPacket=o.a.i.spi.discovery.tcp.internal.DiscoveryDataPacket#492d48ec, discardMsgId=null, discardCustomMsgId=null, top=null, clientTop=null, gridStartTime=1635415317736, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=9d88656cc71-4b9a0352-7d68-4a51-8f31-b4ae2405ee7c, verifierNodeId=4b9a0352-7d68-4a51-8f31-b4ae2405ee7c, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=d5c103a9-4b8a-4430-bab1-fa63ad8066e7, consistentId=115.0.77.41:23054, addrs=ArrayList [115.0.77.41], sockAddrs=HashSet [EulerOS/115.0.77.40:23054, 115.0.77.41/115.0.77.41:23054], discPort=23054, order=0, intOrder=2, lastExchangeTime=1635421284474, loc=false, ver=2.11.0#20210911-sha1:8f3f07d3, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryNodeAddedMessage [node=TcpDiscoveryNode [id=d5c103a9-4b8a-4430-bab1-fa63ad8066e7, consistentId=115.0.77.41:23054, addrs=ArrayList [115.0.77.41], sockAddrs=HashSet [EulerOS/115.0.77.40:23054, 115.0.77.41/115.0.77.41:23054], discPort=23054, order=0, intOrder=2, lastExchangeTime=1635421284474, loc=false, ver=2.11.0#20210911-sha1:8f3f07d3, isClient=false], dataPacket=o.a.i.spi.discovery.tcp.internal.DiscoveryDataPacket#492d48ec, discardMsgId=null, discardCustomMsgId=null, top=null, clientTop=null, gridStartTime=1635415317736, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=9d88656cc71-4b9a0352-7d68-4a51-8f31-b4ae2405ee7c, verifierNodeId=4b9a0352-7d68-4a51-8f31-b4ae2405ee7c, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=d5c103a9-4b8a-4430-bab1-fa63ad8066e7, order=0, addr=[115.0.77.41], daemon=false]]]
[2021-10-28 19:43:04,777][WARN ][0][0][tcp-disco-msg-worker-[crd]-#2-#55][ROOT][IgniteLoggerImp][72] Local node has detected failed nodes and started cluster-wide procedure. To speed up failure detection please see 'Failure Detection' section under javadoc for 'TcpDiscoverySpi'
[2021-10-28 19:43:04,814][WARN ][0][0][disco-event-worker-#57][][IgniteLoggerImp][72] Node FAILED: TcpDiscoveryNode [id=d5c103a9-4b8a-4430-bab1-fa63ad8066e7, consistentId=115.0.77.41:23054, addrs=ArrayList [115.0.77.41], sockAddrs=HashSet [EulerOS/115.0.77.40:23054, 115.0.77.41/115.0.77.41:23054], discPort=23054, order=2, intOrder=2, lastExchangeTime=1635421284474, loc=false, ver=2.11.0#20210911-sha1:8f3f07d3, isClient=false]
[2021-10-28 19:42:20,907][DEBUG][0][0][tcp-disco-msg-worker-[]-#2-#35][ROOT]
[IgniteLoggerImp][51] Message has been added to a worker's queue: TcpDiscoveryStatusCheckMessage [creatorNode=null, failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=04671b6cc71-d5c103a9-4b8a-4430-bab1-fa63ad8066e7, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]]
[2021-10-28 19:42:20,907][DEBUG][0][0][tcp-disco-msg-worker-[]-#2-#35][ROOT]
[IgniteLoggerImp][51] Processing message [cls=TcpDiscoveryStatusCheckMessage, id=04671b6cc71-d5c103a9-4b8a-4430-bab1-fa63ad8066e7]
[2021-10-28 19:42:20,907][DEBUG][0][0][tcp-disco-msg-worker-[]-#2-#35][ROOT]
[IgniteLoggerImp][51] Ignore message, local node order is not initialized
[msg=TcpDiscoveryStatusCheckMessage [creatorNode=null, failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=04671b6cc71-d5c103a9-4b8a-4430-bab1-fa63ad8066e7, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]],
locNode=Tcp
We upgraded a bunch of libraries in our application one of them is ignite. Right now the ignite running in client mode is crashing. My thinking is that one of the upgrades caused the cache to have increased in size. (so I don't think the upgrade of ignite is the problem).
So I increased the heap size from 10 to 20 GB. But when about 50% is used the JVM hangs.
I'm confused on why it does this when there is only 50% in use.
[12/3/20 16:07:58:788 GMT] 000000c4 IgniteKernal I .... Heap [used=9937MB, free=51.48%, comm=10680MB]
followed by
[12/3/20 16:08:26:410 GMT] 000000bd IgniteKernal W Possible too long JVM pause: 2418 milliseconds.
[12/3/20 16:08:27:465 GMT] 000000c5 TcpCommunicat W Client disconnected abruptly due to network connection loss or because the connection was left open on application shutdown. [cls=class o.a.i.i.util.nio.GridNioException, msg=Connection reset by peer]
[12/3/20 16:08:27:411 GMT] 000000c5 TcpCommunicat E Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=48849402273, bytesSent=15994664546, bytesRcvd0=54446, bytesSent0=102, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=null, finished=false, heartbeatTs=1607011706410, hashCode=433635054, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#51]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=9025120, resendCnt=0, rcvCnt=9025150, sentCnt=9025152, reserved=true, lastAck=9025120, nodeLeft=false, node=TcpDiscoveryNode [id=b3ca311e-077f-42a5-884a-807b539730b6, consistentId=10.60.46.12:48500, addrs=ArrayList [10.60.46.12], sockAddrs=HashSet [hex-wgc-p-web02/10.60.46.12:48500], discPort=48500, order=1, intOrder=1, lastExchangeTime=1607006097079, loc=false, ver=2.9.0#20201015-sha1:70742da8, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=9025120, resendCnt=0, rcvCnt=9025150, sentCnt=9025152, reserved=true, lastAck=9025120, nodeLeft=false, node=TcpDiscoveryNode [id=b3ca311e-077f-42a5-884a-807b539730b6, consistentId=10.60.46.12:48500, addrs=ArrayList [10.60.46.12], sockAddrs=HashSet [hex-wgc-p-web02/10.60.46.12:48500], discPort=48500, order=1, intOrder=1, lastExchangeTime=1607006097079, loc=false, ver=2.9.0#20201015-sha1:70742da8, isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], closeSocket=true, outboundMessagesQueueSizeMetric=o.a.i.i.processors.metric.impl.LongAdderMetric#69a257d1, super=GridNioSessionImpl [locAddr=/10.223.132.3:52550, rmtAddr=/10.60.46.12:48100, createTime=1607006097572, closeTime=0, bytesSent=15994657850, bytesRcvd=48849402273, bytesSent0=102, bytesRcvd0=54446, sndSchedTime=1607006097572, lastSndTime=1607011706410, lastRcvTime=1607011706410, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser#93200255, directMode=true], GridConnectionBytesVerifyFilter], accepted=false, markedForClose=false]]]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:235)
at sun.nio.ch.IOUtil.read(IOUtil.java:204)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:394)
at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1330)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2472)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2239)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1880)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:822)
[12/3/20 16:08:44:437 GMT] 000000c4 SystemOut O [16:08:44] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=null, finished=false, heartbeatTs=1607011706420]]]
[12/3/20 16:08:44:436 GMT] 000000c4 W java.util.logging.LogManager$RootLogger log Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=null, finished=false, heartbeatTs=1607011706420]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=null, finished=false, heartbeatTs=1607011706420]
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1806)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$3.apply(IgnitionEx.java:1801)
at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:234)
at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:822)
[12/3/20 16:08:44:434 GMT] 000000c4 G W Thread [name="tcp-comm-worker-#1-#63", id=211, state=WAITING, blockCnt=2, waitCnt=100]
[12/3/20 16:08:44:432 GMT] 000000c4 G E Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [workerName=tcp-comm-worker, threadName=tcp-comm-worker-#1-#63, blockedFor=18s]
[12/3/20 16:09:14:486 GMT] 000000c4 SystemOut O [16:09:14] Possible failure suppressed accordingly to a configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-comm-worker, igniteInstanceName=null, finished=false, heartbeatTs=1607011736000]]]
These look like network issues.
[12/3/20 16:08:27:411 GMT] 000000c5 TcpCommunicat E Failed to process selector key
[workerName=tcp-comm-worker, threadName=tcp-comm-worker-#1-#63, blockedFor=18s]
Check that you are able to connect from your client machines to your server machines and that firewall configs are properly set up.
see: https://ignite.apache.org/docs/latest/clustering/network-configuration
make sure you've set: Djava.net.preferIPv4Stack=true if you are using IP v4 addresses.
If there are containers and/or private addresses involved, it might cause connection issues.
See: https://ignite.apache.org/docs/latest/clustering/running-client-nodes-behind-nat#limitations
When using a near cache everything will work fine until a second client (could be visor) attempts to connect or disconnect to the cluster while a cache operation is in process.
After the second client connects/disconnects, the original client will always miss the near cache until and the original client restarts. Almost as if the cluster informs the client their are issues and to keep the cluster as the source of truth.
We have been able to reproduce this by running our test and connecting/disconnect with visor. During a disconnect we can see a Timeout mentioned in the logs on the original client IgniteTxManager$NodeFailureTimeoutObject.
Below is a snippet of the logs with the org.apache.ignite.internal.processors suppressed.
[2017-10-09 14:26:52.148] boot - 9955 DEBUG [http-nio-8081-exec-8] --- CacheHelper: Total time accessing cache ng-security-service-ORG_SPEC_CACHE for key * | value com.cache.model.PrefixCluster#78475a88: 0 millis
[2017-10-09 14:26:52.150] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridDiscoveryManager: Daemon node left topology: TcpDiscoveryNode [id=4cc6c321-d9cc-4149-a6ef-cba68877a269, addrs=[10.70.255.8, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17
.0.1:0, /127.0.0.1:0, /10.70.255.8:0], discPort=0, order=57, intOrder=31, lastExchangeTime=1507577126368, loc=false, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=true]
[2017-10-09 14:26:52.150] boot - 9955 DEBUG [http-nio-8081-exec-8] --- OrgSpecCacheImpl: OrgSpec Cache Stats: OrgSpec ObjId: IgniteCacheProxy [delegate=GridNearCacheAdapter [], opCtx=null, restartFut=null] HitCount: 120, MissCount: 50, AvgReadTime:
120, Eviction Count: 0
[2017-10-09 14:26:52.150] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridDeploymentPerVersionStore: Processing node departure event: DiscoveryEvent [evtNode=TcpDiscoveryNode [id=4cc6c321-d9cc-4149-a6ef-cba68877a269, addrs=[10.70.255.8, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0, /127.0.0.1:0, /10.70.255.8:0], discPort=0, order=57, intOrder=31, lastExchangeTime=1507577126368, loc=false, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=true], topVer=58, nodeId8=2e573c60, msg=Node left: TcpDiscoveryNode [id=4cc6c321-d9cc-4149-a6ef-cba68877a269, addrs=[10.70.255.8, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0, /127.0.0.1:0, /10.70.255.8:0], discPort=0, order=57, intOrder=31, lastExchangeTime=1507577126368, loc=false, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=true], type=NODE_LEFT, tstamp=1507577212142]
[2017-10-09 14:26:52.163] boot - 9955 INFO [exchange-worker-#27%null%] --- time: Started exchange init [topVer=AffinityTopologyVersion [topVer=58, minorTopVer=0], crd=false, evt=11, node=TcpDiscoveryNode [id=2e573c60-45f0-4429-a3fa-068489663148, addrs=[0:0:0:0:0:0:0:1%lo, 10.70.242.138, 127.0.0.1], sockAddrs=[port-svc-inc-13.tw-test.net/10.70.242.138:0, /0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0], discPort=0, order=56, intOrder=0, lastExchangeTime=1507576971754, loc=true, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=true], evtNode=TcpDiscoveryNode [id=2e573c60-45f0-4429-a3fa-068489663148, addrs=[0:0:0:0:0:0:0:1%lo, 10.70.242.138, 127.0.0.1], sockAddrs=[port-svc-inc-13.tw-test.net/10.70.242.138:0, /0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0], discPort=0, order=56, intOrder=0, lastExchangeTime=1507576971754, loc=true, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=true], customEvt=null]
[2017-10-09 14:26:52.164] boot - 9955 INFO [exchange-worker-#27%null%] --- GridDhtPartitionsExchangeFuture: Snapshot initialization completed [topVer=AffinityTopologyVersion [topVer=58, minorTopVer=0], time=0ms]
[2017-10-09 14:26:52.164] boot - 9955 INFO [exchange-worker-#27%null%] --- GridDhtPartitionsExchangeFuture: Snapshot initialization completed [topVer=AffinityTopologyVersion [topVer=58, minorTopVer=0], time=0ms]
[2017-10-09 14:26:52.164] boot - 9955 INFO [exchange-worker-#27%null%] --- time: Finished exchange init [topVer=AffinityTopologyVersion [topVer=58, minorTopVer=0], crd=false]
[2017-10-09 14:26:52.203] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridDeploymentLocalStore: Deployment meta for local deployment: GridDeploymentMetadata [depMode=SHARED, alias=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1, clsName=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1, userVer=null, sndNodeId=2e573c60-45f0-4429-a3fa-068489663148, clsLdrId=null, clsLdr=null, participants=null, parentLdr=null, record=true, nodeFilter=null, seqNum=n/a]
[2017-10-09 14:26:52.203] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- LocalDeploymentSpi: Registering [ldrRsrcs={org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f={org.apache.ignite.internal.util.typedef.T2=org.apache.ignite.internal.util.typedef.T2, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionHistorySuppliersMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionHistorySuppliersMap, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, java.util.Collections$UnmodifiableList=java.util.Collections$UnmodifiableList, org.apache.ignite.internal.visor.cache.VisorCacheMetricsCollectorTask=org.apache.ignite.internal.visor.cache.VisorCacheMetricsCollectorTask, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, org.apache.ignite.internal.processors.service.GridServiceProcessor$1=org.apache.ignite.internal.processors.service.GridServiceProcessor$1, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap}}, ldr=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f, rsrc=class org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1]
[2017-10-09 14:26:52.203] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- LocalDeploymentSpi: Resources to register: {org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1}
[2017-10-09 14:26:52.203] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- LocalDeploymentSpi: New resources: {org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1}
[2017-10-09 14:26:52.203] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- LocalDeploymentSpi: Removing resources [clsLdrToIgnore=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f, rsrcs={org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1}]
[2017-10-09 14:26:52.203] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridDeploymentLocalStore: Retrieved auto-loaded resource from spi: DeploymentResourceAdapter [name=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1, rsrcCls=class org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1, clsLdr=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f]
[2017-10-09 14:26:52.203] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridDeploymentLocalStore: Acquired deployment class: GridDeployment [ts=1507576972855, depMode=SHARED, clsLdr=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f, clsLdrId=6d9e6920f51-2e573c60-45f0-4429-a3fa-068489663148, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=false, usage=0]
[2017-10-09 14:26:52.203] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridResourceProcessor: Injecting resources [target=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1#61ea2cff]
[2017-10-09 14:26:52.211] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridDeploymentLocalStore: Deployment meta for local deployment: GridDeploymentMetadata [depMode=SHARED, alias=org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1, clsName=org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1, userVer=null, sndNodeId=2e573c60-45f0-4429-a3fa-068489663148, clsLdrId=null, clsLdr=null, participants=null, parentLdr=null, record=true, nodeFilter=null, seqNum=n/a]
[2017-10-09 14:26:52.211] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- LocalDeploymentSpi: Registering [ldrRsrcs={org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f={org.apache.ignite.internal.util.typedef.T2=org.apache.ignite.internal.util.typedef.T2, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionHistorySuppliersMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionHistorySuppliersMap, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, java.util.Collections$UnmodifiableList=java.util.Collections$UnmodifiableList, org.apache.ignite.internal.visor.cache.VisorCacheMetricsCollectorTask=org.apache.ignite.internal.visor.cache.VisorCacheMetricsCollectorTask, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, org.apache.ignite.internal.processors.service.GridServiceProcessor$1=org.apache.ignite.internal.processors.service.GridServiceProcessor$1, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap, org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1}}, ldr=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f, rsrc=class org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1]
[2017-10-09 14:26:52.211] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- LocalDeploymentSpi: Resources to register: {org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1=org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1}
[2017-10-09 14:26:52.211] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- LocalDeploymentSpi: New resources: {org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1=org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1}
[2017-10-09 14:26:52.211] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- LocalDeploymentSpi: Removing resources [clsLdrToIgnore=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f, rsrcs={org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1=org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1}]
[2017-10-09 14:26:52.211] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridDeploymentLocalStore: Retrieved auto-loaded resource from spi: DeploymentResourceAdapter [name=org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1, rsrcCls=class org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1, clsLdr=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f]
[2017-10-09 14:26:52.212] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridDeploymentLocalStore: Acquired deployment class: GridDeployment [ts=1507576972855, depMode=SHARED, clsLdr=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f, clsLdrId=6d9e6920f51-2e573c60-45f0-4429-a3fa-068489663148, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=false, usage=0]
[2017-10-09 14:26:52.212] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridResourceProcessor: Injecting resources [target=org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1#1ff41d49]
[2017-10-09 14:26:52.213] boot - 9955 DEBUG [pub-#34%null%] --- GridClosureProcessor: Grid runnable started: closure-proc-worker
[2017-10-09 14:26:52.213] boot - 9955 DEBUG [pub-#34%null%] --- GridClosureProcessor: Grid runnable finished normally: closure-proc-worker
[2017-10-09 14:26:52.216] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- TcpCommunicationSpi: Forcing NIO client close since node has left [nodeId=4cc6c321-d9cc-4149-a6ef-cba68877a269, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=3, bytesRcvd=714, bytesSent=6799, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-3, igniteInstanceName=null, finished=false, hashCode=1557230104, interrupted=false, runner=grid-nio-worker-tcp-comm-3-#20%null%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=1, sentCnt=1, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=4cc6c321-d9cc-4149-a6ef-cba68877a269, addrs=[10.70.255.8, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0, /127.0.0.1:0, /10.70.255.8:0], discPort=0, order=57, intOrder=31, lastExchangeTime=1507577126368, loc=false, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=1, sentCnt=1, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=4cc6c321-d9cc-4149-a6ef-cba68877a269, addrs=[10.70.255.8, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0, /127.0.0.1:0, /10.70.255.8:0], discPort=0, order=57, intOrder=31, lastExchangeTime=1507577126368, loc=false, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/10.70.242.138:47100, rmtAddr=/10.70.255.8:53916, createTime=1507577162587, closeTime=0, bytesSent=6799, bytesRcvd=714, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1507577162587, lastSndTime=1507577162697, lastRcvTime=1507577162617, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser#9573b3b, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1507577162587, closed=false, connIdx=0]]]
[2017-10-09 14:26:52.217] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- TcpCommunicationSpi: Offered move [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=3, bytesRcvd=714, bytesSent=6799, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-3, igniteInstanceName=null, finished=false, hashCode=1557230104, interrupted=false, runner=grid-nio-worker-tcp-comm-3-#20%null%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=1, sentCnt=1, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=4cc6c321-d9cc-4149-a6ef-cba68877a269, addrs=[10.70.255.8, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0, /127.0.0.1:0, /10.70.255.8:0], discPort=0, order=57, intOrder=31, lastExchangeTime=1507577126368, loc=false, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=1, sentCnt=1, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=4cc6c321-d9cc-4149-a6ef-cba68877a269, addrs=[10.70.255.8, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0, /127.0.0.1:0, /10.70.255.8:0], discPort=0, order=57, intOrder=31, lastExchangeTime=1507577126368, loc=false, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=true], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/10.70.242.138:47100, rmtAddr=/10.70.255.8:53916, createTime=1507577162587, closeTime=0, bytesSent=6799, bytesRcvd=714, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1507577162587, lastSndTime=1507577162697, lastRcvTime=1507577162617, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser#9573b3b, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], fut=NioOperationFuture [op=CLOSE]]
[2017-10-09 14:26:52.217] boot - 9955 DEBUG [disco-event-worker-#26%null%] --- GridIoManager: Removed messages from discovery startup delay list (sender node left topology): null
[2017-10-09 14:26:52.217] boot - 9955 DEBUG [pub-#35%null%] --- GridClosureProcessor: Grid runnable started: closure-proc-worker
[2017-10-09 14:26:52.217] boot - 9955 DEBUG [pub-#35%null%] --- GridClosureProcessor: Grid runnable finished normally: closure-proc-worker
[2017-10-09 14:26:52.283] boot - 9955 DEBUG [grid-timeout-worker-#15%null%] --- GridTimeoutProcessor: Timeout has occurred: org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject#7ff59c90
[2017-10-09 14:26:52.284] boot - 9955 DEBUG [grid-timeout-worker-#15%null%] --- GridDeploymentLocalStore: Deployment meta for local deployment: GridDeploymentMetadata [depMode=SHARED, alias=org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject$2, clsName=org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject$2, userVer=null, sndNodeId=2e573c60-45f0-4429-a3fa-068489663148, clsLdrId=null, clsLdr=null, participants=null, parentLdr=null, record=true, nodeFilter=null, seqNum=n/a]
[2017-10-09 14:26:52.285] boot - 9955 DEBUG [grid-timeout-worker-#15%null%] --- LocalDeploymentSpi: Registering [ldrRsrcs={org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f={org.apache.ignite.internal.util.typedef.T2=org.apache.ignite.internal.util.typedef.T2, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionHistorySuppliersMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionHistorySuppliersMap, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, java.util.Collections$UnmodifiableList=java.util.Collections$UnmodifiableList, org.apache.ignite.internal.visor.cache.VisorCacheMetricsCollectorTask=org.apache.ignite.internal.visor.cache.VisorCacheMetricsCollectorTask, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, org.apache.ignite.internal.processors.service.GridServiceProcessor$1=org.apache.ignite.internal.processors.service.GridServiceProcessor$1, org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1=org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$1$1, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionMap, org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap, org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1=org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1}}, ldr=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f, rsrc=class org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject$2]
2017-10-09 14:26:52.285] boot - 9955 DEBUG [grid-timeout-worker-#15%null%] --- GridDeploymentLocalStore: Retrieved auto-loaded resource from spi: DeploymentResourceAdapter [name=org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject$2, rsrcCls=class org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject$2, clsLdr=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f]
[2017-10-09 14:26:52.285] boot - 9955 DEBUG [grid-timeout-worker-#15%null%] --- GridDeploymentLocalStore: Acquired deployment class: GridDeployment [ts=1507576972855, depMode=SHARED, clsLdr=org.springframework.boot.loader.LaunchedURLClassLoader#7adf9f5f, clsLdrId=6d9e6920f51-2e573c60-45f0-4429-a3fa-068489663148, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=false, usage=0]
[2017-10-09 14:26:52.285] boot - 9955 DEBUG [grid-timeout-worker-#15%null%] --- GridResourceProcessor: Injecting resources [target=org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$NodeFailureTimeoutObject$2#3f183e4]
[2017-10-09 14:26:52.317] boot - 9955 DEBUG [http-nio-8081-exec-8] --- CacheHelper: Total time accessing cache ng-security-service-ORG_SPEC_CACHE for key * | value com.cache.model.PrefixCluster#6954be5d: 167 millis
[2017-10-09 14:26:52.319] boot - 9955 DEBUG [http-nio-8081-exec-8] --- OrgSpecCacheImpl: OrgSpec Cache Stats: OrgSpec ObjId: IgniteCacheProxy [delegate=GridNearCacheAdapter [], opCtx=null, restartFut=null] HitCount: 126, MissCount: 53, AvgReadTime: 126, Eviction Count: 0
[2017-10-09 14:26:52.321] boot - 9955 DEBUG [sys-#36%null%] --- GridClosureProcessor: Grid runnable started: closure-proc-worker
My question is, is this expected behavior? Can we get the near cache not be bypassed, or in the least re-establish using the near cache after the bad client disconnects.
It turns out their is a bug with the near cache. When a topology change occurs it can wipe out the topology version on the NearCacheGridEntry which will cause every call to check if the entry is valid to return false.
Bug has been submitted https://issues.apache.org/jira/browse/IGNITE-6767