I am getting below error while running benchmark test using yardstick with default settings.
It is a standalone setup.
[root#db3 ~]# ./bin/benchmark-run-all.sh config/benchmark-sample.properties
<17:26:41> Failed to set up benchmark drivers (will shutdown and exit).
class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1922)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1235)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1787)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1711)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1141)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:639)
at org.apache.ignite.IgniteSpring.start(IgniteSpring.java:65)
at org.apache.ignite.yardstick.IgniteNode.start(IgniteNode.java:220)
at org.apache.ignite.yardstick.IgniteAbstractBenchmark.setUp(IgniteAbstractBenchmark.java:64)
at org.apache.ignite.yardstick.cache.IgniteCacheAbstractBenchmark.setUp(IgniteCacheAbstractBenchmark.java:107)
at org.yardstickframework.BenchmarkDriverStartUp.main(BenchmarkDriverStartUp.java:130)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1#1eff3cfb], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=5, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:280)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:985)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1917)
... 10 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to join node (Incompatible data region configuration [region=DEFAULT, locNodeId=8c2b6d02-01b3-4c22-8ad2-67c0c5f9ec4e, isPersistenceEnabled=true, rmtNodeId=4011d970-ae2d-4116-bb85-4311195c88a8, isPersistenceEnabled=false])
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:2047)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:1174)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:445)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2149)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:277
(Incompatible data region configuration [region=DEFAULT,
locNodeId=8c2b6d02-01b3-4c22-8ad2-67c0c5f9ec4e,
isPersistenceEnabled=true,
rmtNodeId=4011d970-ae2d-4116-bb85-4311195c88a8,
isPersistenceEnabled=false]) at
You have different persistence configurations for your nodes therefore they won't join each other according to the above error.
Related
I'm trying to change the state of a 3 node Gridgain cluster running on Kubernetes using the control.sh script as documented.
./control.sh --set-state INACTIVE
this usually should return success in a short time, but now it takes forever and the only way to break away is to CTRL+C. But after that cluster moves into an unexpected state where,
./control.sh --set-state ACTIVE
would fail. Below is the exception extracted from the gridgain log.
[1]
[SEVERE][rest-#70%dev%][GridJobWorker] Failed to execute job [jobId=a1fbeb63c71-ab1511ea-8668-442e-8aea-0d51e23026d6, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=o.a.i.i.v.misc.VisorChangeGridActiveStateTask, dep=LocalDeployment [super=GridDeployment [ts=1633005984012, depMode=SHARED, clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader#2c13da15, clsLdrId=c09ddb63c71-ab1511ea-8668-442e-8aea-0d51e23026d6, userVer=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=o.a.i.i.v.misc.VisorChangeGridActiveStateTask, sesId=91fbeb63c71-ab1511ea-8668-442e-8aea-0d51e23026d6, startTime=1633008816019, endTime=9223372036854775807, taskNodeId=ab1511ea-8668-442e-8aea-0d51e23026d6, clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader#2c13da15, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=ContainsNodeIdsPredicate [], subjId=ab1511ea-8668-442e-8aea-0d51e23026d6, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=939562377]], execName=null], jobId=a1fbeb63c71-ab1511ea-8668-442e-8aea-0d51e23026d6]]
class org.apache.ignite.IgniteException: Failed to activate cluster, because another state change operation is currently in progress: deactivate cluster
Following attempts at ./control.sh would immediately throw the below exception.
[2]
Command [SET-STATE] finished with code: 4
Error stack trace:
class org.apache.ignite.internal.client.GridClientException: null
suppressed:
at org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.handleClientResponse(GridClientNioTcpConnection.java:628)
at org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection.handleResponse(GridClientNioTcpConnection.java:559)
at org.apache.ignite.internal.client.impl.connection.GridClientConnectionManagerAdapter$NioListener.onMessage(GridClientConnectionManagerAdapter.java:694)
at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:278)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:108)
at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:115)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:108)
at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3714)
at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:174)
at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1193)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2504)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2269)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1891)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at java.base/java.lang.Thread.run(Thread.java:829)
On the server logs following was then observed.
[3]
class org.apache.ignite.IgniteCheckedException: Failed to send response to node. Unsupported direct type [message=GridDhtAffinityAssignmentRequest [flags=1, futId=23, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=2], super=GridCacheGroupIdMessage [grpId=-149688677]]]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processFailedMessage(GridCacheIoManager.java:1139)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:382)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1726)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1333)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$4800(GridIoManager.java:157)
at org.apache.ignite.internal.managers.communication.GridIoManager$8.execute(GridIoManager.java:1218)
at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:54)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to find message handler for message: GridDhtAffinityAssignmentRequest [flags=1, futId=23, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=2], super=GridCacheGroupIdMessage [grpId=-149688677]]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
... 11 more
Thanks in advance for your help to resolve this issue.
Gridgain version: 8.8.8
Deactivation might take a while because of different reasons and it's better to check the logs.
As for the commands, you can use the force mode:
control.(sh|bat) --set-state INACTIVE|ACTIVE|ACTIVE_READ_ONLY [--force] [--yes]
./control.sh --set-state INACTIVE --force
I have set up a 2 node cluster of apache ignite. Master keys are set up on both nodes with the same password using keytool. I am able to start one of the nodes. On starting the second node, it seems that the node is getting rejected from joining the cluster because of different digests.
Without the Transparent Data Encryption, the cluster had been working fine. I have used AWS-ELB for TcpDiscovery. I had persistence disabled previously. Now, I have enabled persistence and also activated the cluster using the control.sh.
-alias ignite.master.key \
-keystore ./ignite_keystore.jks \
-storetype PKCS12 \
-keyalg aes \
-storepass somepasswd \
-keysize 256```
```class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1#69fe0ed4], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
at org.apache.ignite.Ignition.start(Ignition.java:348)
at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Master key digest differs! Node join is rejected.
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:969)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
... 13 more
[14:47:45,576][SEVERE][main][IgniteKernal] Got exception while starting (will rollback startup routine).
class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1687)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
at org.apache.ignite.Ignition.start(Ignition.java:348)
at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1#69fe0ed4], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
... 11 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Master key digest differs! Node join is rejected.
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:969)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
... 13 more
[14:47:50] Ignite node stopped OK [uptime=00:00:07.942]
class org.apache.ignite.IgniteException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1026)
at org.apache.ignite.Ignition.start(Ignition.java:351)
at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1687)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1066)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
at org.apache.ignite.Ignition.start(Ignition.java:348)
... 1 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1#69fe0ed4], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null]
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:939)
at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1682)
... 11 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Master key digest differs! Node join is rejected.
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:969)
at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391)
at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020)
at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
... 13 more
Failed to start grid: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
An error means you use different master keys on the nodes and its has to be the same.
The easiest way to get equal master keys is to copy JKS file from one node to another.
Please, read the documentation about TDE - https://apacheignite.readme.io/docs/transparent-data-encryption
I have added my maven project' dependency ignite-spring 2.3.0 lib. And then this exeption occured java.util.concurrent.ExecutionException
I have moved ignite-spring file to libs folder of ignite folder.
Do i miss a project configuration and having trouble with creating threads?
Detailed error message;
SEVERE: A child container failed during start
java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/FrontEndService]]
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:939)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:872)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1419)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1409)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/FrontEndService]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
... 6 more
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#7e28b64b]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4869)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5001)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
... 6 more
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [org.apache.catalina.webresources.JarResourceSet#3361151a]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:140)
at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:709)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
... 9 more
Caused by: java.lang.IllegalArgumentException: java.util.zip.ZipException: invalid LOC header (bad signature)
at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:113)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
... 12 more
Caused by: java.util.zip.ZipException: invalid LOC header (bad signature)
at java.util.zip.ZipFile.read(Native Method)
at java.util.zip.ZipFile.access$1400(ZipFile.java:60)
at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:717)
at java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(ZipFile.java:419)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at sun.misc.IOUtils.readFully(IOUtils.java:65)
at java.util.jar.JarFile.getBytes(JarFile.java:425)
at java.util.jar.JarFile.getManifestFromReference(JarFile.java:193)
at java.util.jar.JarFile.getManifest(JarFile.java:180)
at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.initInternal(AbstractSingleArchiveResourceSet.java:111)
... 13 more
I think that you have a corrupted (empty probably) JAR file in libs directory of web app. Check them all with jar tf.
I've configured Apache Ignite 1.8.0 programmatically and can start a server with a single node, but when another node joins, they cannot communicate and I receive many of the following two messages in the logs. These continue until the other node is stopped.
ERROR 12:52:39,187-0800 [*Initialization*] util.nio.GridDirectParser: Failed to read message [msg=null, buf=java.nio.DirectByteBuffer[pos=5 lim=420 cap=32768], reader=null, ses=GridSelectorNioSessionImpl [selectorIdx=0, queueSize=1, writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=5 lim=420 cap=32768], recovery=null, super=GridNioSessionImpl [locAddr=/10.97.184.106:5702, rmtAddr=/10.97.189.92:58788, createTime=1484945559174, closeTime=0, bytesSent=0, bytesRcvd=420, sndSchedTime=1484945559174, lastSndTime=1484945559174, lastRcvTime=1484945559185, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser#21e93eaf, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]]]
class org.apache.ignite.IgniteException: Invalid message type: -84
at org.apache.ignite.internal.managers.communication.GridIoMessageFactory.create(GridIoMessageFactory.java:805)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$5.create(TcpCommunicationSpi.java:1631)
at org.apache.ignite.internal.util.nio.GridDirectParser.decode(GridDirectParser.java:76)
at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:104)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
at org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:113)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:2332)
at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:173)
at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:918)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:1583)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1516)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1289)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
WARN 12:52:39,188-0800 [*Initialization*] communication.tcp.TcpCommunicationSpi: Failed to process selector key (will close): GridSelectorNioSessionImpl [selectorIdx=0, queueSize=1, writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=5 lim=420 cap=32768], recovery=null, super=GridNioSessionImpl [locAddr=/10.97.184.106:5702, rmtAddr=/10.97.189.92:58788, createTime=1484945559174, closeTime=0, bytesSent=0, bytesRcvd=420, sndSchedTime=1484945559174, lastSndTime=1484945559174, lastRcvTime=1484945559185, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser#21e93eaf, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]]
ERROR 12:52:39,189-0800 [*Initialization*] communication.tcp.TcpCommunicationSpi: Closing NIO session because of unhandled exception.
class org.apache.ignite.internal.util.nio.GridNioException: Invalid message type: -84
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:1595)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1516)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1289)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteException: Invalid message type: -84
at org.apache.ignite.internal.managers.communication.GridIoMessageFactory.create(GridIoMessageFactory.java:805)
at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$5.create(TcpCommunicationSpi.java:1631)
at org.apache.ignite.internal.util.nio.GridDirectParser.decode(GridDirectParser.java:76)
at org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:104)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
at org.apache.ignite.internal.util.nio.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:113)
at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:107)
at org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:2332)
at org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:173)
at org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:918)
at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:1583)
... 4 more
Version information.
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 1.8.0#20161205-sha1:9ca40dbeb7d559fcb299bdb6f5c90cdf8ce7e533
>>> +----------------------------------------------------------------------+
>>> OS name: Windows Server 2012 R2 6.3 amd64
>>> CPU(s): 2
>>> Heap: 3.6GB
>>> VM name: 13752#host
>>> Grid name: T-XXX
>>> Local node [ID=983EC5A0-2D9A-40C9-B4C3-3D59739BDDB9, order=1, clientMode=false]
>>> Local node addresses: [hostname.example.com/0:0:0:0:0:0:0:1, /10.97.184.106, /127.0.0.1]
>>> Local ports: TCP:5702 TCP:5703 TCP:5705
One of the similar issues I've found in my research is that it is recommended to disable the shared memory feature (setSharedMemoryPort -1) as a first step in removing a problem like this.
The server is running on Windows and the other server joining the cache is on OSX.
INFO 12:50:17,569-0800 [*Initialization*] ignite.internal.IgniteKernal%T-XXX: OS: Windows Server 2012 R2 6.3 amd64
How do I prevent these errors? Have I configured the cluster poorly or is there an incompatibility between the two machines I am using?
Very likely it's a misconfiguration issue. This can happen if discovery SPI on one node tries to connect to communication SPI on another node. See this post: http://apache-ignite-users.70518.x6.nabble.com/Invalid-message-type-84-error-td9869.html
Without the Phoenix core, spring boot starts tomcat correctly.
Problem: Add
compile('org.apache.phoenix:phoenix-core:4.7.0-HBase-1.1') to the dependencies section of my build.gradle and Tomcat fails to start
dependencies {
compile('org.apache.phoenix:phoenix-core:4.7.0-HBase-1.1')
compile("org.springframework.data:spring-data-commons")
compile("org.springframework.boot:spring-boot-starter-jdbc:1.3.2.RELEASE")
compile('org.springframework.boot:spring-boot-starter-web:1.3.2.RELEASE')
compile("org.springframework:spring-test:4.2.4.RELEASE")
}
with the following exception,
org.apache.catalina.core.ContainerBase startInternal
SEVERE: A child container failed during start
java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Tomcat].StandardHost[localhost].StandardContext[/phoenix]]
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1123)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:816)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Tomcat].StandardHost[localhost].StandardContext[/phoenix]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)
... 6 more
Caused by: java.lang.NoSuchMethodError: javax.servlet.ServletContext.addServlet(Ljava/lang/String;Ljavax/servlet/Servlet;)Ljavax/servlet/ServletRegistration$Dynamic;
at org.springframework.boot.context.embedded.ServletRegistrationBean.onStartup(ServletRegistrationBean.java:190)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.selfInitialize(EmbeddedWebApplicationContext.java:225)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.access$000(EmbeddedWebApplicationContext.java:85)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext$1.onStartup(EmbeddedWebApplicationContext.java:209)
at org.springframework.boot.contex2016-03-07 23:40:29.048 WARN 71400 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
ded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:55)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5513)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
... 6 more
The following line added to the dependencies helped resolve the issue.
providedCompile "javax.servlet:javax.servlet-api:3.0.1"
dependencies {
providedCompile "javax.servlet:javax.servlet-api:3.0.1"
compile('org.apache.phoenix:phoenix-core:4.7.0-HBase-1.1')
compile("org.springframework.data:spring-data-commons")
compile("org.springframework.boot:spring-boot-starter-jdbc:1.3.2.RELEASE")
compile('org.springframework.boot:spring-boot-starter-web:1.3.2.RELEASE')
compile("org.springframework:spring-test:4.2.4.RELEASE")
}