Why is ignite failing to deserialize a GridClosureProcessor object? - ignite

Question

I know a similar stack has been posted by others. My question is "it looks like Ignite is trying to deserialize a GridClosureProcessor (or a closure from it?) . If so, why is it doing that? I am trying to root cause this issue but none of my code is in the stack except MyCallable mentioned at the top (not actually in the stack).
No cache puts are happening in this code path unless they are internal. I mention this because of a comment on another post that "Unknown pair" can be caused by a cache put of the wrong type.
I am focusing on
Failed to deserialize object
[typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
Here's the rest.
[2019-04-08 22:20:23,724][ERROR][pub-#63][GridJobWorker] Failed to
initialize job
[jobId=800890ff961-7ff6a786-9d4d-43d8-91a0-70225c5e3a4a,
ses=GridJobSessionImpl [ses=GridTaskSessionImpl
[taskName=com.obfucorp.aa.project.core.jobs.MyCallable,
dep=GridDeployment [ts=1554761996013, depMode=SHARED,
clsLdr=sun.misc.Launcher$AppClassLoader#764c12b6,
clsLdrId=730290ff961-8d93b961-09f2-48c3-bd2f-49db31aae61e, userVer=0,
loc=true,
sampleClsName=o.a.i.i.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1,
pendingUndeploy=false, undeployed=false, usage=1],
taskClsName=com.obfucorp.aa.project.core.jobs.MyCallable,
sesId=700890ff961-7ff6a786-9d4d-43d8-91a0-70225c5e3a4a,
startTime=1554762023663, endTime=9223372036854775807,
taskNodeId=7ff6a786-9d4d-43d8-91a0-70225c5e3a4a,
clsLdr=sun.misc.Launcher$AppClassLoader#764c12b6, closed=false,
cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false,
internal=false, topPred=null,
subjId=7ff6a786-9d4d-43d8-91a0-70225c5e3a4a, mapFut=IgniteFuture
[orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null,
hash=314803578]], execName=null],
jobId=800890ff961-7ff6a786-9d4d-43d8-91a0-70225c5e3a4a]] class
org.apache.ignite.IgniteCheckedException: Failed to deserialize object
[typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9908)
at
org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:438)
at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1117)
at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1921)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by: class
org.apache.ignite.binary.BinaryObjectException: Failed to deserialize
object
[typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:875)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
at
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310)
at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9902)
... 10 more Caused by: class
org.apache.ignite.binary.BinaryInvalidTypeException: Unknown pair
[platformId=0, typeId=-1409390795] at
org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:696)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1755)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
at
org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1799)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readObject(BinaryReaderExImpl.java:1329)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.readBinary(GridClosureProcessor.java:1872)
at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:834)
... 16 more Caused by: java.lang.ClassNotFoundException: Unknown pair
[platformId=0, typeId=-1409390795] at
org.apache.ignite.internal.MarshallerContextImpl.getClassName(MarshallerContextImpl.java:385)
at
org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerContextImpl.java:335)
at
org.apache.ignite.internal.binary.BinaryContext.descriptorForTypeId(BinaryContext.java:687)
... 22 more
update - wanted to note this is happening a fresh deployment. There are no old files or persisted anything around. All classes are either pulled from even or freshly compiled. ty.
Pavel, here is the (Scala) code (redacted)
object MyCallable {
type FooList = Array[Foo]
}
class MyCallable(cacheName: String) extends IgniteCallable[FooList] with Serializable with LazyLogging {
#IgniteInstanceResource
private var ignite: Ignite = _
override def call(): FooList = {
logger.debug("callable called.");
val fooCache = ignite.getOrCreateCache[String, Foo](cacheName)
val qry = new ScanQuery[String, Foo]()
qry.setLocal(true)
val cursor = fooCache.query(qry)
val ret = cursor.iterator().asScala.map(e => e.getValue).toArray
logger.info("load status array: {}", ret.mkString)
return ret
}
#IgniteInstanceResource
def setIgnite(ignite: Ignite): Unit = {
this.ignite = ignite
}
}

Answer 1

Caused by: class org.apache.ignite.binary.BinaryInvalidTypeException: Unknown pair [platformId=0, typeId=-1409390795]
Seems like you have lost your marshaller cache (marshaller/ dir).
You can do ignite.marshaller().marshal(new WhateverTypeIsCausingThis()); once to make this error go away.