What causes this Serialization error in Apache Spark 1.4.0 when calling:
sc.parallelize(strList, 4)
This exception is thrown:
com.fasterxml.jackson.databind.JsonMappingException:
Could not find creator property with name 'id' (in class org.apache.spark.rdd.RDDOperationScope)
Thrown from addBeanProps in Jackson: com.fasterxml.jackson.databind.deser.BeanDeserializerFactory#addBeanProps
The RDD is a Seq[String], and the #partitions doesn't seem to matter (tried 1, 2, 4).
There is no serialization stack trace, as normal the worker closure cannot be serialized.
What is another way to track this down?
#Interfector is correct. I ran into this issue also, here's a snippet from my sbt file and the 'dependencyOverrides' section which fixed it.
libraryDependencies ++= Seq(
"com.amazonaws" % "amazon-kinesis-client" % "1.4.0",
"org.apache.spark" %% "spark-core" % "1.4.0",
"org.apache.spark" %% "spark-streaming" % "1.4.0",
"org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.4.0",
"com.amazonaws" % "aws-java-sdk" % "1.10.2"
)
dependencyOverrides ++= Set(
"com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4"
)
I suspect that this is caused by the classpath providing you with a different version of jackson than the one Spark is expecting (that is 2.4.4 if I'm not mistaking). You will need to adjust your classpath so that the correct jackson is referenced first for Spark.
I had the same problem with a project built with Gradle and I excluded the transitive dependencies from the project that was creating the problem:
dependencies
{
compile('dependency.causing:problem:version')
{
exclude module: 'jackson-databind'
}
....
}
That worked perfectly for me.
This worked for me <dependency> excludeAll ExclusionRule(organization = "com.fasterxml.jackson.core")
Related
I am porting an existing application Java 7 to Java 11 (Drools 7.42.0). The application works as expected if I compile as Java 1.8 and deploy on Java 11 container (Weblogic 14). But if I compile the same code in Java 11, I get the following error. The only change between 1.8 and 11 is the java compiler level in Maven pom.
I have ASM7 as a dependency in Maven.
Caused by: java.lang.UnsupportedOperationException: This feature requires ASM7
at org.mvel2.asm.ClassVisitor.visitNestMember(ClassVisitor.java:236)
at org.mvel2.asm.ClassReader.accept(ClassReader.java:651)
at org.mvel2.asm.ClassReader.accept(ClassReader.java:391)
at org.drools.core.util.asm.ClassFieldInspector.processClassWithByteCode(ClassFieldInspector.java:107)
at org.drools.core.util.asm.ClassFieldInspector.processClassWithByteCode(ClassFieldInspector.java:114)
at org.drools.core.util.asm.ClassFieldInspector.<init>(ClassFieldInspector.java:86)
at org.drools.core.util.asm.ClassFieldInspector.<init>(ClassFieldInspector.java:74)
at org.drools.core.base.ClassFieldAccessorFactory.getClassFieldInspector(ClassFieldAccessorFactory.java:157)
at org.drools.core.base.ClassFieldAccessorFactory.getFieldType(ClassFieldAccessorFactory.java:141)
at org.drools.core.base.ClassFieldAccessorStore.getFieldType(ClassFieldAccessorStore.java:295)
at org.drools.compiler.rule.builder.PatternBuilder.getValueType(PatternBuilder.java:1079)
at org.drools.compiler.rule.builder.PatternBuilder.normalizeExpression(PatternBuilder.java:1047)
at org.drools.compiler.rule.builder.PatternBuilder.buildExpression(PatternBuilder.java:965)
at org.drools.compiler.rule.builder.PatternBuilder.buildCcdDescr(PatternBuilder.java:942)
at org.drools.compiler.rule.builder.PatternBuilder.build(PatternBuilder.java:767)
at org.drools.compiler.rule.builder.PatternBuilder.processConstraintsAndBinds(PatternBuilder.java:620)
at org.drools.compiler.rule.builder.PatternBuilder.build(PatternBuilder.java:187)
at org.drools.compiler.rule.builder.PatternBuilder.build(PatternBuilder.java:154)
at org.drools.compiler.rule.builder.PatternBuilder.build(PatternBuilder.java:136)
at org.drools.compiler.rule.builder.GroupElementBuilder.build(GroupElementBuilder.java:66)
at org.drools.compiler.rule.builder.RuleBuilder.build(RuleBuilder.java:107)
at org.drools.compiler.builder.impl.KnowledgeBuilderImpl.internalAddRule(KnowledgeBuilderImpl.java:1183)
at org.drools.compiler.builder.impl.KnowledgeBuilderImpl.addRule(KnowledgeBuilderImpl.java:1178)
at org.drools.compiler.builder.impl.KnowledgeBuilderImpl.lambda$null$3(KnowledgeBuilderImpl.java:1134)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
I noticed while debugging that this was caused by an "enum" in our code. Drools works fine if we remove the enum and use constants. The error seems to suggest that this could happen to any nested classes. Is there a better way to handle this other than changing our code to remove nested classes and enums?
I see that in the class org.drools.core.util.asm.ClassFieldInspector, the API is set to ASM5, but MVEL is expecting ASM7. But not sure why it would work in Java 8 and not in Java 11. Any suggestions?
ClassFieldVisitor(final Class< ? > cls,
final boolean includeFinalMethods,
final ClassFieldInspector inspector) {
super(Opcodes.ASM5);
this.clazz = cls;
this.includeFinalMethods = includeFinalMethods;
this.inspector = inspector;
}
Disclaimer: I'm one of the Drools developers
That's interesting as we might not emit valid ASM code.
Please file a bug on https://issues.redhat.com so that we can fix it
Thank you very much
Attempt at Kotlin Cookbook by Ken Kousen -- 1.5 Executing a Kotlin Script produces 'unable to instantiate class' error.
$ cat southpole.kts
import java.time.*
val instant = Instant.now()
val southPole = instant.atZone(ZoneId.of("Antarctica/South_Pole"))
val dst = southPole.zone.rules.isDaylightSavings(instant)
println("It is ${southPole.toLocalTime()} (UTC${southPole.offset}) at the South Pole")
println("The South Pole ${if (dst) "is" else "is not"} on Daylight Savings Time")
$ kotlinc -script southpole.kts
OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
error: unable to instantiate class Southpole (southpole.kts): java.lang.NoClassDefFoundError: kotlin/script/templates/standard/ScriptTemplateWithArgs
kotlin version 1.3.50
In order to work, println(...) needs a kotlin runtime which has to be added manually.
This issue is described here https://discuss.kotlinlang.org/t/possible-kts-bug/10162
..., it (script) takes dependencies from the module, so you need to include kotlin-script-runtime to the module dependencies explicitly.
(...) Unfortunately, it is not very obvious. We’re thinking about possible solutions.
This seems to be improved in the upcoming Kotlin 1.3.60 release
https://youtrack.jetbrains.com/issue/KT-33529
As as a work-around, use:
$ sdk use kotlin 1.3.41
I want to use vertx cluster with hazelcast on karaf. When I try to write messages to the bus (after cluster is formed) I am getting this serialization error. I was thinking about adding a class definition to hazelcast to tell it where to find the vertx server id class (io.vertx.spi.cluster.hazelcast.impl.HazelcastServerID) but I am not sure how.
On Karaf I had to wrap the vertx-hazelcast jar because it doesn't have a proper manifest file.
<bundle start-level="80">wrap:mvn:io.vertx/vertx-hazelcast/${vertx.version}</bundle>
here is my error
com.hazelcast.nio.serialization.HazelcastSerializationException: Problem while reading DataSerializable, namespace: 0, id: 0, class: 'io.vertx.spi.cluster.hazelcast.impl.HazelcastServerID', exception: io.vertx.spi.cluster.hazelcast.impl.
HazelcastServerID
at com.hazelcast.internal.serialization.impl.DataSerializer.read(DataSerializer.java:130)[11:com.hazelcast:3.6.3]
at com.hazelcast.internal.serialization.impl.DataSerializer.read(DataSerializer.java:47)[11:com.hazelcast:3.6.3]
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:46)[11:com.hazelcast:3.6.3]
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:170)[11:com.hazelcast:3.6.3]
at com.hazelcast.map.impl.DataAwareEntryEvent.getOldValue(DataAwareEntryEvent.java:82)[11:com.hazelcast:3.6.3]
at io.vertx.spi.cluster.hazelcast.impl.HazelcastAsyncMultiMap.entryRemoved(HazelcastAsyncMultiMap.java:147)[64:wrap_file__C__Users_gadei_development_github_effectus.io_effectus-core_core.test_core.test.exam_target_paxexam_unpack_
5bf4439f-01ff-4db4-bd3d-e3b6a1542596_system_io_vertx_vertx-hazelcast_3.4.0-SNAPSHOT_vertx-hazelcast-3.4.0-SNAPSHOT.jar:0.0.0]
at com.hazelcast.multimap.impl.MultiMapEventsDispatcher.dispatch0(MultiMapEventsDispatcher.java:111)[11:com.hazelcast:3.6.3]
at com.hazelcast.multimap.impl.MultiMapEventsDispatcher.dispatchEntryEventData(MultiMapEventsDispatcher.java:84)[11:com.hazelcast:3.6.3]
at com.hazelcast.multimap.impl.MultiMapEventsDispatcher.dispatchEvent(MultiMapEventsDispatcher.java:55)[11:com.hazelcast:3.6.3]
at com.hazelcast.multimap.impl.MultiMapService.dispatchEvent(MultiMapService.java:371)[11:com.hazelcast:3.6.3]
at com.hazelcast.multimap.impl.MultiMapService.dispatchEvent(MultiMapService.java:65)[11:com.hazelcast:3.6.3]
at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:56)[11:com.hazelcast:3.6.3]
at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:187)[11:com.hazelcast:3.6.3]
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:171)[11:com.hazelcast:3.6.3]
Caused by: java.lang.ClassNotFoundException: io.vertx.spi.cluster.hazelcast.impl.HazelcastServerID
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)[:1.8.0_101]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)[:1.8.0_101]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)[:1.8.0_101]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)[:1.8.0_101]
at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:137)[11:com.hazelcast:3.6.3]
at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:115)[11:com.hazelcast:3.6.3]
at com.hazelcast.nio.ClassLoaderUtil.newInstance(ClassLoaderUtil.java:68)[11:com.hazelcast:3.6.3]
at com.hazelcast.internal.serialization.impl.DataSerializer.read(DataSerializer.java:119)[11:com.hazelcast:3.6.3]
... 13 more
any suggestions appreciated
thanks.
This normally happens if one object has an acyclic serialization (reading one less / much property). In this case you're on a wrong stream position which means you end up reading the wrong datatype.
Another possible reason is multiple different Hazelcast versions in the classpath (please check that) or different versions on different nodes.
The solution involved classloading magic!
.setClassLoader(HazelcastClusterManager.class.getClassLoader())
I ended up rolling my own hazelcast instance and configuring it the way vertx specification is instructing with the additional classloader configuration trick.
```
ServiceReference serviceRef = context.getServiceReference(HazelcastOSGiService.class);
log.info("Hazelcast OSGi Service Reference: {}", serviceRef);
hazelcastOsgiService = context.getService(serviceRef);
log.info("Hazelcast OSGi Service: {}", hazelcastOsgiService);
hazelcastOsgiService.getClass().getClassLoader();
Map<String, SemaphoreConfig> semaphores = new HashMap<>();
semaphores.put("__vertx.*", new SemaphoreConfig().setInitialPermits(1));
Config hazelcastConfig = new Config("effectus-instance")
.setClassLoader(HazelcastClusterManager.class.getClassLoader())
.setGroupConfig(new GroupConfig("dev").setPassword("effectus"))
// .setSerializationConfig(new SerializationConfig().addClassDefinition()
.addMapConfig(new MapConfig()
.setName("__vertx.subs")
.setBackupCount(1)
.setTimeToLiveSeconds(0)
.setMaxIdleSeconds(0)
.setEvictionPolicy(EvictionPolicy.NONE)
.setMaxSizeConfig(new MaxSizeConfig().setSize(0).setMaxSizePolicy(MaxSizeConfig.MaxSizePolicy.PER_NODE))
.setEvictionPercentage(25)
.setMergePolicy("com.hazelcast.map.merge.LatestUpdateMapMergePolicy"))
.setSemaphoreConfigs(semaphores);
hazelcastOSGiInstance = hazelcastOsgiService.newHazelcastInstance(hazelcastConfig);
log.info("New Hazelcast OSGI instance: {}", hazelcastOSGiInstance);
hazelcastOsgiService.getAllHazelcastInstances().stream().forEach(instance -> {
log.info("Registered Hazelcast OSGI Instance: {}", instance.getName());
});
clusterManager = new HazelcastClusterManager(hazelcastOSGiInstance);
VertxOptions options = new VertxOptions().setClusterManager(clusterManager).setHAGroup("effectus");
Vertx.clusteredVertx(options, res -> {
if (res.succeeded()) {
Vertx v = res.result();
log.info("Vertx is running in cluster mode: {}", v);
// some more code...
```
so the issue is that hazelcast instance doesn't have access to the cleasass inside the vertx-hazelcst bundle.
I am sure there is a shorter cleaner way somewhere..
any better suggestions would be great.
I stuck in place while trying to register cq query with ClientCache. Still getting this exception:
CqService is not available.
java.lang.IllegalStateException: CqService is not available.
at org.apache.geode.cache.query.internal.cq.MissingCqService.start(MissingCqService.java:171)
at org.apache.geode.cache.query.internal.DefaultQueryService.getCqService(DefaultQueryService.java:777)
at org.apache.geode.cache.query.internal.DefaultQueryService.newCq(DefaultQueryService.java:486)
The client cache is created as follow:
def client(): ClientCache = new ClientCacheFactory()
.setPdxPersistent(true)
.setPdxSerializer(new ReflectionBasedAutoSerializer(false, "org.geode.importer.domain.FooBar"))
.addPoolLocator(ConfigProvider.locator.host, ConfigProvider.locator.port)
.setPoolSubscriptionEnabled(true)
.create()
and suggested solution does not help. Actual library version is:
"org.apache.geode" % "geode-core" % "1.0.0-incubating"
You will have to pull in geode-cq as a dependency. In gradle
compile 'org.apache.geode:geode-cq:1.0.0-incubating'
Using Google Dataproc Spark cluster, my sbt built assembly jar can access Cassandra via SparkContext.
However, when I try to access via sqlContext I get spark sql classes not found on the remote cluster - though I believe the dataproc cluster is supposed to be provisioned for spark sql.
java.lang.NoClassDefFoundError: org/apache/spark/sql/types/UTF8String$
at org.apache.spark.sql.cassandra.CassandraSQLRow$$anonfun$fromJavaDriverRow$1.apply$mcVI$sp(CassandraSQLRow.scala:50)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala
my sbt file:
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.0" % "provided",
"org.apache.spark" %% "spark-sql" % "1.5.0" % "provided",
"com.datastax.spark" %% "spark-cassandra-connector" % "1.4.0"
)
Turning off "provided" on spark-sql puts me in jar duplicate merge hell.
Thx for any help.
It looks like you also need version 1.5.0 of the spark-cassandra-connector to ensure your classes are compatible. Here's the commit which upgraded the cassandra connector to 1.5.0, and you can see it removes the import of org.apache.spark.sql.types.UTF8String and adds import org.apache.spark.unsafe.types.UTF8String instead, changing the relevant lines in CassandraSQLRow.scala:
data(i) = GettableData.get(row, i)
data(i) match {
case date: Date => data.update(i, new Timestamp(date.getTime))
- case str: String => data.update(i, UTF8String(str))
+ case bigInt: BigInteger => data.update(i, new JBigDecimal(bigInt))
+ case str: String => data.update(i, UTF8String.fromString(str))
case set: Set[_] => data.update(i, set.toSeq)
case _ =>
}
Though it appears there are only "milestone" artifact types rather than "release" types in Maven central for the cassandra connector, you should still be able to get the latest milestone connector 1.5.0-M2 to work with your code.
EDIT: Additional link to the compatibility table from the Cassandra connector's GitHub README.md