I am trying to use Spring JMSTemplate class inside rdd.foreach method, but i am getting Task Not Serializable error.
When i try with static variable it worked on local but in cluster, i am getting null pointer exception.
Sample Code:
inputRDD.foreach(record -> {
messageServices.send(record);
}
Error log:
org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:315)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:305)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:132)
at org.apache.spark.SparkContext.clean(SparkContext.scala:1891)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:869)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:868)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:148)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:109)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:286)
at org.apache.spark.rdd.RDD.foreach(RDD.scala:868)
at org.apache.spark.api.java.JavaRDDLike$class.foreach(JavaRDDLike.scala:327)
at org.apache.spark.api.java.AbstractJavaRDDLike.foreach(JavaRDDLike.scala:47)
at com.messenger.MessengerDriver.runJob(MessengerDriver.java:108)
at com.messenger.MessengerDriver.main(MessengerDriver.java:60)
Caused by: java.io.NotSerializableException: org.springframework.jms.core.JmsTemplate
Serialization stack:
- object not serializable (class: org.springframework.jms.core.JmsTemplate, value: org.springframework.jms.core.JmsTemplate#3b98b809)
- field (class: com.messenger.Messenger.activemq.MessageProducer, name: jmsTemplate, type: class org.springframework.jms.core.JmsTemplate)
- object (class com.messenger.Messenger.activemq.MessageProducer, com.messenger.Messenger.activemq.MessageProducer#662e682a)
- field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;)
- field (class: org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1, name: f$14, type: interface org.apache.spark.api.java.function.VoidFunction)
- object (class org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1, <function1>)
at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:81)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:312)
... 13 more
Does anyone faced the same kind of issue?
The proper pattern is to use repartition & mapPartitions.
repartition is to map the RDD to suitable size partition;
mapPartitions is to handlle each partition seperately, you can create JMSTemplate for each partition inside the passing function.
Related
Using ByteBuddy how can I create enum with constructors such as this one :
public enum EnumConstructorSample {
STATE1(10),
STATE2(15);
public int count;
EnumConstructorSample(int count){
this.count = count;
}
}
I tried this code and it gives me error.
Class enumClass = new ByteBuddy().makeEnumeration("STATE1", "STATE2")
.name("DynamicEnum")
.defineConstructor(Visibility.PACKAGE_PRIVATE)
.withParameters(int.class)
.intercept(FixedValue.value(1))
.make()
.load(EnumWithConstructor.class.getClassLoader(), ClassLoadingStrategy.Default.WRAPPER)
.getLoaded();
System.out.println(enumClass.getDeclaredConstructors()[0]);
This is the Error and it is happening in enumClass.getDeclaredConstructors()
Exception in thread "main" java.lang.VerifyError: Constructor must call super() or this() before return
Exception Details:
Location:
DynamicEnum.<init>(I)V #2: return
Reason:
Error exists in the bytecode
Bytecode:
0x0000000: 0457 b1
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getDeclaredConstructors(Class.java:2020)
at EnumWithConstructor.main(EnumWithConstructor.java:19)
For constructors, it is required to invoke the super method within the constructor. For enumerations, you'd need to invoke the Enum(String, int) constructor. You can implement this using MethodCall.invoke(...).onSuper().
If you wanted to achieve this, I'd recommend you to subclass Enum manually since you'd otherwise define multiple constructors for the enum you are creating where Byte Buddy would invoke its own enum constructor and the fields would all have its default value.
Rather, implement the method and return the value based on its name. You can for example use a MethodDelegation and then use a #This Enum<?> val injection where you switch over the name to return the correct value as if it was stored in a field.
I’m getting a NoClassDefFoundError when trying to invoke the Foo.serializer() method on a #Serializable class.
Here's my test case:
#Serializable
data class Foo(val data: String)
val jsonString = json.stringify(
Foo.serializer(), // <= Error happens here
Foo(data = "foo")
)
Attempting to run the code results in the following stack trace:
java.lang.NoSuchMethodError: 'void kotlinx.serialization.internal.SerialClassDescImpl.<init>(java.lang.String, kotlinx.serialization.internal.GeneratedSerializer, int)'
at com.example.Foo$$serializer.<clinit>(Foo.kt:7)
at com.example.Foo$Companion.serializer(Foo.kt)
This is the result of version mismatches between Kotlin and Kotlinx.serialization, as they are relatively tightly coupled. In my case I was using Kotlin 1.3.71 and kotlinx.serialization 0.14.0, so the solution was to upgrade kotlinx.serialization to 0.20.0.
Been trying to use Micronaut Data module for db persistence but getting following error when I try to save an entity with ZonedDateTime field.
io.micronaut.data.exceptions.DataAccessException: Cannot convert type [class java.time.ZonedDateTime] to target type: class java.util.Date. Consider defining a TypeConverter bean to handle this case.
at io.micronaut.data.runtime.mapper.QueryStatement.lambda$convertRequired$0(QueryStatement.java:204)
at java.base/java.util.Optional.orElseThrow(Optional.java:408)
at io.micronaut.data.runtime.mapper.QueryStatement.convertRequired(QueryStatement.java:203)
I would have assumed that Micronaut would have default implementation to handle Java8 date types. Not sure why it doesn't.
I have also registered a type converter below (where TypeUtils.toUtilDate is internal class which does the actual conversion). I see that Micronaut picked up this class (so it prints a log stmt if I add one).
#Factory
class TypeConverters {
#Singleton
fun zdatetimeToUtilDate(): TypeConverter<ZonedDateTime, Date> {
return TypeConverter { datetime, targetType, context -> Optional.of(TypeUtils.toUtilDate(datetime)) }
}
}
Even then, I'm still getting the above mentioned error. Any idea why would it be so?
When unmarshalling a object which has a Instant datatype, I am getting the below exception. Any pointers to resolve the issue?
I am trying to resolve it with java 8 Instant datatype, don't prefer to convert to LocalDate or another datatypes.
Exception in thread "main" java.lang.IllegalArgumentException: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of java.time.Instant: no suitable constructor found, can not deserialize from Object value (missing default constructor or creator, or perhaps need to add/enable type information?)
at [Source: [{"transactionType":"ADD_DOC","payload":{"hash":"hashofthedocument12","userId":"123","meta":{"key1":"data1","key2":"data2"}},"consensusTimestamp":{"epochSecond":1522205987,"nano":533000000},"id":"2aaa3863-062d-4fcc-a789-8aad3dfade57"}]; line: 1, column: 148] (through reference chain: java.lang.Object[][0]->ipos.hashgraph.model.ConsensedDocument["consensusTimestamp"])
at ipos.hashgraph.marshaller.JacksonMarshaller.unmarshalLenient(JacksonMarshaller.java:50)
at ipos.hashgraph.loadtest.LoadTestor.getTransaction(LoadTestor.java:61)
at ipos.hashgraph.loadtest.LoadTestor.main(LoadTestor.java:99)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of java.time.Instant: no suitable constructor found, can not deserialize from Object value (missing default constructor or creator, or perhaps need to add/enable type information?)
at [Source: [{"transactionType":"ADD_DOC","payload":{"hash":"hashofthedocument12","userId":"123","meta":{"key1":"data1","key2":"data2"}},"consensusTimestamp":{"epochSecond":1522205987,"nano":533000000},"id":"2aaa3863-062d-4fcc-a789-8aad3dfade57"}]; line: 1, column: 148] (through reference chain: java.lang.Object[][0]->ipos.hashgraph.model.ConsensedDocument["consensusTimestamp"])
Code:
ObjectMapper mapper = new ObjectMapper();
mapper.findAndRegisterModules();
try {
mapper.readValue(responseEntity1.getBody(), ConsensedDocument[].class);
} catch (IOException e) {
e.printStackTrace();
}
Model Object
String transactionType;
Document payload;
Instant consensusTimestamp;
UUID id;
You need an add-in to support JSR-310
Try following the example below:
https://github.com/FasterXML/jackson-modules-java8
I encountered the following exception:
Exception in thread "main" java.io.NotSerializableException: DStream checkpointing has been enabled but the DStreams with their functions are not serializable
I have enabled checkpoting outside, and use this class process something.
And it said that this class not serializablbe:
class EventhubsStateTransformComponent(inStream: DStream[EventhubsEvent]) extends PipelineComponent with Logging{
def process() = {
inStream.foreachRDD(rdd => {
if (rdd.isEmpty()) {
logInfo("Extract outstream is empty...")
} else {
logInfo("Extract outstream is not empty...")
}
})
// TODO eventhubsId is hardcode
val eventhubsId = "1"
val statePairStream = inStream.map(eventhubsEvent => ((eventhubsId, eventhubsEvent.partitionId), eventhubsEvent.eventOffset))
val eventhubsEventStateStream = statePairStream.mapWithState(StateSpec.function(EventhubsStreamState.updateStateFunc _))
val snapshotStateStream = eventhubsEventStateStream.stateSnapshots()
val out = snapshotStateStream.map(state => {
(state._1._1, state._1._2, state._2, System.currentTimeMillis() / 1000)
})
outStream = out
}
}
P.S EventhubsEvent is a case class.
=======================================================
New edited: After I make this class extends Serialzable, the exception disappeared. But I wonder what case we need to make our own class extends Serializable. Does it mean that if a class has foreachRDD operation, it will trigger checkpoint to validate code and it need the whole object which contains foreachRDD operation to be Serializable? Because in my memory, some case just require objects in foreachRDD scope need to be serializable.
Serialization stack:
- object not serializable (class: com.testdm.spark.streaming.etl.common.pipeline.EventhubsStateTransformComponent, value: com.testdm.spark.streaming.etl.common.pipeline.EventhubsStateTransformComponent#2a92a7fd)
- field (class: com.testdm.spark.streaming.etl.common.pipeline.EventhubsStateTransformComponent$$anonfun$process$1, name: $outer, type: class com.testdm.spark.streaming.etl.common.pipeline.EventhubsStateTransformComponent)
- object (class com.testdm.spark.streaming.etl.common.pipeline.EventhubsStateTransformComponent$$anonfun$process$1, <function1>)
- field (class: org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3, name: cleanedF$1, type: interface scala.Function1)
- object (class org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3, <function2>)
- writeObject data (class: org.apache.spark.streaming.dstream.DStream)
- object (class org.apache.spark.streaming.dstream.ForEachDStream, org.apache.spark.streaming.dstream.ForEachDStream#3e1cb83b)
- element of array (index: 0)
- array (class [Ljava.lang.Object;, size 16)
- field (class: scala.collection.mutable.ArrayBuffer, name: array, type: class [Ljava.lang.Object;)
- object (class scala.collection.mutable.ArrayBuffer, ArrayBuffer(org.apache.spark.streaming.dstream.ForEachDStream#3e1cb83b, org.apache.spark.streaming.dstream.ForEachDStream#46034134))
- writeObject data (class: org.apache.spark.streaming.dstream.DStreamCheckpointData)
- object (class org.apache.spark.streaming.dstream.DStreamCheckpointData, [
0 checkpoint files])
- writeObject data (class: org.apache.spark.streaming.dstream.DStream)
- object (class org.apache.spark.streaming.dstream.PluggableInputDStream, org.apache.spark.streaming.dstream.PluggableInputDStream#5066ad14)
- writeObject data (class: org.apache.spark.streaming.dstream.DStreamCheckpointData)
- object (class org.apache.spark.streaming.dstream.DStreamCheckpointData
//....
From Serialization stack:
object not serializable (class: com.testdm.spark.streaming.etl.common.pipeline.EventhubsStateTransformComponent, value: com.testdm.spark.streaming.etl.common.pipeline.EventhubsStateTransformComponent#2a92a7fd)
field (class: com.testdm.spark.streaming.etl.common.pipeline.EventhubsStateTransformComponent$$anonfun$process$1, name: $outer, type: class com.testdm.spark.streaming.etl.common.pipeline.EventhubsStateTransformComponent)
object (class com.testdm.spark.streaming.etl.common.pipeline.EventhubsStateTransformComponent$$anonfun$process$1, )
The name show which object are not serialized,so the outer is the field you should check where you use it.
Some object are not serializable and try use it in driver or in executor but not pass it from driver to other function which execute in executor.