I tried to solve the following problem by providing Kryo Serializer but it still doesn't work. It could not recognize the serializer of ModelCom. Also, any messages by print function don't show up.
I used Apache Flink 1.9.0 and Apache Jena 3.10.0
My code in Kotlin:
val serializer = object : Serializer<Model>(){
override fun write(kryo: Kryo, output: Output?, obj : Model?) {
print("write")
kryo.writeClassAndObject(output, obj)
}
override fun read(kryo: Kryo, input: Input?, type: Class<Model>?): Model {
print("read")
val m = kryo.readObject(input, Model::class.java)
return m
}
}
ExecutionContext.see.config.registerTypeWithKryoSerializer(ModelCom::class.java, serializer::class.java)
Error
Exception in thread "main" org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot serialize operator object class org.apache.flink.streaming.api.operators.SimpleUdfStreamOperatorFactory.
at org.apache.flink.streaming.api.graph.StreamConfig.setStreamOperatorFactory(StreamConfig.java:222)
at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.setVertexConfig(StreamingJobGraphGenerator.java:460)
at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createChain(StreamingJobGraphGenerator.java:272)
at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createChain(StreamingJobGraphGenerator.java:243)
at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createChain(StreamingJobGraphGenerator.java:243)
at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.setChaining(StreamingJobGraphGenerator.java:207)
at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:159)
at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:94)
at org.apache.flink.streaming.api.graph.StreamGraph.getJobGraph(StreamGraph.java:737)
at org.apache.flink.optimizer.plan.StreamingPlan.getJobGraph(StreamingPlan.java:40)
at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:86)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1507)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1489)
at core.EgressEngine.start(EgressEngine.kt:187)
at core.EgressEngineKt.main(EgressEngine.kt:45)
Caused by: java.io.NotSerializableException: org.apache.jena.rdf.model.impl.ModelCom
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
at java.base/java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
at java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
at java.base/java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
at java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
at java.base/java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
at java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:586)
at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(InstantiationUtil.java:515)
at org.apache.flink.streaming.api.graph.StreamConfig.setStreamOperatorFactory(StreamConfig.java:219)
... 14 more
Jena models are not serializable, so this approach isn't going to work. What you could do instead would be to send around just enough serialized data so that each instance that needs a model can instantiate one.
See this thread from the jena-users list about how to resolve this for Spark; the underlying issues are the same for any JVM-based framework that distributes computation.
Related
I am trying to create a class in a unit test that implements an Interface by using byte buddy
interface SomeInterface {}
class ByteBuddyTest {
#Test
fun byteBuddyTest(){
val instrumentation = ByteBuddyAgent.install()
val bb = ByteBuddy()
val loadedRestController = bb
.subclass(SomeInterface::class.java)
.make()
.load(Object::class.java.classLoader, ClassLoadingStrategy.Default.WRAPPER)
.loaded
}
}
Unfortunately I am getting a NoClassFoundException when trying to implement the Interface
org/example/SomeInterface
java.lang.NoClassDefFoundError: org/example/SomeInterface
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at net.bytebuddy.dynamic.loading.ByteArrayClassLoader.access$300(ByteArrayClassLoader.java:56)
at net.bytebuddy.dynamic.loading.ByteArrayClassLoader$ClassDefinitionAction.run(ByteArrayClassLoader.java:686)
at net.bytebuddy.dynamic.loading.ByteArrayClassLoader$ClassDefinitionAction.run(ByteArrayClassLoader.java:638)
at java.security.AccessController.doPrivileged(Native Method)
at net.bytebuddy.dynamic.loading.ByteArrayClassLoader.doPrivileged(ByteArrayClassLoader.java)
at net.bytebuddy.dynamic.loading.ByteArrayClassLoader.findClass(ByteArrayClassLoader.java:405)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
Am I missing something during agent initialization?
You are loading the class into a child loader of the boot loader by using
Object::class.java.classLoader
This class loader is of course unaware of SomeInterface. Instead, load the class into a child of:
SomeInterface::class.java.classLoader
I'd like to inject beans a Kotlin field in Quarkus. The sample file looks like
package org.example
import com.google.inject.Inject
import javax.enterprise.inject.spi.BeanManager
import javax.ws.rs.GET
import javax.ws.rs.Path
#Path("injectDemo")
open class InjectDemo #Inject constructor(val bm1: BeanManager) {
#field:Inject
protected open lateinit var bm2: BeanManager
#GET
fun demo() {
println("bm1 $bm1")
println("bm2 $bm2")
}
}
The constructor parameter injection works fine however field bm2 remains uninitialized.
Console output:
bm1 io.quarkus.arc.impl.BeanManagerImpl#6b7ac97f
2020-05-21 03:45:11,670 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /injectDemo failed, error id: ac118d6b-a26e-47e7-8c10-12e6a96e50ba-3: org.jboss.resteasy.spi.UnhandledException: kotlin.UninitializedPropertyAccessException: lateinit property bm2 has not been initialized
at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:216)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:515)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:259)
at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:160)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:362)
at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:163)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:245)
at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:123)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.access$000(VertxRequestHandler.java:36)
at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:87)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at java.base/java.lang.Thread.run(Thread.java:832)
at org.jboss.threads.JBossThread.run(JBossThread.java:479)
Caused by: kotlin.UninitializedPropertyAccessException: lateinit property bm2 has not been initialized
at org.example.InjectDemo.getBm2(InjectDemo.kt:12)
at org.example.InjectDemo.demo(InjectDemo.kt:17)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:621)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:487)
at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:437)
at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:362)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:439)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:400)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:374)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:67)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:488)
... 17 more
Is it possible possible to use CDI field injection in Quarkus in Kotlin? If yes, what needs to be set up for injection to work?
I'm running the app from an uberjar, not a native image.
The jar contains generated class org.example.InjectDemo_Bean containing a method create() disassembled version of which doesn't show any attempt of inject into bm2 field:
public InjectDemo create(CreationalContext var1) {
Object var2 = this.injectProviderSupplier1.get();
CreationalContextImpl var3 = CreationalContextImpl.child((InjectableReferenceProvider)var2, var1);
Object var4 = ((InjectableReferenceProvider)var2).get((CreationalContext)var3);
return new InjectDemo((BeanManager)var4);
}
According to https://quarkus.io/guides/kotlin#cdi-inject-with-kotlin Kotlin annotation reflection miss Target annotation which caused injection to fail. The solution is to add field javax.enterprise.inject.Default annotation:
#field:Default
#field:Inject
protected open lateinit var bm2: BeanManager
Let's say I have this interface:
interface Things {
fun size(): Int
}
And I want to subclass a List and implement this interface.
class Cars : ArrayList<String>, Things {}
I get a compilation error:
Inherited platform declarations clash: The following declarations have
the same JVM signature (size()I): fun (): Int defined in
Things fun size(): Int defined in Things
I can get around the compilation error by changing the size contract to a var implicit getter:
interface Things {
var size: Int
}
But then I get a long and complicated error that looks like a runtime error (IllegalStateException) but seems to happen when Kotlin is compiling :shrug:
I suspect I know why this is happening - ArrayList already has int size() and Kotlin now tries to add another of the same signature, but even if I'm right (10% chance?) it doesn't help much.
I'll also mention that it allows me to have get as part of the Things interface, which is satisfied by the list. I'm guessing the operator operator let's this be just different enough to slide?
So anyway, no IDE errors, but when I build I get this:
java.lang.IllegalStateException: Backend Internal error: Exception during code generation
Cause: Concrete fake override public open fun <set-size>(<set-?>: kotlin.Int): kotlin.Unit defined in org.blah.Stuff[PropertySetterDescriptorImpl#673f2280] should have exactly one concrete super-declaration: []
File being compiled at position: file:///blah/Stuff.kt
The root cause was thrown at: bridges.kt:122
at org.jetbrains.kotlin.codegen.CompilationErrorHandler.lambda$static$0(CompilationErrorHandler.java:24)
at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generate(PackageCodegenImpl.java:76)
at org.jetbrains.kotlin.codegen.DefaultCodegenFactory.generatePackage(CodegenFactory.kt:96)
at org.jetbrains.kotlin.codegen.DefaultCodegenFactory.generateModule(CodegenFactory.kt:67)
at org.jetbrains.kotlin.codegen.KotlinCodegenFacade.doGenerateFiles(KotlinCodegenFacade.java:47)
at org.jetbrains.kotlin.codegen.KotlinCodegenFacade.compileCorrectFiles(KotlinCodegenFacade.java:39)
at org.jetbrains.kotlin.cli.jvm.compiler.KotlinToJVMBytecodeCompiler.generate(KotlinToJVMBytecodeCompiler.kt:476)
at org.jetbrains.kotlin.cli.jvm.compiler.KotlinToJVMBytecodeCompiler.compileModules$cli(KotlinToJVMBytecodeCompiler.kt:164)
at org.jetbrains.kotlin.cli.jvm.K2JVMCompiler.doExecute(K2JVMCompiler.kt:166)
at org.jetbrains.kotlin.cli.jvm.K2JVMCompiler.doExecute(K2JVMCompiler.kt:56)
at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.kt:84)
at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.kt:42)
at org.jetbrains.kotlin.cli.common.CLITool.exec(CLITool.kt:104)
at org.jetbrains.kotlin.incremental.IncrementalJvmCompilerRunner.runCompiler(IncrementalJvmCompilerRunner.kt:349)
at org.jetbrains.kotlin.incremental.IncrementalJvmCompilerRunner.runCompiler(IncrementalJvmCompilerRunner.kt:105)
at org.jetbrains.kotlin.incremental.IncrementalCompilerRunner.compileIncrementally(IncrementalCompilerRunner.kt:237)
at org.jetbrains.kotlin.incremental.IncrementalCompilerRunner.access$compileIncrementally(IncrementalCompilerRunner.kt:37)
at org.jetbrains.kotlin.incremental.IncrementalCompilerRunner$compile$2.invoke(IncrementalCompilerRunner.kt:79)
at org.jetbrains.kotlin.incremental.IncrementalCompilerRunner.compile(IncrementalCompilerRunner.kt:91)
at org.jetbrains.kotlin.daemon.CompileServiceImpl.execIncrementalCompiler(CompileServiceImpl.kt:579)
at org.jetbrains.kotlin.daemon.CompileServiceImpl.access$execIncrementalCompiler(CompileServiceImpl.kt:102)
at org.jetbrains.kotlin.daemon.CompileServiceImpl$compile$$inlined$ifAlive$lambda$2.invoke(CompileServiceImpl.kt:455)
at org.jetbrains.kotlin.daemon.CompileServiceImpl$compile$$inlined$ifAlive$lambda$2.invoke(CompileServiceImpl.kt:102)
at org.jetbrains.kotlin.daemon.CompileServiceImpl$doCompile$$inlined$ifAlive$lambda$2.invoke(CompileServiceImpl.kt:1005)
at org.jetbrains.kotlin.daemon.CompileServiceImpl$doCompile$$inlined$ifAlive$lambda$2.invoke(CompileServiceImpl.kt:102)
at org.jetbrains.kotlin.daemon.common.DummyProfiler.withMeasure(PerfUtils.kt:138)
at org.jetbrains.kotlin.daemon.CompileServiceImpl.checkedCompile(CompileServiceImpl.kt:1047)
at org.jetbrains.kotlin.daemon.CompileServiceImpl.doCompile(CompileServiceImpl.kt:1004)
at org.jetbrains.kotlin.daemon.CompileServiceImpl.compile(CompileServiceImpl.kt:454)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:346)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Concrete fake override public open fun <set-size>(<set-?>: kotlin.Int): kotlin.Unit defined in org.blah.Stuff[PropertySetterDescriptorImpl#673f2280] should have exactly one concrete super-declaration: []
at org.jetbrains.kotlin.backend.common.bridges.BridgesKt.findConcreteSuperDeclaration(bridges.kt:122)
at org.jetbrains.kotlin.backend.common.bridges.BridgesKt.generateBridges(bridges.kt:59)
at org.jetbrains.kotlin.codegen.JvmBridgesImplKt.generateBridgesForFunctionDescriptorForJvm(JvmBridgesImpl.kt:92)
at org.jetbrains.kotlin.codegen.FunctionCodegen.generateBridges(FunctionCodegen.java:1041)
at org.jetbrains.kotlin.codegen.ClassBodyCodegen.generateBridges(ClassBodyCodegen.java:138)
at org.jetbrains.kotlin.codegen.ClassBodyCodegen.generateBody(ClassBodyCodegen.java:116)
at org.jetbrains.kotlin.codegen.MemberCodegen.generate(MemberCodegen.java:129)
at org.jetbrains.kotlin.codegen.MemberCodegen.genClassOrObject(MemberCodegen.java:302)
at org.jetbrains.kotlin.codegen.MemberCodegen.genClassOrObject(MemberCodegen.java:286)
at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generateClassesAndObjectsInFile(PackageCodegenImpl.java:118)
at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generateFile(PackageCodegenImpl.java:137)
at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generate(PackageCodegenImpl.java:68)
... 44 more
> Task :app:buildInfoGeneratorDebug
Anything I'm missing. I've spent most of my life in Java and am late to the Kotlin party. Any workarounds, alternatives... or is this a known thing?
TYIA
The problem is because kotlin have extension val property for List: size.
The first error says that you have two different kotlin "things" (method and property) which in jvm are the same. And the second error is because you have val implementation in ArrayList and var field in interface so your class Cars needs to implement both setter and getter but implements only getter (from List val size).
The solution is simple: just change your interface property to val size: Int
I'm trying to learn how to implement Retrofit2 and Moshi inside the Kotlin programming language. However, I seem to be having trouble trying to compile my code.
I define the following data classes/models which map to the json response I receive from the api I am hitting:
#JsonClass(generateAdapter = true)
data class Catalogs(
val languages: List<LanguageCatalog>
)
#JsonClass(generateAdapter = true)
data class LanguageCatalog(
val direction: String,
val identifier: String,
val title: String,
val resources: List<ResourceCatalog>
)
#JsonClass(generateAdapter = true)
data class Project(
val identifier: String,
val sort: Int,
val title: String,
val versification: String?
)
#JsonClass(generateAdapter = true)
data class ResourceCatalog(
val identifier: String,
val modified: String,
val projects: List<Project>,
val title: String,
val version: String
)
Then I have an interface which defines the behavior for the API:
interface Door43Service
{
#GET("v3/catalog.json")
fun getFormat() : Observable<Catalogs>
companion object
{
fun create(): Door43Service
{
val retrofit = Retrofit.Builder()
.addCallAdapterFactory(
RxJava2CallAdapterFactory.create()
)
.addConverterFactory(
MoshiConverterFactory.create()
)
.baseUrl("https://api.door32.org/")
.build()
return retrofit.create(Door43Service::class.java)
}
}
}
Lastly, I implemented everything inside a main function to get the json data from the api:
val door43Service by lazy {
Door43Service.create()
}
var disposable: Disposable? = null
fun main(args: Array<String>)
{
door43Service.getFormat()
.subscribe(
{ result -> println(result.languages)},
{ error -> println(error.message)}
)
}
The data that gets returned from the api is pretty long, but an example of it can be found at http://api-info.readthedocs.io/en/latest/door43.html
My issue is I am getting the following error in my stack:
Exception in thread "main" java.lang.IllegalArgumentException: Unable to create converter for class model.Catalogs
for method Door43Service.getFormat
at retrofit2.ServiceMethod$Builder.methodError(ServiceMethod.java:755)
at retrofit2.ServiceMethod$Builder.createResponseConverter(ServiceMethod.java:741)
at retrofit2.ServiceMethod$Builder.build(ServiceMethod.java:172)
at retrofit2.Retrofit.loadServiceMethod(Retrofit.java:170)
at retrofit2.Retrofit$1.invoke(Retrofit.java:147)
at com.sun.proxy.$Proxy0.getFormat(Unknown Source)
at MainKt.main(main.kt:11)
Caused by: java.lang.RuntimeException: Failed to find the generated JsonAdapter class for class model.Catalogs
at com.squareup.moshi.StandardJsonAdapters.generatedAdapter(StandardJsonAdapters.java:249)
at com.squareup.moshi.StandardJsonAdapters$1.create(StandardJsonAdapters.java:62)
at com.squareup.moshi.Moshi.adapter(Moshi.java:130)
at retrofit2.converter.moshi.MoshiConverterFactory.responseBodyConverter(MoshiConverterFactory.java:91)
at retrofit2.Retrofit.nextResponseBodyConverter(Retrofit.java:330)
at retrofit2.Retrofit.responseBodyConverter(Retrofit.java:313)
at retrofit2.ServiceMethod$Builder.createResponseConverter(ServiceMethod.java:739)
... 5 more
Caused by: java.lang.ClassNotFoundException: model.CatalogsJsonAdapter
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.squareup.moshi.StandardJsonAdapters.generatedAdapter(StandardJsonAdapters.java:236)
At first glance, my understanding is that the compiler thinks I haven't defined an adapter for my Catalogs class, but I think that's supposed to be covered using the #JsonClass(generateradapter = true) annotation. Is there anything I'm missing? Why can't my program generate the adapter for my Catalogs class?
So I got things working. All I had to do was rebuild the project by running gradle build in the terminal (I was originally running it in IntelliJ, but that didn't seem to actually run the build). The key issue was since the build wasn't running, the line in my gradle build script that says,
kapt "com.squareup.moshi:moshi-kotlin-codegen:$moshi_version"
wasn't running. This line basically gives the compiler to understand annotations in the code like #JsonClass. Without this line, the compiler won't understand the annotation. This was the root cause of my issue. I am keeping this post up in case anyone runs into the same issue.
I'd like to pass an object from the driver node to other nodes where an RDD resides, so that each partition of the RDD can access that object, as shown in the following snippet.
object HelloSpark {
def main(args: Array[String]): Unit = {
val conf = new SparkConf()
.setAppName("Testing HelloSpark")
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrator", "xt.HelloKryoRegistrator")
val sc = new SparkContext(conf)
val rdd = sc.parallelize(1 to 20, 4)
val bytes = new ImmutableBytesWritable(Bytes.toBytes("This is a test"))
rdd.map(x => x.toString + "-" + Bytes.toString(bytes.get) + " !")
.collect()
.foreach(println)
sc.stop
}
}
// My registrator
class HelloKryoRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) = {
kryo.register(classOf[ImmutableBytesWritable], new HelloSerializer())
}
}
//My serializer
class HelloSerializer extends Serializer[ImmutableBytesWritable] {
override def write(kryo: Kryo, output: Output, obj: ImmutableBytesWritable): Unit = {
output.writeInt(obj.getLength)
output.writeInt(obj.getOffset)
output.writeBytes(obj.get(), obj.getOffset, obj.getLength)
}
override def read(kryo: Kryo, input: Input, t: Class[ImmutableBytesWritable]): ImmutableBytesWritable = {
val length = input.readInt()
val offset = input.readInt()
val bytes = new Array[Byte](length)
input.read(bytes, offset, length)
new ImmutableBytesWritable(bytes)
}
}
In the snippet above, I tried to serialize ImmutableBytesWritable by Kryo in Spark, so I did the follwing:
configure the SparkConf instance passed to spark context, i.e., set "spark.serializer" to "org.apache.spark.serializer.KryoSerializer" and set "spark.kryo.registrator" to "xt.HelloKryoRegistrator";
Write a custom Kryo registrator class in which I register the class ImmutableBytesWritable;
Write a serializer for ImmutableBytesWritable
However, when I submit my Spark application in yarn-client mode, the following exception was thrown:
Exception in thread "main" org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
at org.apache.spark.SparkContext.clean(SparkContext.scala:1242)
at org.apache.spark.rdd.RDD.map(RDD.scala:270)
at xt.HelloSpark$.main(HelloSpark.scala:23)
at xt.HelloSpark.main(HelloSpark.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:325)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.NotSerializableException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:164)
... 12 more
It seems that ImmutableBytesWritable can't be serialized by Kryo. So what is the correct way to let Spark serialize an object using Kryo? Can Kryo serialize any type?
This is happening because you're using ImmutableBytesWritable in your closure. Spark doesn't support closure serialization with Kryo yet (only objects in RDDs). You can take the help of this to solve your problem:
Spark - Task not serializable: How to work with complex map closures that call outside classes/objects?
You simply need to serialize the objects before passing through the closure, and de-serialize afterwards. This approach just works, even if your classes aren't Serializable, because it uses Kryo behind the scenes. All you need is some curry. ;)
Here's an example sketch:
def genMapper(kryoWrapper: KryoSerializationWrapper[(Foo => Bar)])
(foo: Foo) : Bar = {
kryoWrapper.value.apply(foo)
}
val mapper = genMapper(KryoSerializationWrapper(new ImmutableBytesWritable(Bytes.toBytes("This is a test")))) _
rdd.flatMap(mapper).collectAsMap()
object ImmutableBytesWritable(bytes: Bytes) extends (Foo => Bar) {
def apply(foo: Foo) : Bar = { //This is the real function }
}