I'm trying to read a RabbitMQ queue using Apache Beam.
I've written some transformation code to have messages written to Kafka.
I've even tested my scenario using simple text messages.
Now I try to deploy it with the effective messages this transformer is made to run on. These are JSON message of a quite moderate size.
Strangely, when i try to read "production" messages, I get this exception stack trace.
java.lang.IllegalArgumentException: Unable to encode element 'ValueWithRecordId{id=[], value=org.apache.beam.sdk.io.rabbitmq.RabbitMqMessage#f179a7f}' with coder 'ValueWithRecordId$ValueWithRecordIdCoder(org.apache.beam.sdk.coders.SerializableCoder#76190fb2)'.
org.apache.beam.sdk.coders.Coder.getEncodedElementByteSize(Coder.java:300)
org.apache.beam.sdk.coders.Coder.registerByteSizeObserver(Coder.java:291)
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.registerByteSizeObserver(WindowedValue.java:564)
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.registerByteSizeObserver(WindowedValue.java:480)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$ElementByteSizeObservableCoder.registerByteSizeObserver(IntrinsicMapTaskExecutorFactory.java:400)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputObjectAndByteCounter.update(OutputObjectAndByteCounter.java:125)
org.apache.beam.runners.dataflow.worker.DataflowOutputCounter.update(DataflowOutputCounter.java:64)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:43)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1283)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:147)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1020)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Caused by: java.io.NotSerializableException: com.rabbitmq.client.impl.LongStringHelper$ByteArrayLongString
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
java.util.HashMap.internalWriteEntries(HashMap.java:1785)
java.util.HashMap.writeObject(HashMap.java:1362)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
org.apache.beam.sdk.coders.SerializableCoder.encode(SerializableCoder.java:183)
org.apache.beam.sdk.coders.SerializableCoder.encode(SerializableCoder.java:53)
org.apache.beam.sdk.values.ValueWithRecordId$ValueWithRecordIdCoder.encode(ValueWithRecordId.java:105)
org.apache.beam.sdk.values.ValueWithRecordId$ValueWithRecordIdCoder.encode(ValueWithRecordId.java:99)
org.apache.beam.sdk.values.ValueWithRecordId$ValueWithRecordIdCoder.encode(ValueWithRecordId.java:81)
org.apache.beam.sdk.coders.Coder.getEncodedElementByteSize(Coder.java:297)
org.apache.beam.sdk.coders.Coder.registerByteSizeObserver(Coder.java:291)
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.registerByteSizeObserver(WindowedValue.java:564)
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.registerByteSizeObserver(WindowedValue.java:480)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$ElementByteSizeObservableCoder.registerByteSizeObserver(IntrinsicMapTaskExecutorFactory.java:400)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputObjectAndByteCounter.update(OutputObjectAndByteCounter.java:125)
org.apache.beam.runners.dataflow.worker.DataflowOutputCounter.update(DataflowOutputCounter.java:64)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:43)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1283)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:147)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1020)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
My understanding is that the RabbitMQ reader consider the messages big enough to require the use of LongString, which is not serializable.
Am I right on this point ? And if so, how do I suggest RabbitMQ to use a simple String (which will be enough for these messages) ?
This is an Apache Beam (https://issues.apache.org/jira/browse/BEAM-7414) for which solution is ... not yet merged into Apache Beam repo by pure laziness (this is bad). If someone wants to have the fix immediatly, it is possible to build my branch https://github.com/Riduidel/beam/tree/fix/rabbitmq-message-not-serializable
Related
I'm using Payara 5.184 and Java 1.8.241 on Linux (Ubuntu 19.10). I'm also using Liquibase as my database schema change management. A field named myDataSource is being injected as follows:
#Resource(mappedName = "java:global/ds")
private DataSource myDataSource;
When the method createDataSource() from Liquibase is invoked I noticed that the variable myDataSource is null, what makes me understand that the resource is not being properly injected. This error seems to happen only on Linux, as my other colleagues that run Windows did not have any problems so far. We're using exactly the same versions of Payara/Java. Is there any specific step that needs to be done on Linux? Here it is the stacktrace found on Payara logs.
Exception during lifecycle processing
org.glassfish.deployment.common.DeploymentException: CDI deployment failure:Exception List with 2 exceptions:
Exception 0 :
org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void liquibase.integration.cdi.CDILiquibase.onStartup() on liquibase.integration.cdi.CDILiquibase#911a89a
at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:85)
at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:66)
at org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122)
at liquibase.integration.cdi.CDIBootstrap$1.create(CDIBootstrap.java:95)
at liquibase.integration.cdi.CDIBootstrap$1.create(CDIBootstrap.java:38)
at org.jboss.weld.contexts.AbstractContext.get(AbstractContext.java:96)
at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:102)
at org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:131)
at liquibase.integration.cdi.CDILiquibase$Proxy$_$$_WeldClientProxy.toString(Unknown Source)
at liquibase.integration.cdi.CDIBootstrap.afterDeploymentValidation(CDIBootstrap.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:95)
at org.jboss.weld.injection.MethodInvocationStrategy$SpecialParamPlusBeanManagerStrategy.invoke(MethodInvocationStrategy.java:144)
at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:330)
at org.jboss.weld.event.ExtensionObserverMethodImpl.sendEvent(ExtensionObserverMethodImpl.java:123)
at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:308)
at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:286)
at javax.enterprise.inject.spi.ObserverMethod.notify(ObserverMethod.java:124)
at org.jboss.weld.util.Observers.notify(Observers.java:166)
at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:285)
at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:273)
at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:177)
at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:171)
at org.jboss.weld.bootstrap.events.AbstractContainerEvent.fire(AbstractContainerEvent.java:53)
at org.jboss.weld.bootstrap.events.AbstractDeploymentContainerEvent.fire(AbstractDeploymentContainerEvent.java:35)
at org.jboss.weld.bootstrap.events.AfterDeploymentValidationImpl.fire(AfterDeploymentValidationImpl.java:28)
at org.jboss.weld.bootstrap.WeldStartup.validateBeans(WeldStartup.java:499)
at org.jboss.weld.bootstrap.WeldBootstrap.validateBeans(WeldBootstrap.java:93)
at org.glassfish.weld.WeldDeployer.processApplicationLoaded(WeldDeployer.java:517)
at org.glassfish.weld.WeldDeployer.event(WeldDeployer.java:428)
at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:131)
at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:333)
at com.sun.enterprise.v3.server.ApplicationLifecycle.prepare(ApplicationLifecycle.java:497)
at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:540)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:549)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:545)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:544)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:575)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:567)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:566)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1475)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:111)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1857)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1733)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:564)
at com.sun.enterprise.v3.admin.AdminAdapter.onMissingResource(AdminAdapter.java:251)
at org.glassfish.grizzly.http.server.StaticHttpHandlerBase.service(StaticHttpHandlerBase.java:166)
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:83)
... 74 more
Caused by: java.lang.NullPointerException
at liquibase.integration.cdi.CDILiquibase.performUpdate(CDILiquibase.java:126)
at liquibase.integration.cdi.CDILiquibase.onStartup(CDILiquibase.java:116)
... 79 more
EDIT: Added stacktrace
EDIT 2: Problem solved! It ended up being something not related with Liquibase at all. The queue size of the PayaraExecutorService was too small, which leads me to think that some tasks submissions were rejected. Apparently this is a known issue of Payara 5.184, and a related issue had already been opened on Github (https://github.com/payara/Payara/issues/3495). Once I increased the size of the queue to a large number everything started working as expected.
I think you need to use lookup to inject the resource.
Compare this bug report: https://github.com/payara/Payara/issues/4413
I am seeing Cannot resolve type description errors when trying to load classes instrumented using Byte Buddy v1.9.11 within Mule 4 (JDK 1.8). Is there a known issue with Byte Buddy, or a particular combination of isolated classloader Mule 4 is using with Byte Buddy instrumented classes? Any suggestions are appreciated.
I am using Elastic Java APM agent to instrument Mule 4 with its over-complicated classloader isolation mechanism that seems to interfere with Byte Buddy v1.9.11 used by the APM agent.
java.lang.IllegalStateException: Cannot resolve type description for java.lang.String
at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Resolution$Illegal.resolve(TypePool.java:159)
at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType.toErasure(TypePool.java:6505)
at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Default$LazyTypeDescription$LazyTypeList.get(TypePool.java:6313)
at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Default$LazyTypeDescription$LazyTypeList.get(TypePool.java:6286)
at java.util.AbstractList$Itr.next(AbstractList.java:358)
at co.elastic.apm.agent.shaded.bytebuddy.matcher.ElementMatchers.takesArguments(ElementMatchers.java:1301)
at co.elastic.apm.agent.shaded.bytebuddy.dynamic.scaffold.inline.InliningImplementationMatcher.of(InliningImplementationMatcher.java:71)
at co.elastic.apm.agent.shaded.bytebuddy.dynamic.scaffold.inline.RedefinitionDynamicTypeBuilder.make(RedefinitionDynamicTypeBuilder.java:202)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple$Resolution.apply(AgentBuilder.java:10132)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:10551)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10514)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1500(AgentBuilder.java:10280)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:10889)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:10836)
at java.security.AccessController.doPrivileged(Native Method)
at co.elastic.apm.agent.shaded.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:10437)
at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at org.mule.runtime.module.artifact.api.classloader.FineGrainedControlClassLoader.findLocalClass(FineGrainedControlClassLoader.java:171)
at org.mule.runtime.module.artifact.api.classloader.FineGrainedControlClassLoader.loadClass(FineGrainedControlClassLoader.java:88)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at co.elastic.apm.api.NoopTransaction.startSpan(NoopTransaction.java:132)
at co.elastic.apm.agent.utils.SpanUtils.startSpan(SpanUtils.java:34)
at co.elastic.apm.agent.internal.ApmMessageProcessorNotificationListener.onNotification(ApmMessageProcessorNotificationListener.java:27)
at co.elastic.apm.agent.internal.ApmMessageProcessorNotificationListener.onNotification(ApmMessageProcessorNotificationListener.java:10)
at org.mule.runtime.core.api.context.notification.ServerNotificationManager.lambda$fireNotification$0(ServerNotificationManager.java:193)
at org.mule.runtime.core.internal.context.notification.Sender.dispatch(Sender.java:28)
at org.mule.runtime.core.internal.context.notification.Policy.dispatchToSenders(Policy.java:146)
at org.mule.runtime.core.internal.context.notification.Policy.doDispatch(Policy.java:134)
at org.mule.runtime.core.internal.context.notification.Policy.dispatch(Policy.java:106)
at org.mule.runtime.core.api.context.notification.ServerNotificationManager.notifyListeners(ServerNotificationManager.java:211)
at org.mule.runtime.core.api.context.notification.ServerNotificationManager.fireNotification(ServerNotificationManager.java:193)
at org.mule.runtime.core.privileged.processor.chain.AbstractMessageProcessorChain.fireNotification(AbstractMessageProcessorChain.java:373)
at org.mule.runtime.core.privileged.processor.chain.AbstractMessageProcessorChain.lambda$preNotification$23(AbstractMessageProcessorChain.java:339)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:190)
at org.mule.runtime.core.privileged.processor.chain.AbstractMessageProcessorChain$1.onNext(AbstractMessageProcessorChain.java:292)
at org.mule.runtime.core.privileged.processor.chain.AbstractMessageProcessorChain$1.onNext(AbstractMessageProcessorChain.java:285)
at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:127)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:398)
at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:484)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:111)
at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I assume that the class file resolver used by the Elasticsearch agent is querying the class loader directly where this particular class loader is hiding the class files in question.
You can argue that this is an issue of the class loader hiding the class file or an issue of the elastic search agent not attempting other class loaders or reattempting the redefinition of the loaded class. I'd contact both entities as this can disturb different tooling.
Currently refactoring some PySpark code and this specific snippet is causing me issues whereas it has ran fine before:
zip_table=spark.read.format("org.apache.spark.sql.execution.datasources.csv.CSVFileFormat").schema(schemas.zip_schema).load(path_to_file,header=True)
zip_pandas = zip_table.toPandas()
running into:
Py4JJavaError: An error occurred while calling o167.get.
: java.util.NoSuchElementException: spark.sql.execution.pandas.respectSessionTimeZone
at org.apache.spark.sql.internal.SQLConf$$anonfun$getConfString$2.apply(SQLConf.scala:1175)
at org.apache.spark.sql.internal.SQLConf$$anonfun$getConfString$2.apply(SQLConf.scala:1175)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.internal.SQLConf.getConfString(SQLConf.scala:1175)
at org.apache.spark.sql.RuntimeConfig.get(RuntimeConfig.scala:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
Other Spark Dataframes are successfully able to call toPandas() while others such as this return the same error.
The file in question is not particularly big and should be able to fit in the driver no problem. Also, I know it may be better to just load it into Pandas directly and not convert from Spark, but there is further logic in the code that requires both Spark/Pandas dataframes (and a later re-architecture of this work will address this issue).
I have the following route configured in apache-camel
from("direct:hazelCast")
.setHeader(HazelcastConstants.OPERATION, constant(HazelcastConstants.PUT_OPERATION))
.toF("hazelcast:map:testHazel", HazelcastConstants.MAP_PREFIX);
But, when the above route is invoked I'm getting the following error:
java.lang.NullPointerException: Null key is not allowed!
at com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:95)
at com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:89)
at org.apache.camel.component.hazelcast.map.HazelcastMapProducer.put(HazelcastMapProducer.java:125)
at org.apache.camel.component.hazelcast.map.HazelcastMapProducer.process(HazelcastMapProducer.java:60)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:814)
at org.apache.camel.processor.MulticastProcessor.access$200(MulticastProcessor.java:84)
at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:314)
at org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:299)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
The code that I used was almost similar to what was there is camel docs http://camel.apache.org/hazelcast-component.html
The following is the code which produces the data to hazelcast
The following is the code snippet that I have used to produce the data to hazelcast in camel:
from("direct:hazelCast")
.setHeader(HazelcastConstants.OPERATION, constant(HazelcastConstants.PUT_OPERATION))
.setHeader(HazelcastConstants.OBJECT_ID, constant("SOME BLA BLA"))
.split()
.tokenizeXML(<SOMEValidTag>).streaming()
.unmarshal(jaxb)
.convertBodyTo(<Valid>.class)
.marshal().json(JsonLibrary.Jackson)
.toF("hazelcast:%stestHazel", HazelcastConstants.MAP_PREFIX);
Note: We need to convert the body to class which is should be serializeable
You need to set the objectid it is missing
I am Learning Esper, and I want to learn about Data flow more.
I used ActiveMQ, so I want to using data flow and EsperIO to connect ActiveMQ.
I know about AMQP using EsperIO and ActiveMQ, so I hope that is work.
but my source is got error and they didn't find ActiveMQ. So, How I can Connect with ActiveMQ and EsperIO via Data Flow EPL clause?
bellows, my Error text :
Exception in thread "Thread-1" java.lang.NoClassDefFoundError: Lcom/rabbitmq/client/Connection;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Unknown Source)
at java.lang.Class.getDeclaredFields(Unknown Source)
at com.espertech.esper.util.JavaClassHelper.findFieldInternal(JavaClassHelper.java:1799)
at com.espertech.esper.util.JavaClassHelper.findAnnotatedFields(JavaClassHelper.java:1784)
at com.espertech.esper.dataflow.core.DataFlowServiceImpl.injectObjectProperties(DataFlowServiceImpl.java:338)
at com.espertech.esper.dataflow.core.DataFlowServiceImpl.instantiateOperator(DataFlowServiceImpl.java:320)
at com.espertech.esper.dataflow.core.DataFlowServiceImpl.instantiateOperators(DataFlowServiceImpl.java:294)
at com.espertech.esper.dataflow.core.DataFlowServiceImpl.instantiateInternal(DataFlowServiceImpl.java:226)
at com.espertech.esper.dataflow.core.DataFlowServiceImpl.instantiate(DataFlowServiceImpl.java:126)
at com.espertech.esper.dataflow.core.DataFlowServiceImpl.instantiate(DataFlowServiceImpl.java:113)
at Esper.EventReciveThread.run(EventReciveThread.java:40)
at java.lang.Thread.run(Unknown Source)
and bellows, is my EPL code :
"create dataflow writeAMQP EventBusSource -> outstream<Event>{} AMQPSink(outstream){host:'localhost', queueName: 'testQueue', collector:{class:'ObjectToAMQPCollectorSerializable'}}