I have 5 different Jobs, some of them have an InputFormat and others use env.fromElements(...)
When I execute the Jobs using IntelliJ, all of them works correctly. But when I execute them from terminal, only jobs which contains env.fromElements() works.
Here is the error message:
Cluster configuration: Standalone cluster with JobManager at localhost/127.0.0.1:6123
Using address localhost:6123 to connect to JobManager.
JobManager web interface address http://localhost:8081
Starting execution of program
Submitting job with JobID: ab41f30df416c654b406c9b13e70f62e. Waiting for job completion.
Connected to JobManager at Actor[akka.tcp://flink#localhost:6123/user/jobmanager#1320020477]
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Cannot initialize task 'CHAIN DataSource (at main(JobSource.java:49) (es.mypackage.flink.Sources.MyInputFormat)) -> Map (Map at main(JobSource.java:53))': Configuring the InputFormat (es.mypackage.flink.Sources.MyInputFormat#1f81aa00) failed: null
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:101)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:400)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:387)
at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:926)
at es.mypackage.flink.Job.JobSource.main(JobSource.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:339)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:831)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:256)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1073)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1120)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1117)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1116)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'CHAIN DataSource (at main(JobSource.java:49) (es.mypackage.flink.Sources.MyInputFormat)) -> Map (Map at main(JobSource.java:53))': Configuring the InputFormat (es.mypackage.flink.Sources.MyInputFormat#1f81aa00) failed: null
at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:136)
at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1274)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:477)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:118)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Configuring the InputFormat (es.mypackage.flink.Sources.MyInputFormat#1f81aa00) failed: null
at org.apache.flink.runtime.jobgraph.InputFormatVertex.initializeOnMaster(InputFormatVertex.java:89)
at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:133)
... 20 more
Any help is much appreciated.
Edit:
public void configure(Configuration parameters) {
/*This line causes the problem*/
this.client = new KuduClient.KuduClientBuilder(KUDU_MASTER).build();
table = createTable(TABLE_NAME);
if (table != null) {
scanner = client.newScannerBuilder(table)
.setProjectedColumnNames(projectColumns)
.build();
}
}
Related
I have a Springboot Kotlin app running in AWS ECS/EC2 with some code written by my predecessor. It runs into OutOfMemoryError after running a while in the server. I am not familiar with ReactiveX and threading, so would like to ask for some help on this.
I see from the error logs that it should be related to the Observable.blockingForEach method?
The listOfKeys is a limited source of 5 to 50 items, so it should not be due to backpressure?
I also pasted the thread dump when OutOfMemoryError occurred.
Code inside myMethod
Observable.fromIterable(listOfKeys)
.flatMap { key ->
Maybe.fromCallable {
Pair(key, methodWhichCallsExternalAPI))
}
.toObservable()
.subscribeOn(Schedulers.from(Executors.newFixedThreadPool(4)))
}
.blockingForEach {
// some simple assignment of result to variables
}
Exception logs
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at io.reactivex.internal.schedulers.ExecutorScheduler.scheduleDirect(ExecutorScheduler.java:59)
at io.reactivex.internal.operators.observable.ObservableSubscribeOn.subscribeActual(ObservableSubscribeOn.java:36)
at io.reactivex.Observable.subscribe(Observable.java:12284)
at io.reactivex.internal.operators.observable.ObservableMap.subscribeActual(ObservableMap.java:32)
at io.reactivex.Observable.subscribe(Observable.java:12284)
at io.reactivex.internal.operators.observable.ObservableFlatMap$MergeObserver.subscribeInner(ObservableFlatMap.java:165)
at io.reactivex.internal.operators.observable.ObservableFlatMap$MergeObserver.onNext(ObservableFlatMap.java:139)
at io.reactivex.internal.operators.observable.ObservableFromIterable$FromIterableDisposable.run(ObservableFromIterable.java:98)
at io.reactivex.internal.operators.observable.ObservableFromIterable.subscribeActual(ObservableFromIterable.java:58)
at io.reactivex.Observable.subscribe(Observable.java:12284)
at io.reactivex.internal.operators.observable.ObservableFlatMap.subscribeActual(ObservableFlatMap.java:55)
at io.reactivex.Observable.subscribe(Observable.java:12284)
at io.reactivex.internal.operators.observable.BlockingObservableIterable.iterator(BlockingObservableIterable.java:39)
at io.reactivex.Observable.blockingForEach(Observable.java:5183)
at com.myOrg.MyController.myMethod(MyController.kt:240)
at com.myOrg.MyController$$FastClassBySpringCGLIB$$38688575.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)
at com.dbs.localisemanagement.localise.LocaliseController$$EnhancerBySpringCGLIB$$3e102d4f.lokaliseProjectsS3(<generated>)
at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)
Thread dump - each of the 3 parts are repeated many times
at sun.misc.Unsafe.park(Native Method)
"pool-30968-thread-1" Id=36794 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#394a0729
at java.lang.Thread.run(Thread.java:748)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#f0c2d74
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
- locked java.lang.Object#5e78eb81
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.socketRead0(Native Method)
"pool-30982-thread-1" Id=36808 RUNNABLE
- java.util.concurrent.ThreadPoolExecutor$Worker#7d90150f
Number of locked synchronizers = 1
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:306)
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:198)
- locked org.apache.http.pool.AbstractConnPool$2#a0abf72
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:253)
at org.apache.http.pool.AbstractConnPool.access$300(AbstractConnPool.java:70)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:393)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#7399fe63
at sun.misc.Unsafe.park(Native Method)
"pool-30987-thread-1" Id=36813 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#7399fe63
- java.util.concurrent.ThreadPoolExecutor$Worker#5b4dcfd1
Number of locked synchronizers = 1
The most likely problem is the repeated creation of a thread pool that is not released after use.
.subscribeOn(Schedulers.from(Executors.newFixedThreadPool(4)))
The more the method runs the more abandoned threads it creates.
You should probably create a static Scheduler and use that:
static final ExecutorService pool =
Executors.newFixedThreadPool(4);
static final Scheduler scheduler = Schedulers.from(pool);
Later on, after the app's lifecycle ended, call pool.shutdown() to free the threads.
I am using IgniteQueue to store some POJOs in memory which are removed at a 15 minute interval after some processing. During the processing, elements are added and removed from the queue multiple times using removeAll() api.
Below is my queue configuration -
#Override
public IgniteQueue<R> create(QueueName queueName, int capacity) {
return ignite.queue(queueName.getQueueName(), // Queue name.
capacity, // Queue capacity. 0 for unbounded queue.
getCollectionConfiguration());
}
private CollectionConfiguration getCollectionConfiguration() {
CollectionConfiguration colCfg = new CollectionConfiguration();
colCfg.setCollocated(true);
colCfg.setCacheMode(REPLICATED);
colCfg.setAtomicityMode(TRANSACTIONAL);
return colCfg;
}
Recently, we have started receiving the below NPE -
2019-08-09 18:18:39,241 ERROR [Inbound-Main-Pool-13] [TransactionId: e5b5bfe3-5246-4d54-a4d6-acd550240e13 Request ID - 27845] [ APP=AladdinTransferAgentServer, ACTION=TA_SETTLEMENT, USER=tsgops ] SettlementWorkflowProcessor - Error while processing CLIENT settlement
class org.apache.ignite.IgniteException: Failed to serialize object [typeName=LinkedList]
at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:990)
at org.apache.ignite.internal.processors.datastructures.GridCacheQueueAdapter$QueueIterator.remove(GridCacheQueueAdapter.java:687)
at java.util.AbstractCollection.removeAll(AbstractCollection.java:376)
at org.apache.ignite.internal.processors.datastructures.GridCacheQueueProxy.removeAll(GridCacheQueueProxy.java:180)
at com.bfm.app.ta.settlement.service.support.TAOrderSettlementIgniteQueueService.removeAll(TAOrderSettlementIgniteQueueService.java:63)
at com.bfm.app.ta.settlement.service.support.TAOrderContextSettlementInputManager.removeAllFromCurrentProcessing(TAOrderContextSettlementInputManager.java:201)
at com.bfm.app.ta.settlement.service.support.TAOrderContextSettlementInputManager.lambda$removeAll$3(TAOrderContextSettlementInputManager.java:100)
at java.lang.Iterable.forEach(Iterable.java:75)
at com.bfm.app.ta.settlement.service.support.TAOrderContextSettlementInputManager.removeAll(TAOrderContextSettlementInputManager.java:100)
at com.bfm.app.ta.settlement.service.support.TAOrderContextSettlementInputManager.removeAll(TAOrderContextSettlementInputManager.java:90)
at com.bfm.app.ta.settlement.processor.support.SettlementWorkflowProcessor.processOrders(SettlementWorkflowProcessor.java:602)
at com.bfm.app.ta.settlement.processor.support.SettlementWorkflowProcessor.lambda$null$13(SettlementWorkflowProcessor.java:405)
at java.util.HashMap.forEach(HashMap.java:1289)
at com.bfm.app.ta.settlement.processor.support.SettlementWorkflowProcessor.lambda$null$14(SettlementWorkflowProcessor.java:368)
at java.util.HashMap.forEach(HashMap.java:1289)
at com.bfm.app.ta.settlement.processor.support.SettlementWorkflowProcessor.lambda$null$15(SettlementWorkflowProcessor.java:354)
at java.util.HashMap.forEach(HashMap.java:1289)
at com.bfm.app.ta.settlement.processor.support.SettlementWorkflowProcessor.lambda$null$16(SettlementWorkflowProcessor.java:345)
at java.util.HashMap.forEach(HashMap.java:1289)
at com.bfm.app.ta.settlement.processor.support.SettlementWorkflowProcessor.lambda$executeSettlementByClientSettleDateAndCurrency$17(SettlementWorkflowProcessor.java:337)
at java.util.HashMap.forEach(HashMap.java:1289)
at com.bfm.app.ta.settlement.processor.support.SettlementWorkflowProcessor.executeSettlementByClientSettleDateAndCurrency(SettlementWorkflowProcessor.java:330)
at com.bfm.app.ta.settlement.processor.support.SettlementWorkflowProcessor.executeSettlement(SettlementWorkflowProcessor.java:302)
at com.bfm.app.ta.settlement.processor.support.SettlementWorkflowProcessor.lambda$processSettlementFromQueue$6(SettlementWorkflowProcessor.java:282)
at com.bfm.app.ta.settlement.locking.support.IgniteLockingService.execute(IgniteLockingService.java:39)
at com.bfm.app.ta.settlement.locking.support.IgniteLockingService.execute(IgniteLockingService.java:68)
at com.bfm.app.ta.settlement.processor.support.SettlementWorkflowProcessor.processSettlementFromQueue(SettlementWorkflowProcessor.java:281)
at com.bfm.app.ta.settlement.facade.listener.support.TASettlementEventListener.listen(TASettlementEventListener.java:49)
at com.bfm.app.ta.settlement.facade.listener.support.TASettlementEventListener.listen(TASettlementEventListener.java:19)
at com.bfm.app.ta.common.listener.support.AbstractEventListener.onMessage(AbstractEventListener.java:44)
at com.bfm.app.ta.common.listener.support.AbstractEventListener$$FastClassBySpringCGLIB$$f1379f74.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.bfm.app.ta.common.aop.aspect.MetricsTrackingAspect.trackMetrics(MetricsTrackingAspect.java:142)
at com.bfm.app.ta.common.aop.aspect.MetricsTrackingAspect.logMetricsForListener(MetricsTrackingAspect.java:71)
at sun.reflect.GeneratedMethodAccessor657.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.bfm.app.ta.core.aop.aspect.LogContextAspect.logMetricsForListener(LogContextAspect.java:43)
at sun.reflect.GeneratedMethodAccessor596.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:48)
at com.ryantenney.metrics.spring.TimedMethodInterceptor.invoke(TimedMethodInterceptor.java:34)
at com.ryantenney.metrics.spring.AbstractMetricMethodInterceptor.invoke(AbstractMetricMethodInterceptor.java:59)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.bfm.app.ta.settlement.facade.listener.support.TASettlementEventListener$$EnhancerBySpringCGLIB$$de3aafae.onMessage(<generated>)
at com.bfm.beam.listeners.BeamListenerSupport.fireIncomingMessageToListener(BeamListenerSupport.java:140)
at com.bfm.beam.listeners.BeamListenerSupport.fireIncomingMessage(BeamListenerSupport.java:103)
at com.bfm.beam.listeners.BeamListenerSupport.fireOnIncomingMessage(BeamListenerSupport.java:84)
at com.bfm.beam.SimpleBeamConnectionDriver$2.run(SimpleBeamConnectionDriver.java:326)
at com.bfm.util.TransactionId$TransactionRunnable.run(TransactionId.java:116)
at com.bfm.beam.executor.BeamWorkItem.run(BeamWorkItem.java:41)
at com.bfm.beam.executor.BeamWorkerImpl.run(BeamWorkerImpl.java:109)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to serialize object [typeName=LinkedList]
at org.apache.ignite.internal.processors.datastructures.GridTransactionalCacheQueueImpl.removeItem(GridTransactionalCacheQueueImpl.java:211)
at org.apache.ignite.internal.processors.datastructures.GridCacheQueueAdapter$QueueIterator.remove(GridCacheQueueAdapter.java:682)
... 67 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to serialize object [typeName=LinkedList]
at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10050)
at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10108)
at org.apache.ignite.internal.processors.cache.GridCacheUtils.marshal(GridCacheUtils.java:780)
at org.apache.ignite.internal.processors.cache.GridCacheUtils.marshal(GridCacheUtils.java:753)
at org.apache.ignite.internal.processors.cache.transactions.IgniteTxEntry.marshal(IgniteTxEntry.java:902)
at org.apache.ignite.internal.processors.cache.GridCacheMessage.marshalTx(GridCacheMessage.java:372)
at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxPrepareRequest.prepareMarshal(GridDistributedTxPrepareRequest.java:389)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onSend(GridCacheIoManager.java:1114)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1148)
at org.apache.ignite.internal.processors.cache.distributed.near.GridNearPessimisticTxPrepareFuture.preparePessimistic(GridNearPessimisticTxPrepareFuture.java:366)
at org.apache.ignite.internal.processors.cache.distributed.near.GridNearPessimisticTxPrepareFuture.prepare(GridNearPessimisticTxPrepareFuture.java:176)
at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareNearTxLocal(GridNearTxLocal.java:3273)
at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commitNearTxLocalAsync(GridNearTxLocal.java:3333)
at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commit(GridNearTxLocal.java:3300)
at org.apache.ignite.internal.processors.datastructures.GridTransactionalCacheQueueImpl$4.call(GridTransactionalCacheQueueImpl.java:200)
at org.apache.ignite.internal.processors.cache.GridCacheUtils.retryTopologySafe(GridCacheUtils.java:1479)
at org.apache.ignite.internal.processors.datastructures.GridTransactionalCacheQueueImpl.removeItem(GridTransactionalCacheQueueImpl.java:189)
... 68 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to serialize object [typeName=LinkedList]
at org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:835)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:206)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:147)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:134)
at org.apache.ignite.internal.binary.GridBinaryMarshaller.marshal(GridBinaryMarshaller.java:251)
at org.apache.ignite.internal.binary.BinaryMarshaller.marshal0(BinaryMarshaller.java:82)
at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10044)
... 84 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to serialize object [typeName=T2]
at org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:835)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:206)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:147)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:134)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteObject(BinaryWriterExImpl.java:496)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteCollection(BinaryWriterExImpl.java:764)
at org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:707)
... 91 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to write field [name=val1]
at org.apache.ignite.internal.binary.BinaryFieldAccessor.write(BinaryFieldAccessor.java:159)
at org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:807)
... 97 more
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to marshal object with optimized marshaller: org.apache.ignite.internal.processors.datastructures.GridCacheQueueAdapter$RemoveProcessor#6d82c986
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:186)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:147)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:134)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteObject(BinaryWriterExImpl.java:496)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObjectField(BinaryWriterExImpl.java:1160)
at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.write0(BinaryFieldAccessor.java:682)
at org.apache.ignite.internal.binary.BinaryFieldAccessor.write(BinaryFieldAccessor.java:155)
... 98 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to serialize object: org.apache.ignite.internal.processors.datastructures.GridCacheQueueAdapter$RemoveProcessor#6d82c986
at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.marshal0(OptimizedMarshaller.java:206)
at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10044)
at org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:179)
... 104 more
Caused by: java.lang.NullPointerException
at org.apache.ignite.internal.processors.datastructures.GridCacheQueueAdapter$RemoveProcessor.writeExternal(GridCacheQueueAdapter.java:1021)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeExternalizable(OptimizedObjectOutputStream.java:317)
at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.write(OptimizedClassDescriptor.java:814)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObject0(OptimizedObjectOutputStream.java:240)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectOutputStream.writeObjectOverride(OptimizedObjectOutputStream.java:158)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.marshal0(OptimizedMarshaller.java:201)
... 107 more
P.S. We don't have any class T2 with field val1
Also, the error is not reproducible locally nor on environment on demand. Also, it occurs only once and then every subsequent calls to remove from queue work fine.
Can someone please help with this issue?
Thanks.
We have an application that reads RabbitMQ messages from a number of queues using Spring AMPQ and a RabbitListener. Something like this
#RabbitListener(queues = {"#{'${rabbit.queues}'.split(',')}"})
public void processRabbitMessage(#Payload String data, #Header(AmqpHeaders.CONSUMER_QUEUE) String queue, #Header(AmqpHeaders.MESSAGE_ID) String messageId) throws Exception {
// Do some stuff
}
However I'm getting an intermittent error where the message header isn't set.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
Bean [com.service.RabbitService#5b7a7f33]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:135)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy110.invokeListener(Unknown Source)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageHandlingException: Missing header 'amqp_consumerQueue' for method parameter type [class java.lang.String]
at org.springframework.messaging.handler.annotation.support.HeaderMethodArgumentResolver.handleMissingValue(HeaderMethodArgumentResolver.java:100)
at org.springframework.messaging.handler.annotation.support.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:103)
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:135)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126)
... 25 common frames omitted
All the messages are written to the queues in exactly the same way (as below) and come from the same source so are generally the same structure.
MessageProperties msgprop = new MessageProperties()
msgprop.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN)
msgprop.setAppId(routing)
rabbitTemplate.send(exchange, routing, new Message(message.getBytes(), msgprop))
We're not loosing data since we can catch all the rejected messages in a DLX but it's difficult to reprocess when we don't know what queue they came from.
We know that none of the messages are null/malformed (even the ones generating errors) and I was under the impression that this was a header set by Rabbit based on the queue a message is written to so how can it be null?
Any ideas as to how this header is null for some and not all messages and/or a possible fix to insure the headers are set?
EDIT: Added some DEBUG logs of the last attempt at reading in the message. We have a stateless retry policy in place so it's tried twice already.
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.retry.support.RetryTemplate - Checking for rethrow: count=3
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.retry.support.RetryTemplate - Retry failed last attempt: count=3
00:46:17.577 [SimpleAsyncTaskExecutor-1051] WARN o.s.a.r.r.RejectAndDontRequeueRecoverer - Retries exhausted for message (Body:'{Message:Body}' MessageProperties [headers={}, timestamp=null, messageId=a.b-209491726, userId=null, receivedUserId=null, appId=a.b, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=text/plain, contentEncoding=null, contentLength=0, deliveryMode=null, receivedDeliveryMode=PERSISTENT, expiration=null, priority=0, redelivered=false, receivedExchange=exch, receivedRoutingKey=a.b, receivedDelay=null, deliveryTag=561, messageCount=0, consumerTag=amq.ctag-fpVPl0EjHwdKew6eTEMoWA, consumerQueue=null])org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]Bean [com.service.RabbitService#5b7a7f33]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:135)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy110.invokeListener(Unknown Source)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageHandlingException: Missing header 'amqp_consumerQueue' for method parameter type [class java.lang.String]
at org.springframework.messaging.handler.annotation.support.HeaderMethodArgumentResolver.handleMissingValue(HeaderMethodArgumentResolver.java:100)
at org.springframework.messaging.handler.annotation.support.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:103)
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:135)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126)
... 25 common frames omitted
00:46:17.577 [SimpleAsyncTaskExecutor-1051] WARN o.s.a.r.l.ConditionalRejectingErrorHandler - Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Retry Policy Exhausted
at org.springframework.amqp.rabbit.retry.RejectAndDontRequeueRecoverer.recover(RejectAndDontRequeueRecoverer.java:45)
at org.springframework.amqp.rabbit.config.StatelessRetryOperationsInterceptorFactoryBean$1.recover(StatelessRetryOperationsInterceptorFactoryBean.java:66)
at org.springframework.amqp.rabbit.config.StatelessRetryOperationsInterceptorFactoryBean$1.recover(StatelessRetryOperationsInterceptorFactoryBean.java:59)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$ItemRecovererCallback.recover(RetryOperationsInterceptor.java:141)
at org.springframework.retry.support.RetryTemplate.handleRetryExhausted(RetryTemplate.java:512)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:350)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy110.invokeListener(Unknown Source)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
Bean [com.service.RabbitService#5b7a7f33]
... 18 common frames omitted
Caused by: org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
Bean [com.service.RabbitService#5b7a7f33]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:135)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
... 12 common frames omitted
Caused by: org.springframework.messaging.MessageHandlingException: Missing header 'amqp_consumerQueue' for method parameter type [class java.lang.String]
at org.springframework.messaging.handler.annotation.support.HeaderMethodArgumentResolver.handleMissingValue(HeaderMethodArgumentResolver.java:100)
at org.springframework.messaging.handler.annotation.support.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:103)
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:135)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126)
... 25 common frames omitted
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.a.r.l.BlockingQueueConsumer - Rejecting messages (requeue=false)
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.a.r.l.SimpleMessageListenerContainer - Cancelling Consumer#2f48a506: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://exch#1.1.1.1//exch,1055), conn: Proxy#79239726 Shared Rabbit Connection: SimpleConnection#7d1c1c5d [delegate=amqp://exch#10.136.229.112:5672//exch, localPort= 45298], acknowledgeMode=AUTO local queue size=0
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.a.r.l.BlockingQueueConsumer - Closing Rabbit Channel: Cached Rabbit Channel: AMQChannel(amqp://exch#10.136.229.112:5672//exch,1055), conn: Proxy#79239726 Shared Rabbit Connection: SimpleConnection#7d1c1c5d [delegate=amqp://exch#10.136.229.112:5672//exch, localPort= 45298]
I don't see how that's possible; we unconditionally set the consumerQueue property in the received message - which is the source for that header. We maintain a map of consumerTag -> queue name for this purpose.
A DEBUG log for a message delivery that exhibits this behavior would be useful.
I was use sparkContext.broadcast in my spark streaming application for share redis connection pool(JedisPool).
The code like this:
lazy val redisPool = {
val pool = Redis.createRedisPool(redisHost, redisPort)
ssc.sparkContext.broadcast(pool)
}
Redis.createRedisPool is:
object Redis {
def createRedisPool(host: String, port: Int, maxIdle: Int, maxTotal: Int, timeout: Int): JedisPool = {
val pc = new JedisPoolConfig
pc.setMaxIdle(maxIdle)
pc.setMaxTotal(maxTotal)
pc.setMaxWaitMillis(timeout)
new JedisPool(pc, host, port)
}
def createRedisPool(host: String, port: Int): JedisPool = {
createRedisPool(host, port, 5, 5, 5000)
}
}
It works at local deploy mode, but when I run this at yarn/standalone mode like
spark-submit --master "yarn-client" --class ...
will get an error:
Exception in thread "main" java.io.NotSerializableException: redis.clients.jedis.JedisPool
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1165)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:210)
at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:83)
at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:68)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:36)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:809)
at org.culiu.bd.streaming.AdSysStreaming$.redisPool$lzycompute$1(AdSysStreaming.scala:84)
at org.culiu.bd.streaming.AdSysStreaming$.redisPool$1(AdSysStreaming.scala:82)
at org.culiu.bd.streaming.AdSysStreaming$.main(AdSysStreaming.scala:154)
at org.culiu.bd.streaming.AdSysStreaming.main(AdSysStreaming.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
I have tried set spark.serializer = org.apache.spark.serializer.KryoSerializer in my application, and then got error like:
Exception in thread "main" com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
Serialization trace:
classes (sun.misc.Launcher$AppClassLoader)
classloader (java.security.ProtectionDomain)
context (java.security.AccessControlContext)
acc (org.apache.spark.executor.ExecutorURLClassLoader)
factoryClassLoader (org.apache.commons.pool2.impl.GenericObjectPool)
internalPool (redis.clients.jedis.JedisPool)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:549)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:570)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:119)
at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:210)
at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:83)
at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:68)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:36)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:809)
at org.culiu.bd.streaming.AdSysStreaming$.redisPool$lzycompute$1(AdSysStreaming.scala:85)
at org.culiu.bd.streaming.AdSysStreaming$.redisPool$1(AdSysStreaming.scala:83)
at org.culiu.bd.streaming.AdSysStreaming$.main(AdSysStreaming.scala:155)
at org.culiu.bd.streaming.AdSysStreaming.main(AdSysStreaming.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:74)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
... 39 more
how can I solve this?
It looks like the problem here is that the redis.clients.jedis.JedisPool class is not serializable. This doesn't seem like a Spark-specific issue, since I think that any attempt to serialize that class would fail.
I ran into a strange problem. We have to poll a mailbox every 10 minutes and check for new mails (POP3, IMAP). We tested our implementation with several mail providers (gmail, 1and1, web.de, gmx, ...) and all worked fine.
Now we have to use a mailbox from outlook.office365.com with POP3 and SSL. Starting our service everything is working, but after a few hours the mailbox check throws errors while opening the mailbox store.
javax.mail.MessagingException: Connect failed;
nested exception is:
javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair
at com.sun.mail.pop3.POP3Store.protocolConnect(POP3Store.java:210)
at javax.mail.Service.connect(Service.java:295)
at com.heiler.hbc.sil.mail.internal.MailerServiceImpl.createStore(MailerServiceImpl.java:298)
at com.heiler.hbc.sil.mail.internal.MailerServiceImpl.retrieveMails(MailerServiceImpl.java:499)
at com.heiler.hbc.sil.mail.internal.MailerServiceImpl.accessMailbox(MailerServiceImpl.java:180)
at sun.reflect.GeneratedMethodAccessor1272.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58)
at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:132)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:120)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56)
at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:132)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:120)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy323.accessMailbox(Unknown Source)
at com.heiler.hbc.sil.pinquiry.internal.ReceiveResponseService.service(ReceiveResponseService.java:103)
at com.heiler.hbc.sil.pinquiry.internal.ReceiveResponseService.service(ReceiveResponseService.java:34)
at com.heiler.hbc.service.executor.internal.PInquiryCheckMailboxService.execute(PInquiryCheckMailboxService.java:110)
at com.heiler.hbc.service.executor.internal.AbstractSchedulerService.run(AbstractSchedulerService.java:61)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:53)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1842)
at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1825)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1346)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at com.sun.mail.util.SocketFetcher.configureSSLSocket(SocketFetcher.java:548)
at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:352)
at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:207)
at com.sun.mail.pop3.Protocol.<init>(Protocol.java:111)
at com.sun.mail.pop3.POP3Store.getPort(POP3Store.java:261)
at com.sun.mail.pop3.POP3Store.protocolConnect(POP3Store.java:206)
... 37 more
Caused by: java.lang.RuntimeException: Could not generate DH keypair
at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:80)
at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:632)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:218)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
... 44 more
Caused by: java.security.InvalidAlgorithmParameterException: parameter object not a ECParameterSpec
at org.bouncycastle.jce.provider.JDKKeyPairGenerator$EC.initialize(Unknown Source)
at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:75)
... 51 more
This is my createStore method:
private void createStore() throws MessagingException
{
if ( store == null )
{
store = session.getStore( authType.getType() );
}
if ( store != null && !store.isConnected() )
{
try
{
store.connect( popHost, popPort, user, password );
}
catch ( MessagingException e )
{
store = null;
logger.error( "Cannot connect to configured mailbox: [Host: " + popHost + ", Port: " + popPort + ", User: " //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$
+ user + "]" ); //$NON-NLS-1$
logger.debug( "", e );
}
}
}
On store.connect(..) the error occurs.
Anyone out there with an idea what can couse this error?
Thanks
Stephan
Updating the BouncyCastle dependencies solved the problem.
It looks like the BouncyCastle provider has a problem:
Caused by: java.security.InvalidAlgorithmParameterException: parameter object not a ECParameterSpec
at org.bouncycastle.jce.provider.JDKKeyPairGenerator$EC.initialize(Unknown Source)
Is your program doing anything to change the BouncyCastle configuration while it's running? If not, perhaps it's a bug in BouncyCastle? Otherwise, I don't know what would cause it to work for awhile and then fail.
Check if you have an outdated org.bouncycastle lib in your classpath (incompatible with your java version).
See http://iwang.github.io/support/2014/03/14/cxf-cause-https-error.html for an in-depth explanation.