I have a Springboot Kotlin app running in AWS ECS/EC2 with some code written by my predecessor. It runs into OutOfMemoryError after running a while in the server. I am not familiar with ReactiveX and threading, so would like to ask for some help on this.
I see from the error logs that it should be related to the Observable.blockingForEach method?
The listOfKeys is a limited source of 5 to 50 items, so it should not be due to backpressure?
I also pasted the thread dump when OutOfMemoryError occurred.
Code inside myMethod
Observable.fromIterable(listOfKeys)
.flatMap { key ->
Maybe.fromCallable {
Pair(key, methodWhichCallsExternalAPI))
}
.toObservable()
.subscribeOn(Schedulers.from(Executors.newFixedThreadPool(4)))
}
.blockingForEach {
// some simple assignment of result to variables
}
Exception logs
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at io.reactivex.internal.schedulers.ExecutorScheduler.scheduleDirect(ExecutorScheduler.java:59)
at io.reactivex.internal.operators.observable.ObservableSubscribeOn.subscribeActual(ObservableSubscribeOn.java:36)
at io.reactivex.Observable.subscribe(Observable.java:12284)
at io.reactivex.internal.operators.observable.ObservableMap.subscribeActual(ObservableMap.java:32)
at io.reactivex.Observable.subscribe(Observable.java:12284)
at io.reactivex.internal.operators.observable.ObservableFlatMap$MergeObserver.subscribeInner(ObservableFlatMap.java:165)
at io.reactivex.internal.operators.observable.ObservableFlatMap$MergeObserver.onNext(ObservableFlatMap.java:139)
at io.reactivex.internal.operators.observable.ObservableFromIterable$FromIterableDisposable.run(ObservableFromIterable.java:98)
at io.reactivex.internal.operators.observable.ObservableFromIterable.subscribeActual(ObservableFromIterable.java:58)
at io.reactivex.Observable.subscribe(Observable.java:12284)
at io.reactivex.internal.operators.observable.ObservableFlatMap.subscribeActual(ObservableFlatMap.java:55)
at io.reactivex.Observable.subscribe(Observable.java:12284)
at io.reactivex.internal.operators.observable.BlockingObservableIterable.iterator(BlockingObservableIterable.java:39)
at io.reactivex.Observable.blockingForEach(Observable.java:5183)
at com.myOrg.MyController.myMethod(MyController.kt:240)
at com.myOrg.MyController$$FastClassBySpringCGLIB$$38688575.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:779)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:750)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:692)
at com.dbs.localisemanagement.localise.LocaliseController$$EnhancerBySpringCGLIB$$3e102d4f.lokaliseProjectsS3(<generated>)
at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)
Thread dump - each of the 3 parts are repeated many times
at sun.misc.Unsafe.park(Native Method)
"pool-30968-thread-1" Id=36794 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#394a0729
at java.lang.Thread.run(Thread.java:748)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#f0c2d74
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
- locked java.lang.Object#5e78eb81
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.socketRead0(Native Method)
"pool-30982-thread-1" Id=36808 RUNNABLE
- java.util.concurrent.ThreadPoolExecutor$Worker#7d90150f
Number of locked synchronizers = 1
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:306)
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:198)
- locked org.apache.http.pool.AbstractConnPool$2#a0abf72
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:253)
at org.apache.http.pool.AbstractConnPool.access$300(AbstractConnPool.java:70)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:393)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#7399fe63
at sun.misc.Unsafe.park(Native Method)
"pool-30987-thread-1" Id=36813 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#7399fe63
- java.util.concurrent.ThreadPoolExecutor$Worker#5b4dcfd1
Number of locked synchronizers = 1
The most likely problem is the repeated creation of a thread pool that is not released after use.
.subscribeOn(Schedulers.from(Executors.newFixedThreadPool(4)))
The more the method runs the more abandoned threads it creates.
You should probably create a static Scheduler and use that:
static final ExecutorService pool =
Executors.newFixedThreadPool(4);
static final Scheduler scheduler = Schedulers.from(pool);
Later on, after the app's lifecycle ended, call pool.shutdown() to free the threads.
E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.example.poster, PID: 23677
java.lang.RuntimeException: Unable to start activity ComponentInfo{com.example.poster/com.example.poster.MainActivity}: com.google.firebase.database.DatabaseException: Failed to get FirebaseDatabase instance: Specify DatabaseURL within FirebaseApp or from your getInstance() call.
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2984)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3045)
at android.app.ActivityThread.-wrap14(ActivityThread.java)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1642)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:154)
at android.app.ActivityThread.main(ActivityThread.java:6776)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1496)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1386)
Caused by: com.google.firebase.database.DatabaseException: Failed to get FirebaseDatabase instance: Specify DatabaseURL within FirebaseApp or from your getInstance() call.
at com.google.firebase.database.FirebaseDatabase.getInstance(com.google.firebase:firebase-database##16.0.4:114)
at com.google.firebase.database.FirebaseDatabase.getInstance(com.google.firebase:firebase-database##16.0.4:71)
at com.example.poster.MainActivity.onCreate(MainActivity.java:48)
at android.app.Activity.performCreate(Activity.java:6955)
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1126)
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2927)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3045)
at android.app.ActivityThread.-wrap14(ActivityThread.java)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1642)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:154)
at android.app.ActivityThread.main(ActivityThread.java:6776)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1496)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1386)
add these dipendencies for java
implementation platform('com.google.firebase:firebase-bom:28.4.2')
implementation 'com.google.firebase:firebase-database'
for kotlin
implementation platform('com.google.firebase:firebase-bom:28.4.2')
implementation 'com.google.firebase:firebase-database-ktx'
then if using instance outside us1 for the database add url
FirebaseDatabase database = FirebaseDatabase.getInstance("**url here**);
DatabaseReference myRef = database.getReference("**path here**");
We have an application that reads RabbitMQ messages from a number of queues using Spring AMPQ and a RabbitListener. Something like this
#RabbitListener(queues = {"#{'${rabbit.queues}'.split(',')}"})
public void processRabbitMessage(#Payload String data, #Header(AmqpHeaders.CONSUMER_QUEUE) String queue, #Header(AmqpHeaders.MESSAGE_ID) String messageId) throws Exception {
// Do some stuff
}
However I'm getting an intermittent error where the message header isn't set.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
Bean [com.service.RabbitService#5b7a7f33]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:135)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy110.invokeListener(Unknown Source)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageHandlingException: Missing header 'amqp_consumerQueue' for method parameter type [class java.lang.String]
at org.springframework.messaging.handler.annotation.support.HeaderMethodArgumentResolver.handleMissingValue(HeaderMethodArgumentResolver.java:100)
at org.springframework.messaging.handler.annotation.support.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:103)
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:135)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126)
... 25 common frames omitted
All the messages are written to the queues in exactly the same way (as below) and come from the same source so are generally the same structure.
MessageProperties msgprop = new MessageProperties()
msgprop.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN)
msgprop.setAppId(routing)
rabbitTemplate.send(exchange, routing, new Message(message.getBytes(), msgprop))
We're not loosing data since we can catch all the rejected messages in a DLX but it's difficult to reprocess when we don't know what queue they came from.
We know that none of the messages are null/malformed (even the ones generating errors) and I was under the impression that this was a header set by Rabbit based on the queue a message is written to so how can it be null?
Any ideas as to how this header is null for some and not all messages and/or a possible fix to insure the headers are set?
EDIT: Added some DEBUG logs of the last attempt at reading in the message. We have a stateless retry policy in place so it's tried twice already.
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.retry.support.RetryTemplate - Checking for rethrow: count=3
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.retry.support.RetryTemplate - Retry failed last attempt: count=3
00:46:17.577 [SimpleAsyncTaskExecutor-1051] WARN o.s.a.r.r.RejectAndDontRequeueRecoverer - Retries exhausted for message (Body:'{Message:Body}' MessageProperties [headers={}, timestamp=null, messageId=a.b-209491726, userId=null, receivedUserId=null, appId=a.b, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=text/plain, contentEncoding=null, contentLength=0, deliveryMode=null, receivedDeliveryMode=PERSISTENT, expiration=null, priority=0, redelivered=false, receivedExchange=exch, receivedRoutingKey=a.b, receivedDelay=null, deliveryTag=561, messageCount=0, consumerTag=amq.ctag-fpVPl0EjHwdKew6eTEMoWA, consumerQueue=null])org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]Bean [com.service.RabbitService#5b7a7f33]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:135)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy110.invokeListener(Unknown Source)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageHandlingException: Missing header 'amqp_consumerQueue' for method parameter type [class java.lang.String]
at org.springframework.messaging.handler.annotation.support.HeaderMethodArgumentResolver.handleMissingValue(HeaderMethodArgumentResolver.java:100)
at org.springframework.messaging.handler.annotation.support.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:103)
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:135)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126)
... 25 common frames omitted
00:46:17.577 [SimpleAsyncTaskExecutor-1051] WARN o.s.a.r.l.ConditionalRejectingErrorHandler - Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Retry Policy Exhausted
at org.springframework.amqp.rabbit.retry.RejectAndDontRequeueRecoverer.recover(RejectAndDontRequeueRecoverer.java:45)
at org.springframework.amqp.rabbit.config.StatelessRetryOperationsInterceptorFactoryBean$1.recover(StatelessRetryOperationsInterceptorFactoryBean.java:66)
at org.springframework.amqp.rabbit.config.StatelessRetryOperationsInterceptorFactoryBean$1.recover(StatelessRetryOperationsInterceptorFactoryBean.java:59)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$ItemRecovererCallback.recover(RetryOperationsInterceptor.java:141)
at org.springframework.retry.support.RetryTemplate.handleRetryExhausted(RetryTemplate.java:512)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:350)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy110.invokeListener(Unknown Source)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
Bean [com.service.RabbitService#5b7a7f33]
... 18 common frames omitted
Caused by: org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
Bean [com.service.RabbitService#5b7a7f33]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:135)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
... 12 common frames omitted
Caused by: org.springframework.messaging.MessageHandlingException: Missing header 'amqp_consumerQueue' for method parameter type [class java.lang.String]
at org.springframework.messaging.handler.annotation.support.HeaderMethodArgumentResolver.handleMissingValue(HeaderMethodArgumentResolver.java:100)
at org.springframework.messaging.handler.annotation.support.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:103)
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:135)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126)
... 25 common frames omitted
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.a.r.l.BlockingQueueConsumer - Rejecting messages (requeue=false)
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.a.r.l.SimpleMessageListenerContainer - Cancelling Consumer#2f48a506: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://exch#1.1.1.1//exch,1055), conn: Proxy#79239726 Shared Rabbit Connection: SimpleConnection#7d1c1c5d [delegate=amqp://exch#10.136.229.112:5672//exch, localPort= 45298], acknowledgeMode=AUTO local queue size=0
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.a.r.l.BlockingQueueConsumer - Closing Rabbit Channel: Cached Rabbit Channel: AMQChannel(amqp://exch#10.136.229.112:5672//exch,1055), conn: Proxy#79239726 Shared Rabbit Connection: SimpleConnection#7d1c1c5d [delegate=amqp://exch#10.136.229.112:5672//exch, localPort= 45298]
I don't see how that's possible; we unconditionally set the consumerQueue property in the received message - which is the source for that header. We maintain a map of consumerTag -> queue name for this purpose.
A DEBUG log for a message delivery that exhibits this behavior would be useful.
I have 5 different Jobs, some of them have an InputFormat and others use env.fromElements(...)
When I execute the Jobs using IntelliJ, all of them works correctly. But when I execute them from terminal, only jobs which contains env.fromElements() works.
Here is the error message:
Cluster configuration: Standalone cluster with JobManager at localhost/127.0.0.1:6123
Using address localhost:6123 to connect to JobManager.
JobManager web interface address http://localhost:8081
Starting execution of program
Submitting job with JobID: ab41f30df416c654b406c9b13e70f62e. Waiting for job completion.
Connected to JobManager at Actor[akka.tcp://flink#localhost:6123/user/jobmanager#1320020477]
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Cannot initialize task 'CHAIN DataSource (at main(JobSource.java:49) (es.mypackage.flink.Sources.MyInputFormat)) -> Map (Map at main(JobSource.java:53))': Configuring the InputFormat (es.mypackage.flink.Sources.MyInputFormat#1f81aa00) failed: null
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:101)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:400)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:387)
at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:926)
at es.mypackage.flink.Job.JobSource.main(JobSource.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:339)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:831)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:256)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1073)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1120)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1117)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1116)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'CHAIN DataSource (at main(JobSource.java:49) (es.mypackage.flink.Sources.MyInputFormat)) -> Map (Map at main(JobSource.java:53))': Configuring the InputFormat (es.mypackage.flink.Sources.MyInputFormat#1f81aa00) failed: null
at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:136)
at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1274)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:477)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:118)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Configuring the InputFormat (es.mypackage.flink.Sources.MyInputFormat#1f81aa00) failed: null
at org.apache.flink.runtime.jobgraph.InputFormatVertex.initializeOnMaster(InputFormatVertex.java:89)
at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:133)
... 20 more
Any help is much appreciated.
Edit:
public void configure(Configuration parameters) {
/*This line causes the problem*/
this.client = new KuduClient.KuduClientBuilder(KUDU_MASTER).build();
table = createTable(TABLE_NAME);
if (table != null) {
scanner = client.newScannerBuilder(table)
.setProjectedColumnNames(projectColumns)
.build();
}
}
I'm beginner in java so need help.
I have a javafx app that works with Apache POI (for Excel files).
When I click menubutton I have an Error in Controller class which should load Excel data in Map collection and then show new Stage window.
But I have a strange error "Not on FX application thread; currentThread = JavaFX Application Thread" when press the button.
THere is a list of red masseges in output window with such main titles:
Exception in thread "JavaFX Application Thread"
java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.IllegalStateException: Not on FX application thread;
currentThread = JavaFX Application Thread
Here is the code for Method in Controller
public void SKULoading(ActionEvent ae) {
Stage stage = (Stage) menuBar.getScene().getWindow();
stage.hide();
XSSFWorkbook wb = ExcelModel.OpenFile(stage); //using Filechooser
ExcelModel book = new ExcelModel(wb);
Map<Integer, List<String>> data = book.SheetToArray(0);
book.WBclose();
Stage primarystage;
// Since that moment error occurs!
primarystage = new Stage();
FXMLLoader loader = new FXMLLoader();
Pane root = null;
try {
root = loader.load(getClass().getResource("/FProject/SKUmanager.fxml").openStream());
} catch (IOException ex) {
Logger.getLogger(MainWindowController.class.getName()).log(Level.SEVERE, null, ex);
}
Scene scene = new Scene(root);
primarystage.setScene(scene);
primarystage.show();
}
I saw this topic How to avoid Not on FX application thread; currentThread = JavaFX Application Thread error? but solution like "Platform.runLater(new Runnable(){});" seems to solve the problem if I wrap in it almost all code in the Method. but I'm not sure it is right desicion because I didn't create any new Thread before.
When I removed row stage.hide(); new Stage was shown
Or When I left row stage.hide(); and removed rows
XSSFWorkbook wb = ExcelModel.OpenFile(stage);
ExcelModel book = new ExcelModel(wb);
Map<Integer, List<String>> data = book.SheetToArray(0);
book.WBclose();
Stage was shown again.
When I replace code
FileChooser fileChooser = new FileChooser();
fileChooser.setTitle("Open Resource File");
File file = fileChooser.showOpenDialog(stage);
in method ExcelModel.OpenFile(stage); by this
File file = new File("C:\\Users\\Maks\\Desktop\\Test.xlsx");
Stage was also shown.
So, I think, answer is lying near creating/hiding Stages or using FileChooser (Looks like using Filechooser somehow influences on the appearence of a new Stage after privious Stage was hidden)
UPD
Exception in thread "JavaFX Application Thread" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at javafx.fxml.FXMLLoader$MethodHandler.invoke(FXMLLoader.java:1774)
at javafx.fxml.FXMLLoader$ControllerMethodEventHandler.handle(FXMLLoader.java:1657)
at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86)
at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)
at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)
at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:49)
at javafx.event.Event.fireEvent(Event.java:198)
at javafx.scene.control.MenuItem.fire(MenuItem.java:462)
at com.sun.javafx.scene.control.skin.ContextMenuContent$MenuItemContainer.doSelect(ContextMenuContent.java:1405)
at com.sun.javafx.scene.control.skin.ContextMenuContent$MenuItemContainer.lambda$createChildren$343(ContextMenuContent.java:1358)
at com.sun.javafx.event.CompositeEventHandler$NormalEventHandlerRecord.handleBubblingEvent(CompositeEventHandler.java:218)
at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:80)
at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)
at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)
at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)
at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:54)
at javafx.event.Event.fireEvent(Event.java:198)
at javafx.scene.Scene$MouseHandler.process(Scene.java:3757)
at javafx.scene.Scene$MouseHandler.access$1500(Scene.java:3485)
at javafx.scene.Scene.impl_processMouseEvent(Scene.java:1762)
at javafx.scene.Scene$ScenePeerListener.mouseEvent(Scene.java:2494)
at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:352)
at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:275)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$354(GlassViewEventHandler.java:388)
at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:389)
at com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(GlassViewEventHandler.java:387)
at com.sun.glass.ui.View.handleMouseEvent(View.java:555)
at com.sun.glass.ui.View.notifyMouse(View.java:937)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$148(WinApplication.java:191)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at javafx.fxml.FXMLLoader$MethodHandler.invoke(FXMLLoader.java:1769)
... 43 more
Caused by: java.lang.IllegalStateException: Not on FX application thread; currentThread = JavaFX Application Thread
at com.sun.javafx.tk.Toolkit.checkFxUserThread(Toolkit.java:236)
at com.sun.javafx.tk.quantum.QuantumToolkit.checkFxUserThread(QuantumToolkit.java:423)
at javafx.stage.Stage.<init>(Stage.java:241)
at javafx.stage.Stage.<init>(Stage.java:227)
at FProject.MainWindowController.SKULoading(MainWindowController.java:124)
... 53 more
Help me to find out what is the root of problem to avoid such errors in future. Thanks!