I am loading data from kafka to ignite data streamers in parallel in multiple ignite nodes. But when I kill one of these nodes(for checking fault tolerance), I am getting following error. Some Messages are also getting lost.
class org.apache.ignite.IgniteCheckedException: Failed to finish operation (too many remaps): 32
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:863)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:828)
at org.apache.ignite.internal.util.future.GridFutureAdapter$ArrayListener.apply(GridFutureAdapter.java:456)
at org.apache.ignite.internal.util.future.GridFutureAdapter$ArrayListener.apply(GridFutureAdapter.java:439)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:259)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:389)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:355)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:343)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.submit(DataStreamerImpl.java:1716)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.update(DataStreamerImpl.java:1416)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.load0(DataStreamerImpl.java:932)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addData(DataStreamerImpl.java:576)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addData(DataStreamerImpl.java:544)
at org.apache.ignite.stream.StreamAdapter.addMessage(StreamAdapter.java:184)
at org.apache.ignite.stream.kafka.KafkaStreamer.access$100(KafkaStreamer.java:47)
at org.apache.ignite.stream.kafka.KafkaStreamer$1.run(KafkaStreamer.java:156)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send request (node has left): 0a04411f-a797-4526-826a-96c52549b0d0
... 11 more
[21:45:24] (err) Failed to execute compound future reducer: GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=1, done=false, cancelled=false, err=null, futs=[false, false, true, false, false, true, false, false]]class org.apache.ignite.IgniteCheckedException: DataStreamer request failed [node=f48edaa8-74df-4503-bf9b-1fdc6d642744]
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.onResponse(DataStreamerImpl.java:1777)
at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$3.onMessage(DataStreamerImpl.java:335)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1215)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:843)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
at org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:783)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Related
InterProcessMutex zLock=new InterProcessMutex(curatorFramework,lockPath);
zlock.acquire(time,unit);
just a simple example. I try to test the Network jitter problem, so I start the service,connect zookeeper and start the curator client, and then disable the network.
after that, I create a request which would call the zlock.acquire(time,unit).
It shows that the request return error response at least 1min later.
to debug the codes,the program stop in "createsTheLock" method.
In fact, I guess the expectation is to return false when zookeeper is not available.
Thanks for answerring the question!
update the question:
It's the base setting:
RetryPolicy retryPolicy=new ExponentialBackoffRetry(1000,3);
CuratorFramework curatorFramework= CuratorFrameworkFactory.builder().connectionTimeoutMs(15000).sessionTimeoutMs(60000).connectString(CommonConstant.ZK_CONNECT_HOSTS).retryPolicy(retryPolicy).build();
and it's the log:
2020-08-09 10:27:12.582 INFO 91181 --- [224.71.42:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server 10.224.71.42/10.224.71.42:2181. Will not attempt to authenticate using SASL (unknown error)
2020-08-09 10:27:12.582 ERROR 91181 --- [224.71.42:2181)] o.apache.zookeeper.ClientCnxnSocketNIO : Unable to open socket to 10.224.71.42/10.224.71.42:2181
2020-08-09 10:27:12.582 INFO 91181 --- [224.71.42:2181)] org.apache.zookeeper.ClientCnxn : Socket error occurred: 10.224.71.42/10.224.71.42:2181: Network is unreachable
2020-08-09 10:27:13.456 ERROR 91181 --- [tor-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl : Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[zookeeper-3.5.6.jar:3.5.6]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:851) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:999) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:952) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:342) [curator-framework-5.1.0.jar:5.1.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_241]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_241]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_241]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_241]
2020-08-09 10:27:13.457 ERROR 91181 --- [tor-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl : Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:981) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:952) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:342) [curator-framework-5.1.0.jar:5.1.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_241]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_241]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_241]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_241]
2020-08-09 10:27:14.684 INFO 91181 --- [26.102.59:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server 10.226.102.59/10.226.102.59:2181. Will not attempt to authenticate using SASL (unknown error)
2020-08-09 10:27:14.684 ERROR 91181 --- [26.102.59:2181)] o.apache.zookeeper.ClientCnxnSocketNIO : Unable to open socket to 10.226.102.59/10.226.102.59:2181
2020-08-09 10:27:14.684 INFO 91181 --- [26.102.59:2181)] org.apache.zookeeper.ClientCnxn : Socket error occurred: 10.226.102.59/10.226.102.59:2181: Network is unreachable
The request just hang up until 59s later, then return response;
We have an application that reads RabbitMQ messages from a number of queues using Spring AMPQ and a RabbitListener. Something like this
#RabbitListener(queues = {"#{'${rabbit.queues}'.split(',')}"})
public void processRabbitMessage(#Payload String data, #Header(AmqpHeaders.CONSUMER_QUEUE) String queue, #Header(AmqpHeaders.MESSAGE_ID) String messageId) throws Exception {
// Do some stuff
}
However I'm getting an intermittent error where the message header isn't set.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
Bean [com.service.RabbitService#5b7a7f33]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:135)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy110.invokeListener(Unknown Source)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageHandlingException: Missing header 'amqp_consumerQueue' for method parameter type [class java.lang.String]
at org.springframework.messaging.handler.annotation.support.HeaderMethodArgumentResolver.handleMissingValue(HeaderMethodArgumentResolver.java:100)
at org.springframework.messaging.handler.annotation.support.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:103)
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:135)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126)
... 25 common frames omitted
All the messages are written to the queues in exactly the same way (as below) and come from the same source so are generally the same structure.
MessageProperties msgprop = new MessageProperties()
msgprop.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN)
msgprop.setAppId(routing)
rabbitTemplate.send(exchange, routing, new Message(message.getBytes(), msgprop))
We're not loosing data since we can catch all the rejected messages in a DLX but it's difficult to reprocess when we don't know what queue they came from.
We know that none of the messages are null/malformed (even the ones generating errors) and I was under the impression that this was a header set by Rabbit based on the queue a message is written to so how can it be null?
Any ideas as to how this header is null for some and not all messages and/or a possible fix to insure the headers are set?
EDIT: Added some DEBUG logs of the last attempt at reading in the message. We have a stateless retry policy in place so it's tried twice already.
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.retry.support.RetryTemplate - Checking for rethrow: count=3
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.retry.support.RetryTemplate - Retry failed last attempt: count=3
00:46:17.577 [SimpleAsyncTaskExecutor-1051] WARN o.s.a.r.r.RejectAndDontRequeueRecoverer - Retries exhausted for message (Body:'{Message:Body}' MessageProperties [headers={}, timestamp=null, messageId=a.b-209491726, userId=null, receivedUserId=null, appId=a.b, clusterId=null, type=null, correlationId=null, correlationIdString=null, replyTo=null, contentType=text/plain, contentEncoding=null, contentLength=0, deliveryMode=null, receivedDeliveryMode=PERSISTENT, expiration=null, priority=0, redelivered=false, receivedExchange=exch, receivedRoutingKey=a.b, receivedDelay=null, deliveryTag=561, messageCount=0, consumerTag=amq.ctag-fpVPl0EjHwdKew6eTEMoWA, consumerQueue=null])org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]Bean [com.service.RabbitService#5b7a7f33]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:135)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy110.invokeListener(Unknown Source)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageHandlingException: Missing header 'amqp_consumerQueue' for method parameter type [class java.lang.String]
at org.springframework.messaging.handler.annotation.support.HeaderMethodArgumentResolver.handleMissingValue(HeaderMethodArgumentResolver.java:100)
at org.springframework.messaging.handler.annotation.support.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:103)
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:135)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126)
... 25 common frames omitted
00:46:17.577 [SimpleAsyncTaskExecutor-1051] WARN o.s.a.r.l.ConditionalRejectingErrorHandler - Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Retry Policy Exhausted
at org.springframework.amqp.rabbit.retry.RejectAndDontRequeueRecoverer.recover(RejectAndDontRequeueRecoverer.java:45)
at org.springframework.amqp.rabbit.config.StatelessRetryOperationsInterceptorFactoryBean$1.recover(StatelessRetryOperationsInterceptorFactoryBean.java:66)
at org.springframework.amqp.rabbit.config.StatelessRetryOperationsInterceptorFactoryBean$1.recover(StatelessRetryOperationsInterceptorFactoryBean.java:59)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$ItemRecovererCallback.recover(RetryOperationsInterceptor.java:141)
at org.springframework.retry.support.RetryTemplate.handleRetryExhausted(RetryTemplate.java:512)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:350)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:115)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy110.invokeListener(Unknown Source)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
Bean [com.service.RabbitService#5b7a7f33]
... 18 common frames omitted
Caused by: org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void com.service.RabbitService.processRabbitMessage(java.lang.String,java.lang.String,java.lang.String) throws java.lang.Exception]
Bean [com.service.RabbitService#5b7a7f33]
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:135)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:106)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
... 12 common frames omitted
Caused by: org.springframework.messaging.MessageHandlingException: Missing header 'amqp_consumerQueue' for method parameter type [class java.lang.String]
at org.springframework.messaging.handler.annotation.support.HeaderMethodArgumentResolver.handleMissingValue(HeaderMethodArgumentResolver.java:100)
at org.springframework.messaging.handler.annotation.support.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:103)
at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:112)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:135)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:107)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:49)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:126)
... 25 common frames omitted
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.a.r.l.BlockingQueueConsumer - Rejecting messages (requeue=false)
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.a.r.l.SimpleMessageListenerContainer - Cancelling Consumer#2f48a506: tags=[{}], channel=Cached Rabbit Channel: AMQChannel(amqp://exch#1.1.1.1//exch,1055), conn: Proxy#79239726 Shared Rabbit Connection: SimpleConnection#7d1c1c5d [delegate=amqp://exch#10.136.229.112:5672//exch, localPort= 45298], acknowledgeMode=AUTO local queue size=0
00:46:17.577 [SimpleAsyncTaskExecutor-1051] DEBUG o.s.a.r.l.BlockingQueueConsumer - Closing Rabbit Channel: Cached Rabbit Channel: AMQChannel(amqp://exch#10.136.229.112:5672//exch,1055), conn: Proxy#79239726 Shared Rabbit Connection: SimpleConnection#7d1c1c5d [delegate=amqp://exch#10.136.229.112:5672//exch, localPort= 45298]
I don't see how that's possible; we unconditionally set the consumerQueue property in the received message - which is the source for that header. We maintain a map of consumerTag -> queue name for this purpose.
A DEBUG log for a message delivery that exhibits this behavior would be useful.
I am getting the below error during the igniteCache.put() in the IgniteRunnable run()
I have only 2 nodes (client and server) .
1) Client creates the cache
CacheConfiguration<Integer, LAttribute> cfg = new CacheConfiguration<Integer, LAttribute>();
cfg.setIndexedTypes(Integer.class, LoanAttribute.class);
cfg.setCacheMode(CacheMode.PARTITIONED);
cfg.setName("inv_result");
cfg.setCopyOnRead(false);
cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
2) Client Submit the IgniteRunnable task to Server
3) Client exit the cluster
On the Server (with in run() method)
1) Get the cache and put a value
IgniteCache<Integer, LAttribute> iCache = Ignition.localIgnite().cache("inv_result");
System.out.println("Begin .. "+iCache.size(CachePeekMode.ALL));
iCache.put(la.getId(), la);
Error :
[21:41:14,859][SEVERE][pub-#67%null%][GridJobWorker] Failed to execute job due to unexpected runtime exception [jobId=f4606f39b51-21c994a7-6b35-49fa-b696-582fa7825c31, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=com.test.ignite.compute.AssetRestrictionComputeJob, dep=GridDeployment [ts=1492836063447, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader#73d16e93, clsLdrId=438a5f39b51-76a937b0-7831-458b-aee4-cec662f02b0d, userVer=0, loc=true, sampleClsName=o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionMap2, pendingUndeploy=false, undeployed=false, usage=1], taskClsName=com.bfm.seclending.ignite.compute.AssetRestrictionComputeJob, sesId=c4606f39b51-21c994a7-6b35-49fa-b696-582fa7825c31, startTime=1492836072790, endTime=9223372036854775807, taskNodeId=21c994a7-6b35-49fa-b696-582fa7825c31, clsLdr=sun.misc.Launcher$AppClassLoader#73d16e93, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false, subjId=21c994a7-6b35-49fa-b696-582fa7825c31, mapFut=IgniteFuture [orig=GridFutureAdapter [resFlag=0, res=null, startTime=1492836072829, endTime=0, ignoreInterrupts=false, state=INIT]]], jobId=f4606f39b51-21c994a7-6b35-49fa-b696-582fa7825c31]]
javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException: Failed to wait for asynchronous operation permit (thread got interrupted).
at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1440)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2183)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1383)
at co.test.ignite.compute.AssetRestrictionComputeJob.run(AssetRestrictionComputeJob.java:110)
at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4V2.execute(GridClosureProcessor.java:2215)
at org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:556)
at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6564)
at org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:550)
at org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:479)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1180)
at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1894)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1082)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:710)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:102)
at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:673)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to wait for asynchronous operation permit (thread got interrupted).
at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:766)
at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:764)
... 19 more
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOpAcquire(GridCacheAdapter.java:4597)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:817)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1148)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:618)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAsync(GridCacheAdapter.java:2541)
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:595)
at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2215)
at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1376)
... 16 more
Most likely server node was stopped in the middle of execution. That's the only case when internal Ignite threads are interrupted. When this happens, job can be automatically failed over to another node: https://apacheignite.readme.io/docs/fault-tolerance
I found the reason for the InterruptedException , executionService that i am using on the client side to submit the jobs is not waiting for the job completion .
when i call future.get() .. all good now. Thanks
I'm still following the still following the Getting Started version 0.9.
When I run the sample application, and I quit it, the following exception are reported in console:
Stopping JavaFX Application "MyApplication 1.0.0-SNAPSHOT"...
Exception in thread "FelixStartLevel" java.lang.IllegalStateException: This operation is permitted on the event thread only; currentThread = FelixStartLevel
at com.sun.glass.ui.Application.checkEventThread(Application.java:443)
at com.sun.glass.ui.MenuItem.setEnabled(MenuItem.java:108)
at com.sun.javafx.tk.quantum.GlassSystemMenu.lambda$insertMenuItem$347(GlassSystemMenu.java:261)
at com.sun.javafx.binding.ExpressionHelper$SingleInvalidation.fireValueChangedEvent(ExpressionHelper.java:137)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:81)
at javafx.beans.property.BooleanPropertyBase.fireValueChangedEvent(BooleanPropertyBase.java:103)
at javafx.beans.property.BooleanPropertyBase.markInvalid(BooleanPropertyBase.java:110)
at javafx.beans.property.BooleanPropertyBase.access$000(BooleanPropertyBase.java:49)
at javafx.beans.property.BooleanPropertyBase$Listener.invalidated(BooleanPropertyBase.java:245)
at com.sun.javafx.binding.ExpressionHelper$Generic.fireValueChangedEvent(ExpressionHelper.java:349)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:81)
at javafx.beans.property.BooleanPropertyBase.fireValueChangedEvent(BooleanPropertyBase.java:103)
at javafx.beans.property.BooleanPropertyBase.markInvalid(BooleanPropertyBase.java:110)
at javafx.beans.property.BooleanPropertyBase.access$000(BooleanPropertyBase.java:49)
at javafx.beans.property.BooleanPropertyBase$Listener.invalidated(BooleanPropertyBase.java:245)
at com.sun.javafx.binding.ExpressionHelper$SingleInvalidation.fireValueChangedEvent(ExpressionHelper.java:137)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:81)
at javafx.beans.binding.BooleanBinding.invalidate(BooleanBinding.java:176)
at com.sun.javafx.binding.BindingHelperObserver.invalidated(BindingHelperObserver.java:51)
at com.sun.javafx.binding.ExpressionHelper$SingleInvalidation.fireValueChangedEvent(ExpressionHelper.java:137)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:81)
at javafx.beans.property.ReadOnlyBooleanPropertyBase.fireValueChangedEvent(ReadOnlyBooleanPropertyBase.java:72)
at org.drombler.commons.action.fx.AbstractFXAction$EnabledProperty.set(AbstractFXAction.java:122)
at org.drombler.commons.action.fx.AbstractFXAction$EnabledProperty.access$100(AbstractFXAction.java:110)
at org.drombler.commons.action.fx.AbstractFXAction.setEnabled(AbstractFXAction.java:87)
at org.drombler.commons.action.fx.ActionListenerAdapter.lambda$new$0(ActionListenerAdapter.java:36)
at java.beans.PropertyChangeSupport.fire(PropertyChangeSupport.java:335)
at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:328)
at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:263)
at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:303)
at org.drombler.commons.action.AbstractActionListener.setEnabled(AbstractActionListener.java:54)
at org.drombler.acp.core.standard.action.data.file.impl.OpenFilesAction.setFileChooserProvider(OpenFilesAction.java:75)
at org.drombler.acp.core.commons.util.SimpleServiceTrackerCustomizer.removedService(SimpleServiceTrackerCustomizer.java:37)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:967)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:870)
at org.osgi.util.tracker.AbstractTracked.untrack(AbstractTracked.java:341)
at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:909)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
at org.apache.felix.framework.Felix.access$000(Felix.java:105)
at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:419)
at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:908)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:873)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:139)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.unregisterService(AbstractComponentManager.java:950)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:805)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:787)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.dispose(AbstractComponentManager.java:579)
at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.disposeComponents(ConfigurableComponentHolder.java:706)
at org.apache.felix.scr.impl.BundleComponentActivator.dispose(BundleComponentActivator.java:523)
at org.apache.felix.scr.impl.Activator.disposeComponents(Activator.java:452)
at org.apache.felix.scr.impl.Activator.access$300(Activator.java:54)
at org.apache.felix.scr.impl.Activator$ScrExtension.destroy(Activator.java:306)
at org.apache.felix.utils.extender.AbstractExtender$2.run(AbstractExtender.java:290)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:312)
at org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:186)
at org.apache.felix.framework.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:915)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:834)
at org.apache.felix.framework.EventDispatcher.fireBundleEvent(EventDispatcher.java:516)
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4541)
at org.apache.felix.framework.Felix.stopBundle(Felix.java:2600)
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1389)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "FelixStartLevel" java.lang.IllegalStateException: This operation is permitted on the event thread only; currentThread = FelixStartLevel
at com.sun.glass.ui.Application.checkEventThread(Application.java:443)
at com.sun.glass.ui.MenuItem.setEnabled(MenuItem.java:108)
at com.sun.javafx.tk.quantum.GlassSystemMenu.lambda$insertMenuItem$347(GlassSystemMenu.java:261)
at com.sun.javafx.binding.ExpressionHelper$SingleInvalidation.fireValueChangedEvent(ExpressionHelper.java:137)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:81)
at javafx.beans.property.BooleanPropertyBase.fireValueChangedEvent(BooleanPropertyBase.java:103)
at javafx.beans.property.BooleanPropertyBase.markInvalid(BooleanPropertyBase.java:110)
at javafx.beans.property.BooleanPropertyBase.access$000(BooleanPropertyBase.java:49)
at javafx.beans.property.BooleanPropertyBase$Listener.invalidated(BooleanPropertyBase.java:245)
at com.sun.javafx.binding.ExpressionHelper$Generic.fireValueChangedEvent(ExpressionHelper.java:349)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:81)
at javafx.beans.property.BooleanPropertyBase.fireValueChangedEvent(BooleanPropertyBase.java:103)
at javafx.beans.property.BooleanPropertyBase.markInvalid(BooleanPropertyBase.java:110)
at javafx.beans.property.BooleanPropertyBase.access$000(BooleanPropertyBase.java:49)
at javafx.beans.property.BooleanPropertyBase$Listener.invalidated(BooleanPropertyBase.java:245)
at com.sun.javafx.binding.ExpressionHelper$SingleInvalidation.fireValueChangedEvent(ExpressionHelper.java:137)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:81)
at javafx.beans.binding.BooleanBinding.invalidate(BooleanBinding.java:176)
at com.sun.javafx.binding.BindingHelperObserver.invalidated(BindingHelperObserver.java:51)
at com.sun.javafx.binding.ExpressionHelper$SingleInvalidation.fireValueChangedEvent(ExpressionHelper.java:137)
at com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:81)
at javafx.beans.property.ReadOnlyBooleanPropertyBase.fireValueChangedEvent(ReadOnlyBooleanPropertyBase.java:72)
at org.drombler.commons.action.fx.AbstractFXAction$EnabledProperty.set(AbstractFXAction.java:122)
at org.drombler.commons.action.fx.AbstractFXAction$EnabledProperty.access$100(AbstractFXAction.java:110)
at org.drombler.commons.action.fx.AbstractFXAction.setEnabled(AbstractFXAction.java:87)
at org.drombler.commons.action.fx.ActionListenerAdapter.lambda$new$0(ActionListenerAdapter.java:36)
at java.beans.PropertyChangeSupport.fire(PropertyChangeSupport.java:335)
at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:328)
at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:263)
at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:303)
at org.drombler.commons.action.AbstractActionListener.setEnabled(AbstractActionListener.java:54)
at org.drombler.acp.core.standard.action.data.file.impl.OpenFilesAction.setFileChooserProvider(OpenFilesAction.java:75)
at org.drombler.acp.core.commons.util.SimpleServiceTrackerCustomizer.removedService(SimpleServiceTrackerCustomizer.java:37)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:967)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:870)
at org.osgi.util.tracker.AbstractTracked.untrack(AbstractTracked.java:341)
at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:909)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
at org.apache.felix.framework.Felix.access$000(Felix.java:105)
at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:419)
at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:908)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:873)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:139)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.unregisterService(AbstractComponentManager.java:950)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:805)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:787)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.dispose(AbstractComponentManager.java:579)
at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.disposeComponents(ConfigurableComponentHolder.java:706)
at org.apache.felix.scr.impl.BundleComponentActivator.dispose(BundleComponentActivator.java:523)
at org.apache.felix.scr.impl.Activator.disposeComponents(Activator.java:452)
at org.apache.felix.scr.impl.Activator.access$300(Activator.java:54)
at org.apache.felix.scr.impl.Activator$ScrExtension.destroy(Activator.java:306)
at org.apache.felix.utils.extender.AbstractExtender$2.run(AbstractExtender.java:290)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:312)
at org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:186)
at org.apache.felix.framework.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:915)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:834)
at org.apache.felix.framework.EventDispatcher.fireBundleEvent(EventDispatcher.java:516)
at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4541)
at org.apache.felix.framework.Felix.stopBundle(Felix.java:2600)
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1389)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
at java.lang.Thread.run(Thread.java:745)
After the error messages and before the last one (Stopped JavaFX Application "MyApplication 1.0.0-SNAPSHOT") the application is blocked for a while (a minute, ore or less).
Claudio
This seems to be an issue when running on Mac.
Please file an issue.
What I want to achieve is that a user in the flash client calls a handler method defined in the red5 app which creates a server side stream and then plays it so that other users with flash client can view the broadcasted stream.
Here is my actionscript client side code triggered by a button click:
private function playLiveStream():void {
nc.call("remoteControl.playLiveStream", null, "demo_video.flv");
}
Here is my server side java code defined in a registered handler:
public void playLiveStream(String fileName) {
IScope scope=Red5.getConnectionLocal().getScope();
IServerStream stream=
StreamUtils.getServerStream(scope,"livestream1");
if (stream==null) {
stream=StreamUtils.createServerStream(scope, "livestream1");
StreamUtils.putServerStream(scope, "livestream1");
}
stream.addItem(SimplePlayItem.build(fileName));
stream.start();
}
Here is the actionscript to play the broadcasted stream:
ns = new NetStream(nc);
ns.play("livestream1");
However, in flash client, the stream plays for only one second and then stopped.
If I click the button to call the handler method again, there is error in the java log. Here is my java log for the error:
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_25]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_25]
at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_25]
at org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:193) [red5-server-common-1.0.6-RELEASE.jar:na]
at org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:115) [red5-server-common-1.0.6-RELEASE.jar:na]
at org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:186) [red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at org.red5.server.net.rtmp.RTMPHandler.onCommand(RTMPHandler.java:293) [red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:105) [red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:63) [red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:14) [red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_25]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_25]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]
Caused by: java.lang.IllegalStateException: State STOPPED not valid to start
at org.red5.server.stream.ServerStream.start(ServerStream.java:428) ~[red5-server.jar:1.0.6-RELEASE]
at com.eeplace.red5.core.RemoteControlHandler.playLiveStream(RemoteControlHandler.java:144) ~[classes/:na]
... 17 common frames omitted
Did anyone encounter the same error before?
Thank you for any suggestion.