After starting installing spinnaker on Oracle Kubernates engine(halyard on docker), we are unable to get all spinnaker pods in Ready state as below:
$ kubectl get pod -n spinnaker
NAME READY STATUS RESTARTS AGE
spin-clouddriver-6795648f5c-7695g 0/1 Running 0 10m
spin-deck-6766bfb78-ttqff 1/1 Running 0 10m
spin-echo-76d84bccf4-wqgq4 1/1 Running 0 10m
spin-front50-6fdc874759-jtrfp 1/1 Running 0 6m
spin-gate-6ffdfd7657-xhjh2 0/1 Running 0 10m
spin-igor-6c564fc765-589cz 1/1 Running 0 10m
spin-orca-567879df75-4t2gn 0/1 Running 0 10m
spin-redis-5bd65c56b6-82rxp 1/1 Running 0 10m
spin-rosco-7cdf5858fc-n5vrx 0/1 Running 0 9m
Please find the logs for the pods which are not ready and please help me with any clue to resolve the issue.
LMK in case you need more logs/info.
Also I am able to get the UI of spinnaker but its not able to create anything
Logs:
**************spin-clouddriver logs**************
#################################################
----------------------------------------------------------------
$ kubectl log -n spinnaker spin-clouddriver-6795648f5c-7695g|grep -i error
log is DEPRECATED and will be removed in a future version. Use logs instead.
2018-09-14 07:35:13.842 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)
----------------------------------------------------------------
$ kubectl log -n spinnaker spin-clouddriver-6795648f5c-7695g|tail -20
log is DEPRECATED and will be removed in a future version. Use logs instead.
2018-09-14 07:35:17.245 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/heapdump || /heapdump.json],methods=[GET],produces=[application/octet-stream]}" onto public void org.springframework.boot.actuate.endpoint.mvc.HeapdumpMvcEndpoint.invoke(boolean,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException,javax.servlet.ServletException
2018-09-14 07:35:17.247 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/actuator || /actuator.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public org.springframework.hateoas.ResourceSupport org.springframework.boot.actuate.endpoint.mvc.HalJsonMvcEndpoint.links()
2018-09-14 07:35:17.259 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/trace || /trace.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.276 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/dump || /dump.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.277 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/mappings || /mappings.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.277 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/autoconfig || /autoconfig.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.303 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/env/{name:.*}],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EnvironmentMvcEndpoint.value(java.lang.String)
2018-09-14 07:35:17.304 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/env || /env.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.305 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/auditevents || /auditevents.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public org.springframework.http.ResponseEntity<?> org.springframework.boot.actuate.endpoint.mvc.AuditEventsMvcEndpoint.findByPrincipalAndAfterAndType(java.lang.String,java.util.Date,java.lang.String)
2018-09-14 07:35:19.677 INFO 1 --- [ main] c.n.s.c.security.ProviderUtils : Adding accounts [my-k8s-v2-account] of type KubernetesNamedAccountCredentials...
2018-09-14 07:37:04.151 WARN 1 --- [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:37:04.154 WARN 1 --- [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : There are no namespaces configured (or loadable) -- please check that the list of 'omitNamespaces' for account 'my-k8s-v2-account' doesn't prevent access from all namespaces in this cluster, or that the cluster is reachable.
2018-09-14 07:38:47.351 WARN 1 --- [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:40:30.099 WARN 1 --- [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:42:12.815 WARN 1 --- [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
**************spin-gate logs**************
##########################################
----------------------------------------------------------------
$ kubectl -n spinnaker logs spin-gate-6ffdfd7657-xhjh2|grep -i error
2018-09-14 07:35:35.599 INFO 1 --- [ost-startStop-1] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)
2018-09-14 07:35:42.556 WARN 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'enableRedisKeyspaceNotificationsInitializer' defined in class path resource [com/netflix/spinnaker/gate/config/GateConfig.class]: Invocation of init method failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2018-09-14 07:35:42.643 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'enableRedisKeyspaceNotificationsInitializer' defined in class path resource [com/netflix/spinnaker/gate/config/GateConfig.class]: Invocation of init method failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
2018-09-14 07:36:38.056 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#661f6038 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:37:38.065 ERROR 1 --- [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#1df18e11 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:38:38.066 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#6e311e73 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:39:38.067 ERROR 1 --- [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#1709e561 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:40:38.069 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#533017fd rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:41:38.070 ERROR 1 --- [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#431be4b2 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:42:38.071 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#6fc06d32 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
----------------------------------------------------------------
$ kubectl -n spinnaker logs spin-gate-6ffdfd7657-xhjh2|tail -40
Caused by: org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:204) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:348) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
at org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration$EnableRedisKeyspaceNotificationsInitializer.afterPropertiesSet(RedisHttpSessionConfiguration.java:249) ~[spring-session-1.3.1.RELEASE.jar:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687) ~[spring-beans-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624) ~[spring-beans-4.3.14.RELEASE.jar:4.3.14.RELEASE]
... 19 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16) ~[jedis-2.9.0.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
... 23 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out
at redis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.0.jar:na]
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:888) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.4.3.jar:2.4.3]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
... 26 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_171]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_171]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_171]
at redis.clients.jedis.Connection.connect(Connection.java:184) ~[jedis-2.9.0.jar:na]
... 33 common frames omitted
2018-09-14 07:36:38.056 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#661f6038 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:37:38.065 ERROR 1 --- [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#1df18e11 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:38:38.066 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#6e311e73 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
**************spin-orca logs**************
##########################################
--------------------------------------------------------------------
$ kubectl -n spinnaker logs spin-orca-567879df75-4t2gn|grep -i error
2018-09-14 07:43:56.069 ERROR 1 --- [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.080 ERROR 1 --- [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.090 ERROR 1 --- [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.101 ERROR 1 --- [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.111 ERROR 1 --- [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
..............
..............
2018-09-14 07:43:59.065 ERROR 1 --- [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.075 ERROR 1 --- [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.086 ERROR 1 --- [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.097 ERROR 1 --- [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.110 ERROR 1 --- [ scheduler-3] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: orca.task.queue:buffered:pipeline
at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:118)
2018-09-14 07:43:59.110 ERROR 1 --- [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.120 ERROR 1 --- [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.131 ERROR 1 --- [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
----------------------------------------------------------------
$ kubectl -n spinnaker logs spin-orca-567879df75-4t2gn|tail -40
2018-09-14 07:45:04.126 ERROR 1 --- [ scheduler-6] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
at com.netflix.spinnaker.q.redis.RedisQueue.poll(RedisQueue.kt:98)
at com.netflix.spinnaker.q.QueueProcessor.pollOnce(QueueProcessor.kt:76)
at com.netflix.spinnaker.q.QueueProcessor.access$pollOnce(QueueProcessor.kt:37)
at com.netflix.spinnaker.q.QueueProcessor$poll$1.invoke(QueueProcessor.kt:62)
at com.netflix.spinnaker.q.QueueProcessor$poll$1.invoke(QueueProcessor.kt:37)
at com.netflix.spinnaker.q.QueueProcessor.ifEnabled(QueueProcessor.kt:119)
at com.netflix.spinnaker.q.QueueProcessor.poll(QueueProcessor.kt:57)
at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.UnknownHostException: spin-redis.spinnaker
at redis.clients.jedis.Connection.connect(Connection.java:207)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93)
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:888)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 21 common frames omitted
Caused by: java.net.UnknownHostException: spin-redis.spinnaker
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at redis.clients.jedis.Connection.connect(Connection.java:184)
... 28 common frames omitted
**************spin-rosco logs**************
###########################################
---------------------------------------------------------------------
$ kubectl -n spinnaker logs spin-rosco-7cdf5858fc-n5vrx|grep -i error
2018-09-14 07:36:12.914 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)
2018-09-14 07:36:24.195 ERROR 1 --- [readScheduler-2] c.n.spinnaker.rosco.executor.BakePoller : Zombie Killer Polling Error:
2018-09-14 07:36:24.195 ERROR 1 --- [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:34.089 ERROR 1 --- [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:49.089 ERROR 1 --- [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:49.099 ERROR 1 --- [readScheduler-2] c.n.spinnaker.rosco.executor.BakePoller : Zombie Killer Polling Error:
----------------------------------------------------------------
$ kubectl -n spinnaker logs spin-rosco-7cdf5858fc-n5vrx|tail -40
at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.callGetProperty(GetEffectivePogoPropertySite.java:48) ~[groovy-all-2.4.13.jar:2.4.13]
at com.netflix.spinnaker.rosco.executor.BakePoller$_onApplicationEvent_closure1.doCall(BakePoller.groovy:81) [rosco-core-0.108.0-SNAPSHOT.jar:0.108.0-SNAPSHOT]
at com.netflix.spinnaker.rosco.executor.BakePoller$_onApplicationEvent_closure1.doCall(BakePoller.groovy) [rosco-core-0.108.0-SNAPSHOT.jar:0.108.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.Closure.call(Closure.java:414) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124) [groovy-all-2.4.13.jar:2.4.13]
at com.sun.proxy.$Proxy116.call(Unknown Source) [na:na]
at rx.Scheduler$Worker$1.call(Scheduler.java:120) [rxjava-1.0.16.jar:1.0.16]
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.0.16.jar:1.0.16]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.UnknownHostException: spin-redis.spinnaker
at redis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.0.jar:na]
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.4.2.jar:2.4.2]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
... 32 common frames omitted
Caused by: java.net.UnknownHostException: spin-redis.spinnaker
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[na:1.8.0_171]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_171]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_171]
at redis.clients.jedis.Connection.connect(Connection.java:184) ~[jedis-2.9.0.jar:na]
... 39 common frames omitted
I had the same issue with K8s V2 provider and I figured out that the main cause was an HAL misconfiguration. I just put the providers > kubernetes > [account] > serviceAccount to true. This was the trick.
If it is your case the configuration command is:
hal config provider kubernetes account edit <account-name> --service-account true
...then re-deploy Spinnaker via hal deploy apply
Hope this helps you. Good luck. :)
Related
InterProcessMutex zLock=new InterProcessMutex(curatorFramework,lockPath);
zlock.acquire(time,unit);
just a simple example. I try to test the Network jitter problem, so I start the service,connect zookeeper and start the curator client, and then disable the network.
after that, I create a request which would call the zlock.acquire(time,unit).
It shows that the request return error response at least 1min later.
to debug the codes,the program stop in "createsTheLock" method.
In fact, I guess the expectation is to return false when zookeeper is not available.
Thanks for answerring the question!
update the question:
It's the base setting:
RetryPolicy retryPolicy=new ExponentialBackoffRetry(1000,3);
CuratorFramework curatorFramework= CuratorFrameworkFactory.builder().connectionTimeoutMs(15000).sessionTimeoutMs(60000).connectString(CommonConstant.ZK_CONNECT_HOSTS).retryPolicy(retryPolicy).build();
and it's the log:
2020-08-09 10:27:12.582 INFO 91181 --- [224.71.42:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server 10.224.71.42/10.224.71.42:2181. Will not attempt to authenticate using SASL (unknown error)
2020-08-09 10:27:12.582 ERROR 91181 --- [224.71.42:2181)] o.apache.zookeeper.ClientCnxnSocketNIO : Unable to open socket to 10.224.71.42/10.224.71.42:2181
2020-08-09 10:27:12.582 INFO 91181 --- [224.71.42:2181)] org.apache.zookeeper.ClientCnxn : Socket error occurred: 10.224.71.42/10.224.71.42:2181: Network is unreachable
2020-08-09 10:27:13.456 ERROR 91181 --- [tor-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl : Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[zookeeper-3.5.6.jar:3.5.6]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:851) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:999) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:952) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:342) [curator-framework-5.1.0.jar:5.1.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_241]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_241]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_241]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_241]
2020-08-09 10:27:13.457 ERROR 91181 --- [tor-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl : Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:981) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:952) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) [curator-framework-5.1.0.jar:5.1.0]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:342) [curator-framework-5.1.0.jar:5.1.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_241]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_241]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_241]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_241]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_241]
2020-08-09 10:27:14.684 INFO 91181 --- [26.102.59:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server 10.226.102.59/10.226.102.59:2181. Will not attempt to authenticate using SASL (unknown error)
2020-08-09 10:27:14.684 ERROR 91181 --- [26.102.59:2181)] o.apache.zookeeper.ClientCnxnSocketNIO : Unable to open socket to 10.226.102.59/10.226.102.59:2181
2020-08-09 10:27:14.684 INFO 91181 --- [26.102.59:2181)] org.apache.zookeeper.ClientCnxn : Socket error occurred: 10.226.102.59/10.226.102.59:2181: Network is unreachable
The request just hang up until 59s later, then return response;
I am trying to do channel.basicReject() to requeue message based on some condition by creating an MethodInterceptor ConsumerAdvice and adding it to SMLC factor.setAdviceChain(new ConsumerAdvice()). I also have concurrentConsumer configuration which is set to 10. The moment my reject condition is met I issue basicReject command and it gets redelivered and processed by another consumer. During this redelivery process I get the below error,
2019-11-07 17:34:13.268 ERROR 29385 --- [ 127.0.0.1:5672] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=80)
2019-11-07 17:34:13.268 DEBUG 29385 --- [ool-2-thread-13] o.s.a.r.listener.BlockingQueueConsumer : Received shutdown signal for consumer tag=amq.ctag-HUaN71TZUqMfLDR7k6LwGQ
com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=80)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:516)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:346)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:178)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111)
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:670)
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:597)
at java.lang.Thread.run(Thread.java:748)
My message is not getting lost but I am seeing a bunch of above errors and unable to understand why this is happening. If anyone has any clues please guide me.
Below are the trace logs,
2019-11-08 02:11:31.883 TRACE 8695 --- [askExecutor-138] o.s.a.r.c.CachingConnectionFactory : AMQChannel(amqp://guest#127.0.0.1:5672/,99) channel.getChannelNumber()
2019-11-08 02:11:31.883 INFO 8695 --- [askExecutor-138] c.g.s.w.consumer.advice.ArgumentUtils : Channel number before triggering redelivery : 99
2019-11-08 02:11:31.883 TRACE 8695 --- [askExecutor-138] o.s.a.r.c.CachingConnectionFactory : AMQChannel(amqp://guest#127.0.0.1:5672/,99) channel.basicReject([2, true])
2019-11-08 02:11:31.883 INFO 8695 --- [askExecutor-138] c.g.s.w.consumer.advice.ArgumentUtils : ==============================================================================
2019-11-08 02:11:31.883 INFO 8695 --- [askExecutor-138] c.g.s.w.consumer.advice.ConsumerAdvice : Requeue Message attempted, status : true
2019-11-08 02:11:31.884 TRACE 8695 --- [askExecutor-138] o.s.a.r.l.SimpleMessageListenerContainer : Waiting for message from consumer.
2019-11-08 02:11:31.884 TRACE 8695 --- [askExecutor-138] o.s.a.r.listener.BlockingQueueConsumer : Retrieving delivery for Consumer#7783912f: tags=[[amq.ctag-eY7LN-1pSXPX8FKRBgt-ug]], channel=Cached Rabbit Channel: AMQChannel(amqp://guest#127.0.0.1:5672/,99), conn: Proxy#37ffe4f3 Shared Rabbit Connection: SimpleConnection#708dfe10 [delegate=amqp://guest#127.0.0.1:5672/, localPort= 58638], acknowledgeMode=AUTO local queue size=0
2019-11-08 02:11:31.884 DEBUG 8695 --- [askExecutor-138] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=80)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.checkShutdown(BlockingQueueConsumer.java:436)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.nextMessage(BlockingQueueConsumer.java:501)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:843)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:832)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:78)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1073)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=80)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:516)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:346)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:178)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111)
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:670)
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:597)
... 1 common frames omitted
2019-11-08 02:11:31.884 ERROR 8695 --- [ 127.0.0.1:5672] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 2, class-id=60, method-id=90)
You need to show your code and configuration.
It seems like the SMLC has its default configuration to automatically acknowledge messages and this failure is because you already rejected it; why are you interacting with the channel directly?
You can simply throw an exception and the container will reject the message on your behalf.
I don't know if it will helpful for someone.
I had the same error because of invalid input data. But when I added next properties:
"rabbit.listener.acknowledgeMode": "MANUAL",
"rabbit.listener.defaultRequeueRejected": "true",
"rabbit.listener.prefetchCount": "1",
the problem stop to break my program but only had stopping my listener
2018-12-14 17:41:57,313 [INFO] [InputInitializer {Map 1} #0] |io.HiveInputFormat|: Generating splits
2018-12-14 17:41:57,331 [INFO] [InputInitializer {Map 1} #0] |log.PerfLogger|: <PERFLOG method=OrcGetSplits from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
2018-12-14 17:41:57,335 [INFO] [InputInitializer {Map 1} #0] |dag.RootInputInitializerManager|: Failed InputInitializer for Input: t1 on vertex vertex_1535529560012_804349_1_02 [Map 1]
2018-12-14 17:41:57,336 [ERROR] [Dispatcher thread {Central}] |impl.VertexImpl|: Vertex Input: t1 initializer failed, vertex=vertex_1535529560012_804349_1_02 [Map 1]
org.apache.tez.dag.app.dag.impl.AMUserCodeException: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:327)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:977)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
at com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:86)
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384)
at java.util.concurrent.FutureTask.setException(FutureTask.java:251)
at java.util.concurrent.FutureTask.run(FutureTask.java:271)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.hadoop.hive.common.ValidReadTxnList.readFromString(ValidReadTxnList.java:142)
at org.apache.hadoop.hive.common.ValidReadTxnList.<init>(ValidReadTxnList.java:57)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$Context.<init>(OrcInputFormat.java:413)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:941)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:971)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:306)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:408)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:130)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
The problem solved, it is a class conflict.
I import hive-storage-api-2.6.0.jar Hive-common-1.1.0-cdh5.11.0.jar to one pom.xml .
org.apache.hadoop.hive.common.ValidReadTxnList in hive-storage-api-2.6.0.jar
conflicts with
org.apache.hadoop.hive.common.ValidReadTxnList Hive-common-1.1.0-cdh5.11.0.jar .
When I run my Spring Cloud Config Client project config-client, I found these error:
2018-02-09 10:31:09.885 INFO 13933 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at: http://localhost:8888
2018-02-09 10:31:10.022 WARN 13933 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/config-client/dev/master": 拒绝连接 (Connection refused); nested exception is java.net.ConnectException: 拒绝连接 (Connection refused)
2018-02-09 10:31:10.026 INFO 13933 --- [ main] c.y.c.ConfigClientApplication : No active profile set, falling back to default profiles: default
2018-02-09 10:31:10.040 INFO 13933 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#33b1c5c5: startup date [Fri Feb 09 10:31:10 CST 2018]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext#1ffe63b9
2018-02-09 10:31:10.419 INFO 13933 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=65226c2b-524f-3b14-8e17-9fdbc9f72d85
2018-02-09 10:31:10.471 INFO 13933 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$25380e89] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2018-02-09 10:31:10.688 INFO 13933 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 10001 (http)
2018-02-09 10:31:10.697 INFO 13933 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2018-02-09 10:31:10.698 INFO 13933 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.27
2018-02-09 10:31:10.767 INFO 13933 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2018-02-09 10:31:10.768 INFO 13933 --- [ost-startStop-1] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 727 ms
2018-02-09 10:31:10.861 INFO 13933 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean : Mapping servlet: 'dispatcherServlet' to [/]
2018-02-09 10:31:10.864 INFO 13933 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'characterEncodingFilter' to: [/*]
2018-02-09 10:31:10.864 INFO 13933 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2018-02-09 10:31:10.864 INFO 13933 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2018-02-09 10:31:10.865 INFO 13933 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'requestContextFilter' to: [/*]
2018-02-09 10:31:10.895 WARN 13933 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configClientApplication': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'content' in value "${content}"
2018-02-09 10:31:10.896 INFO 13933 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2018-02-09 10:31:10.914 INFO 13933 --- [ main] utoConfigurationReportLoggingInitializer :
Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2018-02-09 10:31:10.923 ERROR 13933 --- [ main] o.s.boot.SpringApplication : Application startup failed
Apparently, the config server is wrong. However, the Spring Cloud Config Server is running at localhost:10000/ and application.yml of the project(config-client) is below. Why the spring.cloud.config.uri doesn't work?
application.yml [config-client]
server:
port: 10001
spring:
application:
name: config-client
cloud:
config:
label: master
profile: dev
uri: http://localhost:10000
Fur future readers, as answered here, when using Spring Cloud Config Server, we should specify basic bootstrap settings such as : spring.application.name and spring.cloud.config.uri inside bootstrap.yml (or "bootstrap.properties").
Upon startup, Spring Cloud makes an HTTP call to the config server with the name of the application and retrieves back that application's configuration.
That's said, since we're externalizing our settings using Spring Cloud Config Server, any default configurations defined in application.yml (or "application.properties") will be overridden during the bootstrap process upon startup.
IntelliJ Users: add the following override parameter in the run/Debug Configuration:
Name: spring.cloud.config.uri
Value: http://your-server-here/config-server
you can load configuration servers before starting the Application, using bootstrap.yml
just add configuration server and application name
spring:
application:
name: clientTest
cloud:
config:
uri: http://localhost:8889
enabled: true
fail-fast: true
if we are using bootstrap.properties. we have to include this dependency in pom for spring-2.4.0+
agregado para evitar un error al usar spring mayor que 2.4.0
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bootstrap</artifactId>
</dependency>
in my case i was testing spring consul, which usually runs in 8500, i saw a different port in the log. Found that the different port is due to following deplendency of spring cloud. Hence i just have to remove it.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
I had a Turbine and Hystrix setup working, but decided to change it over to Turbine AMQP so I could aggregate multiple services into one stream/dashboard.
I have set up a Turbine AMQP server running on localhost:8989, but it doesn't appear to be getting Hystrix data from the client service. When I hit the Turbine server's IP in my browser, I see data: {"type":"Ping"} repeatedly, even while I am polling the URL of the Hystrix. If I attempt to show the Turbine AMQP stream in the Hystrix Dashboard, I get: Unable to connect to Command Metric Stream.
I have a default install of RabbitMQ running on port 5672.
My client service using Hystrix-AMQP has a application.yml file that looks like so:
spring:
application:
name: policy-service
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
spring:
rabbitmq:
addresses: ${vcap.services.${PREFIX:}rabbitmq.credentials.uri:amqp://${RABBITMQ_HOST:localhost}:${RABBITMQ_PORT:5672}}
The tail end of the startup log looks like this:
2015-09-14 16:31:13.030 INFO 52844 --- [ main] com.netflix.discovery.DiscoveryClient : Starting heartbeat executor: renew interval is: 30
2015-09-14 16:31:13.047 INFO 52844 --- [ main] c.n.e.EurekaDiscoveryClientConfiguration : Registering application policy-service with eureka with status UP
2015-09-14 16:31:13.194 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.i.channel.PublishSubscribeChannel : Channel 'policy-service:8088.errorChannel' has 1 subscriber(s).
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started _org.springframework.integration.errorLogger
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {filter} as a subscriber to the 'cloudBusOutboundFlow.channel#0' channel
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy- service:8088.cloudBusOutboundFlow.channel#0' has 1 subscriber(s).
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {filter} as a subscriber to the 'cloudBusInboundChannel' channel
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.cloudBusInboundChannel' has 1 subscriber(s).
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {message-handler} as a subscriber to the 'cloudBusInboundFlow.channel#0' channel
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.cloudBusInboundFlow.channel#0' has 1 subscriber(s).
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#2
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter} as a subscriber to the 'cloudBusWiretapChannel' channel
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.cloudBusWiretapChannel' has 1 subscriber(s).
2015-09-14 16:31:13.197 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#3
2015-09-14 16:31:13.197 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {amqp:outbound-channel-adapter} as a subscriber to the 'cloudBusOutboundChannel' channel
2015-09-14 16:31:13.197 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.cloudBusOutboundChannel' has 1 subscriber(s).
2015-09-14 16:31:13.197 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#4
2015-09-14 16:31:13.198 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {bridge} as a subscriber to the 'cloudBusAmqpInboundFlow.channel#0' channel
2015-09-14 16:31:13.198 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.cloudBusAmqpInboundFlow.channel#0' has 1 subscriber(s).
2015-09-14 16:31:13.198 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#5
2015-09-14 16:31:13.198 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {amqp:outbound-channel-adapter} as a subscriber to the 'hystrixStream' channel
2015-09-14 16:31:13.199 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.hystrixStream' has 1 subscriber(s).
2015-09-14 16:31:13.199 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#6
2015-09-14 16:31:13.219 INFO 52844 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 1073741823
2015-09-14 16:31:13.219 INFO 52844 --- [ main] ApplicationEventListeningMessageProducer : started org.springframework.integration.event.inbound.ApplicationEventListeningMessageProducer#0
2015-09-14 16:31:13.555 INFO 52844 --- [cTaskExecutor-1] o.s.amqp.rabbit.core.RabbitAdmin : Auto-declaring a non-durable, auto-delete, or exclusive Queue (4640c1c8-ff8f-45d7-8426-19d1b7a4cdb0) durable:false, auto-delete:true, exclusive:true. It will be redeclared if the broker stops and is restarted while the connection factory is alive, but all messages will be lost.
2015-09-14 16:31:13.572 INFO 52844 --- [ main] o.s.i.a.i.AmqpInboundChannelAdapter : started org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter#0
2015-09-14 16:31:13.573 INFO 52844 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 2147483647
2015-09-14 16:31:13.576 INFO 52844 --- [ main] c.n.h.c.m.e.HystrixMetricsPoller : Starting HystrixMetricsPoller
2015-09-14 16:31:13.609 INFO 52844 --- [ main] ration$HystrixMetricsPollerConfiguration : Starting poller
2015-09-14 16:31:13.803 INFO 52844 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8088 (http)
2015-09-14 16:31:13.805 INFO 52844 --- [ main] com.ml.springboot.PolicyService : Started PolicyService in 22.544 seconds (JVM running for 23.564)
So it looks like PolicyService successfully connects to the message broker.
The Turbine AMQP server's end of log:
2015-09-14 16:58:05.887 INFO 51944 --- [ main] i.reactivex.netty.server.AbstractServer : Rx server started at port: 8989
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.i.channel.PublishSubscribeChannel : Channel 'bootstrap:-1.errorChannel' has 1 subscriber(s).
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started _org.springframework.integration.errorLogger
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {bridge} as a subscriber to the 'hystrixStreamAggregatorInboundFlow.channel#0' channel
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.integration.channel.DirectChannel : Channel 'bootstrap:-1.hystrixStreamAggregatorInboundFlow.channel#0' has 1 subscriber(s).
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 1073741823
2015-09-14 16:58:06.238 INFO 51944 --- [cTaskExecutor-1] o.s.amqp.rabbit.core.RabbitAdmin : Auto-declaring a non-durable, auto-delete, or exclusive Queue (spring.cloud.hystrix.stream) durable:false, auto-delete:false, exclusive:false. It will be redeclared if the broker stops and is restarted while the connection factory is alive, but all messages will be lost.
2015-09-14 16:58:06.289 INFO 51944 --- [ main] o.s.i.a.i.AmqpInboundChannelAdapter : started org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter#0
2015-09-14 16:58:06.290 INFO 51944 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 2147483647
2015-09-14 16:58:06.434 INFO 51944 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): -1 (http)
Any ideas why the Turbine AMQP server is not receiving communication from the Hystrix AMQP client?
EDIT: Turbine-AMQP main looks like:
package com.turbine.amqp;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.netflix.turbine.amqp.EnableTurbineAmqp;
import org.springframework.context.annotation.Configuration;
#Configuration
#EnableAutoConfiguration
#EnableTurbineAmqp
#EnableDiscoveryClient
public class TurbineAmqpApplication {
public static void main(String[] args) {
SpringApplication.run(TurbineAmqpApplication.class, args);
}
}
Here's its application.yml:
server:
port: 8989
spring:
rabbitmq:
addresses: ${vcap.services.${PREFIX:}rabbitmq.credentials.uri:amqp://${RABBITMQ_HOST:localhost}:${RABBITMQ_PORT:5672}}
Hitting http://localhost:8989/turbine.stream produces a repeating stream of data: {"type":"Ping"}
and shows this in console:
2015-09-15 08:54:37.960 INFO 83480 --- [o-eventloop-3-1] o.s.c.n.t.amqp.TurbineAmqpConfiguration : SSE Request Received
2015-09-15 08:54:38.025 INFO 83480 --- [o-eventloop-3-1] o.s.c.n.t.amqp.TurbineAmqpConfiguration : Starting aggregation
EDIT: The below exception is thrown when I stop listening to the turbine stream, not when I try to listen with the dashboard.
2015-09-15 08:56:47.934 INFO 83480 --- [o-eventloop-3-3] o.s.c.n.t.amqp.TurbineAmqpConfiguration : SSE Request Received
2015-09-15 08:56:47.946 WARN 83480 --- [o-eventloop-3-3] io.netty.channel.DefaultChannelPipeline : An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.NoSuchMethodError: rx.Observable.collect(Lrx/functions/Func0;Lrx/functions/Action2;)Lrx/Observable;
at com.netflix.turbine.aggregator.StreamAggregator.lambda$null$36(StreamAggregator.java:89)
at rx.internal.operators.OnSubscribeMulticastSelector.call(OnSubscribeMulticastSelector.java:60)
at rx.internal.operators.OnSubscribeMulticastSelector.call(OnSubscribeMulticastSelector.java:40)
at rx.Observable.unsafeSubscribe(Observable.java:8591)
at rx.internal.operators.OperatorMerge$MergeSubscriber.handleNewSource(OperatorMerge.java:190)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:160)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:96)
at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54)
at rx.internal.operators.OperatorGroupBy$GroupBySubscriber.onNext(OperatorGroupBy.java:173)
at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onNext(SubjectSubscriptionManager.java:224)
at rx.subjects.PublishSubject.onNext(PublishSubject.java:101)
at org.springframework.cloud.netflix.turbine.amqp.Aggregator.handle(Aggregator.java:53)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:112)
at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:102)
at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:342)
at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:131)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:330)
at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:164)
at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:276)
at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:142)
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:75)
at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:71)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:99)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:277)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:239)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:95)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:248)
at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:171)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:119)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:105)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:277)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:239)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:95)
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:101)
at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter.access$400(AmqpInboundChannelAdapter.java:45)
at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$1.onMessage(AmqpInboundChannelAdapter.java:93)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:756)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1241)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:660)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1005)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:989)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1103)
at java.lang.Thread.run(Thread.java:745)
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: GroupedObservable.class
at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:98)
at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:56)
... 58 common frames omitted
My dependencies for turbine-amqp are as follows:
dependencies {
compile('org.springframework.cloud:spring-cloud-starter-turbine-amqp:1.0.3.RELEASE')
compile 'org.springframework.boot:spring-boot-starter-web:1.2.5.RELEASE'
compile 'org.springframework.boot:spring-boot-starter-actuator:1.2.5.RELEASE'
testCompile("org.springframework.boot:spring-boot-starter-test")
}
dependencyManagement {
imports {
mavenBom 'org.springframework.cloud:spring-cloud-starter-parent:1.0.2.RELEASE'
}
}
It is so hard to find a solution.
Using Spring cloud 2.1.4.RELEASE I faced with similar problem.
The main cause is the incompatibility [exchanges] name in rabbitMQ between:
spring-cloud-netflix-hystrix-stream and spring-cloud-starter-netflix-turbine-stream.
So solve it:
See the name created exchange name when you start the service componente {the same that declare hystrix-stream}
on the componente that declare {turbine-stream}
update the property
turbine.stream.destination=
in my case
turbine.stream.destination=hystrixStreamOutput
I faced with similar problem and I find a solution.
My Spring Cloud version is 2.1.0.RELEASE
The solution:
add property
spring.cloud.stream.bindings.turbineStreamInput.destination: hystrixStreamOutput
turbine.stream.enabled: false
add auto configuration
#EnableBinding(TurbineStreamClient.class)
public class TurbineStreamAutoConfiguration {
#Autowired
private BindingServiceProperties bindings;
#Autowired
private TurbineStreamProperties properties;
#PostConstruct
public void init() {
BindingProperties inputBinding = this.bindings.getBindings()
.get(TurbineStreamClient.INPUT);
if (inputBinding == null) {
this.bindings.getBindings().put(TurbineStreamClient.INPUT,
new BindingProperties());
}
BindingProperties input = this.bindings.getBindings()
.get(TurbineStreamClient.INPUT);
if (input.getDestination() == null) {
input.setDestination(this.properties.getDestination());
}
if (input.getContentType() == null) {
input.setContentType(this.properties.getContentType());
}
}
#Bean
public HystrixStreamAggregator hystrixStreamAggregator(ObjectMapper mapper,
PublishSubject<Map<String, Object>> publisher) {
return new HystrixStreamAggregator(mapper, publisher);
}
}