I have written redis test cases with embedded redis and its worked fine without any issues in local. But when I moved to CI / CD pipeline with jenkins or gitlab. sometime, i am facing connection refused issue.
enviroment:
spring-data-redis:2.2.8
lettuce:5.5.2
eu.monniot.redis:embedded-redis:1.2.2
log:
org.springframework.test.context.event.EventPublishingTestExecutionListener]
2020-08-24 14:22:02.281 INFO 4273 --- [ Test worker] t.a.d.r.DataRedisTestContextBootstrapper : Using TestExecutionListeners: [org.springframework.test.context.web.ServletTestExecutionListener#5c88a8a1, org.springframework.test.context.support.DirtiesContextBeforeModesTestExecutionListener#365ab4bb, org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener#65d657e9, org.springframework.boot.test.autoconfigure.SpringBootDependencyInjectionTestExecutionListener#34ad82d5, org.springframework.test.context.support.DirtiesContextTestExecutionListener#2d79e941, org.springframework.test.context.transaction.TransactionalTestExecutionListener#4ee57f39, org.springframework.test.context.jdbc.SqlScriptsTestExecutionListener#19db0013, org.springframework.test.context.event.EventPublishingTestExecutionListener#413892d7, org.springframework.boot.test.autoconfigure.restdocs.RestDocsTestExecutionListener#23e94850, org.springframework.boot.test.autoconfigure.web.client.MockRestServiceServerResetTestExecutionListener#4690f7a3, org.springframework.boot.test.autoconfigure.web.servlet.MockMvcPrintOnlyOnFailureTestExecutionListener#78aacfa, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverTestExecutionListener#2a128f6, org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListener#6be35b3a]
2020-08-24 14:22:02.405 INFO 4273 --- [llEventLoop-4-6] i.l.core.protocol.ReconnectionHandler : Reconnected to 127.0.0.1:16379
2020-08-24 14:22:02.405 INFO 4273 --- [llEventLoop-4-5] i.l.core.protocol.ReconnectionHandler : Reconnected to localhost:16379
2020-08-24 14:22:02.413 INFO 4273 --- [llEventLoop-7-2] i.l.core.protocol.ReconnectionHandler : Reconnected to localhost:16379
2020-08-24 14:22:02.414 INFO 4273 --- [llEventLoop-7-8] i.l.core.protocol.ReconnectionHandler : Reconnected to 127.0.0.1:16379
2020-08-24 14:22:02.504 INFO 4273 --- [llEventLoop-4-8] i.l.core.protocol.ReconnectionHandler : Reconnected to 127.0.0.1:16379
2020-08-24 14:22:02.613 INFO 4273 --- [llEventLoop-7-6] i.l.core.protocol.ReconnectionHandler : Reconnected to 127.0.0.1:17379
2020-08-24 14:22:02.613 INFO 4273 --- [llEventLoop-7-5] i.l.core.protocol.ReconnectionHandler : Reconnected to 127.0.0.1:16379
2020-08-24 14:22:02.704 INFO 4273 --- [llEventLoop-4-2] i.l.core.protocol.ReconnectionHandler : Reconnected to 127.0.0.1:17379
2020-08-24 14:22:05.313 INFO 4273 --- [llEventLoop-7-1] i.l.core.protocol.ReconnectionHandler : Reconnected to 127.0.0.1:19379
2020-08-24 14:22:05.505 INFO 4273 --- [llEventLoop-4-1] i.l.core.protocol.ReconnectionHandler : Reconnected to 127.0.0.1:18369
2020-08-24 14:22:07.012 INFO 4273 --- [xecutorLoop-5-1] i.l.core.protocol.ConnectionWatchdog : Reconnecting, last destination was 127.0.0.1:16379
2020-08-24 14:22:07.012 INFO 4273 --- [xecutorLoop-5-8] i.l.core.protocol.ConnectionWatchdog : Reconnecting, last destination was localhost:16379
2020-08-24 14:22:07.013 WARN 4273 --- [llEventLoop-7-4] i.l.core.protocol.ConnectionWatchdog : Cannot reconnect to [127.0.0.1:16379]: finishConnect(..) failed: Connection refused: /127.0.0.1:16379
2020-08-24 14:22:07.013 WARN 4273 --- [llEventLoop-7-3] i.l.core.protocol.ConnectionWatchdog : Cannot reconnect to [localhost:16379]: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:16379
2020-08-24 14:22:07.104 INFO 4273 --- [xecutorLoop-1-1] i.l.core.protocol.ConnectionWatchdog : Reconnecting, last destination was localhost:16379
2020-08-24 14:22:07.104 INFO 4273 --- [xecutorLoop-1-2] i.l.core.protocol.ConnectionWatchdog : Reconnecting, last destination was 127.0.0.1:17379
2020-08-24 14:22:07.104 INFO 4273 --- [xecutorLoop-1-8] i.l.core.protocol.ConnectionWatchdog : Reconnecting, last destination was 127.0.0.1:16379
2020-08-24 14:22:07.105 WARN 4273 --- [llEventLoop-4-2] i.l.core.protocol.ConnectionWatchdog : Cannot reconnect to [127.0.0.1:16379]: finishConnect(..) failed: Connection refused: /127.0.0.1:16379
2020-08-24 14:22:07.105 WARN 4273 --- [llEventLoop-4-1] i.l.core.protocol.ConnectionWatchdog : Cannot reconnect to [127.0.0.1:17379]: finishConnect(..) failed: Connection refused: /127.0.0.1:17379
2020-08-24 14:22:07.105 WARN 4273 --- [llEventLoop-4-8] i.l.core.protocol.ConnectionWatchdog : Cannot reconnect to [localhost:16379]: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:16379
code:
companion object {
private lateinit var cluster: Redis
#JvmStatic
#BeforeAll
#Timeout(120)
fun beforeAll() {
cluster = RedisCluster.Builder()
.serverPorts(arrayListOf(16379, 17379, 18369, 19379))
.numOfReplicates(1)
.numOfRetries(12)
.build()
cluster.start()
}
#JvmStatic
#AfterAll
fun afterAll() {
cluster.stop()
}
}
I can't fix it. Can anybody help me?
Related
I am using Spring Boot 2.4.0 with Spring Boot Data JPA to connect to PostgreSQL and perform typical read and write operations with JPA based repositories. Since the database is also used by other services, I use the LISTEN/NOTIFY functionality (https://www.postgresql.org/docs/9.1/sql-listen.html) to be notified about changes from PostgeSQL. For this I use the driver com.impossibl.postgres.jdbc.PGDriver instead of the default driver and the following code to make Spring listen for changes to the database:
#Service
class PostgresChangeListener(
val dataSource: HikariDataSource,
#Qualifier("dbToPGReceiverQueue") val postgresQueue: RBlockingQueue<String>
) {
init {
listenToNotifyMessage()
}
final fun listenToNotifyMessage() {
val notificationListener = object:PGNotificationListener {
override fun notification(processId: Int, channelName: String, payload: String) {
log.info("Received change from PostgresQL: $processId, $channelName, $payload")
postgresQueue.add(payload)
}
override fun closed() {
log.debug("Connection to Postgres lost! Try to reconnect...")
listenToNotifyMessage()
}
}
try {
val connection = DataSourceUtils.getConnection(dataSource).unwrap(PGConnection::class.java)
connection.addNotificationListener(notificationListener)
connection.createStatement().use { statement -> statement.execute("LISTEN change_notifier;") }
} catch (e: SQLException) {
throw RuntimeException(e)
}
}
}
This is the Kotlin-like implementation of the listener discribed here: https://impossibl.github.io/pgjdbc-ng/docs/current/user-guide/#extensions-notifications
The listener works, however after one or more days I get the following error:
2021-03-03 06:33:00.185 WARN 1 --- [nio-8080-exec-8] o.s.b.a.jdbc.DataSourceHealthIndicator : DataSource health check failed
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30001ms.
...
To find the problem, i enabled logging from Hikari as recommended on https://github.com/brettwooldridge/HikariCP/issues/1111#issuecomment-569552070. Here is the output of an excerpt of the logs:
2021-03-02 21:31:59.055 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
...
2021-03-02 21:31:59.055 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2021-03-02 22:00:53.139 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#201ab69f: (connection has passed maxLifetime)
2021-03-02 22:00:53.162 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#f2ffd1ea
2021-03-02 22:00:54.709 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#3bb847ef: (connection has passed maxLifetime)
2021-03-02 22:00:54.730 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#fd5932d7
2021-03-02 22:00:59.110 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2021-03-02 22:00:59.111 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-03-02 22:01:04.782 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#1d081266: (connection has passed maxLifetime)
2021-03-02 22:01:04.803 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#e0b396bc
2021-03-02 22:01:09.295 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#a2b0bd29: (connection has passed maxLifetime)
2021-03-02 22:01:09.313 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#ca9c8226
2021-03-02 22:01:10.075 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#ec8746aa: (connection has passed maxLifetime)
2021-03-02 22:01:10.093 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#aff2bfd8
2021-03-02 22:01:12.820 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#a7e0fc39: (connection has passed maxLifetime)
2021-03-02 22:01:12.840 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#d637554
2021-03-02 22:01:15.099 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#dadcba66: (connection has passed maxLifetime)
2021-03-02 22:01:15.119 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#e29805ef
2021-03-02 22:01:21.558 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#762f0753: (connection has passed maxLifetime)
2021-03-02 22:01:21.576 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#d5b8d008
2021-03-02 22:01:23.351 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#5e4721b0: (connection has passed maxLifetime)
2021-03-02 22:01:23.370 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#a8606b56
2021-03-02 22:01:29.111 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2021-03-02 22:01:29.111 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-03-02 22:01:59.112 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
...
For me the log looks correct but after a while the active connections increase more and more...
...
2021-03-03 06:31:29.664 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=9, idle=1, waiting=0)
2021-03-03 06:31:48.687 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase : HikariPool-1 - Closing connection com.impossibl.postgres.jdbc.PGDirectConnection#4fa5ec41: (connection is dead)
2021-03-03 06:31:48.707 DEBUG 1 --- [onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection com.impossibl.postgres.jdbc.PGDirectConnection#693052fe
2021-03-03 06:31:48.709 DEBUG 1 --- [nnection closer] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-03-03 06:31:59.665 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=10, idle=0, waiting=1)
2021-03-03 06:31:59.665 DEBUG 1 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-03-03 06:32:20.199 DEBUG 1 --- [io-8080-exec-10] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Timeout failure stats (total=10, active=10, idle=0, waiting=2)
2021-03-03 06:32:20.208 WARN 1 --- [io-8080-exec-10] o.s.b.a.jdbc.DataSourceHealthIndicator : DataSource health check failed
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
...
... until it comes to the described error message.
I wonder how I need to configure Hikari correctly or change my code to avoid the errors described? I hope you can help.
Not the same issue, but I had a similar problem. When my database was restarted, Hikari couldn't close the active listener connection, and the whole notification stopped working.
I found a possible solution for this. The reason why Hikari can't close the connection when it's dead because you are unwrapping the connection from the proxied Connection here:
DataSourceUtils.getConnection(dataSource).unwrap(PGConnection::class.java)
After this you are attaching a notificationListener to the PGConnection, so it remains alive.
First thing first to avoid hikaripool leaking you should seperate the 2 connection, and after initializing the listener you should close the hikariConnection.
private hikariConnection: Connection;
...
hikariConnection = DataSourceUtils.getConnection(dataSource)
val pgConnection: PGConnection = hikariConnection.unwrap(PGConnection::class.java)
... init the listener
hikariConnection.close()
And in the PGNotificationListener.closed() you have to reinitalize the listener, get a new Connection from the datasource. But beware, getting new connection while the Hikaripool filling it's pool(because the database outage was only a few seconds), can block each other. We solved it by getting the new connection on a dedicated new thread.
override fun closed() {
... get a new PGConnection, and start listening for the notifications
}
Sorry if it's not correctly answering your question, but it may help to some.
Connecting to local redis, Lettuce takes nearly 5000ms, but Jedis only takes 30ms.
I refer to thie example ConnectToRedis
I use the default spring-boot-starter with lombok dependency:
My Code：
#Component
#Slf4j
class LettuceRunner implements CommandLineRunner {
#Override
public void run(String... args) throws Exception {
StopWatch watch = new StopWatch();
RedisClient redisClient = RedisClient.create("redis://localhost:6379");
watch.start();
StatefulRedisConnection<String, String> connection = redisClient.connect();
watch.stop();
log.info("lettuce : {} ms", watch.getLastTaskTimeMillis());
connection.close();
redisClient.shutdown();
}
}
#Component
#Slf4j
class JedisRunner implements CommandLineRunner {
#Override
public void run(String... args) throws Exception {
StopWatch watch = new StopWatch();
watch.start();
Jedis jedis = new Jedis("localhost");
jedis.get("redis_key");
watch.stop();
log.info("jedis : {} ms", watch.getLastTaskInfo().getTimeMillis());
}
}
and the result is：
2020-08-14 17:02:28.236 INFO 21760 --- [ main] com.example.demo.JedisRunner : jedis : 27 ms
2020-08-14 17:02:33.318 INFO 21760 --- [ main] com.example.demo.LettuceRunner : lettuce : 4815 ms
Because Lettuce uses Netty and it spends the bulk of time initiating things in Netty.
Check the logs, as you can see, the bulk of the time spent is inside io.netty package:
2020-08-15 00:54:06.030 DEBUG 728 --- [ main] i.l.c.r.DefaultEventLoopGroupProvider : Creating executor io.netty.util.concurrent.DefaultEventExecutorGroup
2020-08-15 00:54:06.031 DEBUG 728 --- [ main] io.lettuce.core.RedisClient : Trying to get a Redis connection for: RedisURI [host='localhost', port=6379]
2020-08-15 00:54:06.120 DEBUG 728 --- [ main] io.lettuce.core.EpollProvider : Starting without optional epoll library
2020-08-15 00:54:06.122 DEBUG 728 --- [ main] io.lettuce.core.KqueueProvider : Starting without optional kqueue library
2020-08-15 00:54:06.123 DEBUG 728 --- [ main] i.l.c.r.DefaultEventLoopGroupProvider : Allocating executor io.netty.channel.nio.NioEventLoopGroup
2020-08-15 00:54:06.123 DEBUG 728 --- [ main] i.l.c.r.DefaultEventLoopGroupProvider : Creating executor io.netty.channel.nio.NioEventLoopGroup
2020-08-15 00:54:06.124 DEBUG 728 --- [ main] i.n.channel.MultithreadEventLoopGroup : -Dio.netty.eventLoopThreads: 12
2020-08-15 00:54:06.129 DEBUG 728 --- [ main] io.netty.channel.nio.NioEventLoop : -Dio.netty.noKeySetOptimization: false
2020-08-15 00:54:06.129 DEBUG 728 --- [ main] io.netty.channel.nio.NioEventLoop : -Dio.netty.selectorAutoRebuildThreshold: 512
2020-08-15 00:54:06.421 DEBUG 728 --- [ main] i.l.c.r.DefaultEventLoopGroupProvider : Adding reference to io.netty.channel.nio.NioEventLoopGroup#7c59cf66, existing ref count 0
2020-08-15 00:54:06.431 DEBUG 728 --- [ main] io.lettuce.core.RedisClient : Resolved SocketAddress localhost:6379 using RedisURI [host='localhost', port=6379]
2020-08-15 00:54:06.432 DEBUG 728 --- [ main] io.lettuce.core.RedisClient : Connecting to Redis at localhost:6379
2020-08-15 00:54:06.435 DEBUG 728 --- [ main] io.netty.channel.DefaultChannelId : -Dio.netty.processId: 728 (auto-detected)
2020-08-15 00:54:06.437 DEBUG 728 --- [ main] io.netty.util.NetUtil : -Djava.net.preferIPv4Stack: false
2020-08-15 00:54:06.437 DEBUG 728 --- [ main] io.netty.util.NetUtil : -Djava.net.preferIPv6Addresses: false
2020-08-15 00:54:06.659 DEBUG 728 --- [ main] io.netty.util.NetUtil : Loopback interface: lo (Software Loopback Interface 1, 127.0.0.1)
2020-08-15 00:54:06.660 DEBUG 728 --- [ main] io.netty.util.NetUtil : Failed to get SOMAXCONN from sysctl and file \proc\sys\net\core\somaxconn. Default: 200
2020-08-15 00:54:06.898 DEBUG 728 --- [ main] io.netty.channel.DefaultChannelId : -Dio.netty.machineId: 00:50:56:ff:fe:c0:00:08 (auto-detected)
2020-08-15 00:54:06.911 DEBUG 728 --- [ main] io.netty.buffer.ByteBufUtil : -Dio.netty.allocator.type: pooled
2020-08-15 00:54:06.912 DEBUG 728 --- [ main] io.netty.buffer.ByteBufUtil : -Dio.netty.threadLocalDirectBufferSize: 0
2020-08-15 00:54:06.912 DEBUG 728 --- [ main] io.netty.buffer.ByteBufUtil : -Dio.netty.maxThreadLocalCharBufferSize: 16384
2020-08-15 00:54:06.928 DEBUG 728 --- [ioEventLoop-8-1] io.netty.util.Recycler : -Dio.netty.recycler.maxCapacityPerThread: 4096
2020-08-15 00:54:06.928 DEBUG 728 --- [ioEventLoop-8-1] io.netty.util.Recycler : -Dio.netty.recycler.maxSharedCapacityFactor: 2
2020-08-15 00:54:06.928 DEBUG 728 --- [ioEventLoop-8-1] io.netty.util.Recycler : -Dio.netty.recycler.linkCapacity: 16
2020-08-15 00:54:06.928 DEBUG 728 --- [ioEventLoop-8-1] io.netty.util.Recycler : -Dio.netty.recycler.ratio: 8
2020-08-15 00:54:06.928 DEBUG 728 --- [ioEventLoop-8-1] io.netty.util.Recycler : -Dio.netty.recycler.delayedQueue.ratio: 8
2020-08-15 00:54:06.933 DEBUG 728 --- [ioEventLoop-8-1] io.netty.buffer.AbstractByteBuf : -Dio.netty.buffer.checkAccessible: true
2020-08-15 00:54:06.933 DEBUG 728 --- [ioEventLoop-8-1] io.netty.buffer.AbstractByteBuf : -Dio.netty.buffer.checkBounds: true
2020-08-15 00:54:06.933 DEBUG 728 --- [ioEventLoop-8-1] i.n.util.ResourceLeakDetectorFactory : Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector#20e9fc6c
2020-08-15 00:54:06.950 DEBUG 728 --- [ioEventLoop-8-1] io.lettuce.core.protocol.CommandHandler : [channel=0x1ced470d, [id: 0x7bd077d9] (inactive), chid=0x1] channelRegistered()
2020-08-15 00:54:06.953 DEBUG 728 --- [ioEventLoop-8-1] io.lettuce.core.protocol.CommandHandler : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, chid=0x1] channelActive()
2020-08-15 00:54:06.954 DEBUG 728 --- [ioEventLoop-8-1] i.lettuce.core.protocol.DefaultEndpoint : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, epid=0x1] activateEndpointAndExecuteBufferedCommands 0 command(s) buffered
2020-08-15 00:54:06.954 DEBUG 728 --- [ioEventLoop-8-1] i.lettuce.core.protocol.DefaultEndpoint : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, epid=0x1] activating endpoint
2020-08-15 00:54:06.954 DEBUG 728 --- [ioEventLoop-8-1] i.lettuce.core.protocol.DefaultEndpoint : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, epid=0x1] flushCommands()
2020-08-15 00:54:06.954 DEBUG 728 --- [ioEventLoop-8-1] i.lettuce.core.protocol.DefaultEndpoint : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, epid=0x1] flushCommands() Flushing 0 commands
2020-08-15 00:54:06.954 DEBUG 728 --- [ioEventLoop-8-1] i.l.core.protocol.ConnectionWatchdog : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, last known addr=localhost/127.0.0.1:6379] channelActive()
2020-08-15 00:54:06.954 DEBUG 728 --- [ioEventLoop-8-1] io.lettuce.core.protocol.CommandHandler : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, chid=0x1] channelActive() done
2020-08-15 00:54:06.955 DEBUG 728 --- [ioEventLoop-8-1] io.lettuce.core.RedisClient : Connecting to Redis at localhost:6379: Success
2020-08-15 00:54:06.956 INFO 728 --- [ main] c.h.s.c.c.CacheStudyApplicationTests : lettuce : 925 ms
2020-08-15 00:54:06.956 DEBUG 728 --- [ main] io.lettuce.core.RedisChannelHandler : close()
2020-08-15 00:54:06.956 DEBUG 728 --- [ main] io.lettuce.core.RedisChannelHandler : closeAsync()
2020-08-15 00:54:06.956 DEBUG 728 --- [ main] i.lettuce.core.protocol.DefaultEndpoint : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, epid=0x1] closeAsync()
2020-08-15 00:54:06.957 DEBUG 728 --- [ioEventLoop-8-1] i.l.core.protocol.ConnectionWatchdog : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, last known addr=localhost/127.0.0.1:6379] userEventTriggered(ctx, io.lettuce.core.ConnectionEvents$Activated#1cda757f)
2020-08-15 00:54:06.958 DEBUG 728 --- [ main] io.lettuce.core.RedisClient : Initiate shutdown (0, 2, SECONDS)
2020-08-15 00:54:06.959 DEBUG 728 --- [ioEventLoop-8-1] io.lettuce.core.protocol.CommandHandler : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, chid=0x1] channelInactive()
2020-08-15 00:54:06.959 DEBUG 728 --- [ioEventLoop-8-1] i.lettuce.core.protocol.DefaultEndpoint : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, epid=0x1] deactivating endpoint handler
2020-08-15 00:54:06.960 DEBUG 728 --- [ioEventLoop-8-1] io.lettuce.core.protocol.CommandHandler : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, chid=0x1] channelInactive() done
2020-08-15 00:54:06.960 DEBUG 728 --- [ioEventLoop-8-1] i.l.core.protocol.ConnectionWatchdog : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, last known addr=localhost/127.0.0.1:6379] channelInactive()
2020-08-15 00:54:06.960 DEBUG 728 --- [ioEventLoop-8-1] i.l.core.protocol.ConnectionWatchdog : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, last known addr=localhost/127.0.0.1:6379] Reconnect scheduling disabled
2020-08-15 00:54:06.960 DEBUG 728 --- [ioEventLoop-8-1] io.lettuce.core.protocol.CommandHandler : [channel=0x1ced470d, /127.0.0.1:2106 -> localhost/127.0.0.1:6379, chid=0x1] channelUnregistered()
2020-08-15 00:54:06.961 DEBUG 728 --- [ main] i.l.c.resource.DefaultClientResources : Initiate shutdown (0, 2, SECONDS)
2020-08-15 00:54:06.963 DEBUG 728 --- [ main] i.l.c.r.DefaultEventLoopGroupProvider : Initiate shutdown (0, 2, SECONDS)
2020-08-15 00:54:06.963 DEBUG 728 --- [ main] i.l.c.r.DefaultEventLoopGroupProvider : Release executor io.netty.channel.nio.NioEventLoopGroup#7c59cf66
2020-08-15 00:54:06.965 DEBUG 728 --- [ioEventLoop-8-1] io.netty.buffer.PoolThreadCache : Freed 1 thread-local buffer(s) from thread: lettuce-nioEventLoop-8-1
After starting installing spinnaker on Oracle Kubernates engine(halyard on docker), we are unable to get all spinnaker pods in Ready state as below:
$ kubectl get pod -n spinnaker
NAME READY STATUS RESTARTS AGE
spin-clouddriver-6795648f5c-7695g 0/1 Running 0 10m
spin-deck-6766bfb78-ttqff 1/1 Running 0 10m
spin-echo-76d84bccf4-wqgq4 1/1 Running 0 10m
spin-front50-6fdc874759-jtrfp 1/1 Running 0 6m
spin-gate-6ffdfd7657-xhjh2 0/1 Running 0 10m
spin-igor-6c564fc765-589cz 1/1 Running 0 10m
spin-orca-567879df75-4t2gn 0/1 Running 0 10m
spin-redis-5bd65c56b6-82rxp 1/1 Running 0 10m
spin-rosco-7cdf5858fc-n5vrx 0/1 Running 0 9m
Please find the logs for the pods which are not ready and please help me with any clue to resolve the issue.
LMK in case you need more logs/info.
Also I am able to get the UI of spinnaker but its not able to create anything
Logs:
**************spin-clouddriver logs**************
#################################################
----------------------------------------------------------------
$ kubectl log -n spinnaker spin-clouddriver-6795648f5c-7695g|grep -i error
log is DEPRECATED and will be removed in a future version. Use logs instead.
2018-09-14 07:35:13.842 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)
----------------------------------------------------------------
$ kubectl log -n spinnaker spin-clouddriver-6795648f5c-7695g|tail -20
log is DEPRECATED and will be removed in a future version. Use logs instead.
2018-09-14 07:35:17.245 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/heapdump || /heapdump.json],methods=[GET],produces=[application/octet-stream]}" onto public void org.springframework.boot.actuate.endpoint.mvc.HeapdumpMvcEndpoint.invoke(boolean,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException,javax.servlet.ServletException
2018-09-14 07:35:17.247 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/actuator || /actuator.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public org.springframework.hateoas.ResourceSupport org.springframework.boot.actuate.endpoint.mvc.HalJsonMvcEndpoint.links()
2018-09-14 07:35:17.259 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/trace || /trace.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.276 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/dump || /dump.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.277 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/mappings || /mappings.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.277 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/autoconfig || /autoconfig.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.303 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/env/{name:.*}],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EnvironmentMvcEndpoint.value(java.lang.String)
2018-09-14 07:35:17.304 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/env || /env.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.305 INFO 1 --- [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped "{[/auditevents || /auditevents.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public org.springframework.http.ResponseEntity<?> org.springframework.boot.actuate.endpoint.mvc.AuditEventsMvcEndpoint.findByPrincipalAndAfterAndType(java.lang.String,java.util.Date,java.lang.String)
2018-09-14 07:35:19.677 INFO 1 --- [ main] c.n.s.c.security.ProviderUtils : Adding accounts [my-k8s-v2-account] of type KubernetesNamedAccountCredentials...
2018-09-14 07:37:04.151 WARN 1 --- [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:37:04.154 WARN 1 --- [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : There are no namespaces configured (or loadable) -- please check that the list of 'omitNamespaces' for account 'my-k8s-v2-account' doesn't prevent access from all namespaces in this cluster, or that the cluster is reachable.
2018-09-14 07:38:47.351 WARN 1 --- [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:40:30.099 WARN 1 --- [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:42:12.815 WARN 1 --- [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
**************spin-gate logs**************
##########################################
----------------------------------------------------------------
$ kubectl -n spinnaker logs spin-gate-6ffdfd7657-xhjh2|grep -i error
2018-09-14 07:35:35.599 INFO 1 --- [ost-startStop-1] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)
2018-09-14 07:35:42.556 WARN 1 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'enableRedisKeyspaceNotificationsInitializer' defined in class path resource [com/netflix/spinnaker/gate/config/GateConfig.class]: Invocation of init method failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2018-09-14 07:35:42.643 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'enableRedisKeyspaceNotificationsInitializer' defined in class path resource [com/netflix/spinnaker/gate/config/GateConfig.class]: Invocation of init method failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
2018-09-14 07:36:38.056 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#661f6038 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:37:38.065 ERROR 1 --- [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#1df18e11 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:38:38.066 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#6e311e73 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:39:38.067 ERROR 1 --- [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#1709e561 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:40:38.069 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#533017fd rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:41:38.070 ERROR 1 --- [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#431be4b2 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:42:38.071 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#6fc06d32 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
----------------------------------------------------------------
$ kubectl -n spinnaker logs spin-gate-6ffdfd7657-xhjh2|tail -40
Caused by: org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:204) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:348) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
at org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration$EnableRedisKeyspaceNotificationsInitializer.afterPropertiesSet(RedisHttpSessionConfiguration.java:249) ~[spring-session-1.3.1.RELEASE.jar:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687) ~[spring-beans-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624) ~[spring-beans-4.3.14.RELEASE.jar:4.3.14.RELEASE]
... 19 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16) ~[jedis-2.9.0.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
... 23 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out
at redis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.0.jar:na]
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:888) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.4.3.jar:2.4.3]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
... 26 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_171]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_171]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_171]
at redis.clients.jedis.Connection.connect(Connection.java:184) ~[jedis-2.9.0.jar:na]
... 33 common frames omitted
2018-09-14 07:36:38.056 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#661f6038 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:37:38.065 ERROR 1 --- [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#1df18e11 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:38:38.066 ERROR 1 --- [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask#6e311e73 rejected from java.util.concurrent.ThreadPoolExecutor#18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
**************spin-orca logs**************
##########################################
--------------------------------------------------------------------
$ kubectl -n spinnaker logs spin-orca-567879df75-4t2gn|grep -i error
2018-09-14 07:43:56.069 ERROR 1 --- [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.080 ERROR 1 --- [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.090 ERROR 1 --- [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.101 ERROR 1 --- [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.111 ERROR 1 --- [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
..............
..............
2018-09-14 07:43:59.065 ERROR 1 --- [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.075 ERROR 1 --- [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.086 ERROR 1 --- [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.097 ERROR 1 --- [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.110 ERROR 1 --- [ scheduler-3] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: orca.task.queue:buffered:pipeline
at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:118)
2018-09-14 07:43:59.110 ERROR 1 --- [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.120 ERROR 1 --- [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.131 ERROR 1 --- [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
----------------------------------------------------------------
$ kubectl -n spinnaker logs spin-orca-567879df75-4t2gn|tail -40
2018-09-14 07:45:04.126 ERROR 1 --- [ scheduler-6] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
at com.netflix.spinnaker.q.redis.RedisQueue.poll(RedisQueue.kt:98)
at com.netflix.spinnaker.q.QueueProcessor.pollOnce(QueueProcessor.kt:76)
at com.netflix.spinnaker.q.QueueProcessor.access$pollOnce(QueueProcessor.kt:37)
at com.netflix.spinnaker.q.QueueProcessor$poll$1.invoke(QueueProcessor.kt:62)
at com.netflix.spinnaker.q.QueueProcessor$poll$1.invoke(QueueProcessor.kt:37)
at com.netflix.spinnaker.q.QueueProcessor.ifEnabled(QueueProcessor.kt:119)
at com.netflix.spinnaker.q.QueueProcessor.poll(QueueProcessor.kt:57)
at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.UnknownHostException: spin-redis.spinnaker
at redis.clients.jedis.Connection.connect(Connection.java:207)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93)
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:888)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 21 common frames omitted
Caused by: java.net.UnknownHostException: spin-redis.spinnaker
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at redis.clients.jedis.Connection.connect(Connection.java:184)
... 28 common frames omitted
**************spin-rosco logs**************
###########################################
---------------------------------------------------------------------
$ kubectl -n spinnaker logs spin-rosco-7cdf5858fc-n5vrx|grep -i error
2018-09-14 07:36:12.914 INFO 1 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)
2018-09-14 07:36:24.195 ERROR 1 --- [readScheduler-2] c.n.spinnaker.rosco.executor.BakePoller : Zombie Killer Polling Error:
2018-09-14 07:36:24.195 ERROR 1 --- [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:34.089 ERROR 1 --- [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:49.089 ERROR 1 --- [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:49.099 ERROR 1 --- [readScheduler-2] c.n.spinnaker.rosco.executor.BakePoller : Zombie Killer Polling Error:
----------------------------------------------------------------
$ kubectl -n spinnaker logs spin-rosco-7cdf5858fc-n5vrx|tail -40
at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.callGetProperty(GetEffectivePogoPropertySite.java:48) ~[groovy-all-2.4.13.jar:2.4.13]
at com.netflix.spinnaker.rosco.executor.BakePoller$_onApplicationEvent_closure1.doCall(BakePoller.groovy:81) [rosco-core-0.108.0-SNAPSHOT.jar:0.108.0-SNAPSHOT]
at com.netflix.spinnaker.rosco.executor.BakePoller$_onApplicationEvent_closure1.doCall(BakePoller.groovy) [rosco-core-0.108.0-SNAPSHOT.jar:0.108.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.Closure.call(Closure.java:414) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124) [groovy-all-2.4.13.jar:2.4.13]
at com.sun.proxy.$Proxy116.call(Unknown Source) [na:na]
at rx.Scheduler$Worker$1.call(Scheduler.java:120) [rxjava-1.0.16.jar:1.0.16]
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.0.16.jar:1.0.16]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.UnknownHostException: spin-redis.spinnaker
at redis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.0.jar:na]
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.4.2.jar:2.4.2]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
... 32 common frames omitted
Caused by: java.net.UnknownHostException: spin-redis.spinnaker
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[na:1.8.0_171]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_171]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_171]
at redis.clients.jedis.Connection.connect(Connection.java:184) ~[jedis-2.9.0.jar:na]
... 39 common frames omitted
I had the same issue with K8s V2 provider and I figured out that the main cause was an HAL misconfiguration. I just put the providers > kubernetes > [account] > serviceAccount to true. This was the trick.
If it is your case the configuration command is:
hal config provider kubernetes account edit <account-name> --service-account true
...then re-deploy Spinnaker via hal deploy apply
Hope this helps you. Good luck. :)
When I run my Spring Cloud Config Client project config-client, I found these error:
2018-02-09 10:31:09.885 INFO 13933 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at: http://localhost:8888
2018-02-09 10:31:10.022 WARN 13933 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/config-client/dev/master": 拒绝连接 (Connection refused); nested exception is java.net.ConnectException: 拒绝连接 (Connection refused)
2018-02-09 10:31:10.026 INFO 13933 --- [ main] c.y.c.ConfigClientApplication : No active profile set, falling back to default profiles: default
2018-02-09 10:31:10.040 INFO 13933 --- [ main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#33b1c5c5: startup date [Fri Feb 09 10:31:10 CST 2018]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext#1ffe63b9
2018-02-09 10:31:10.419 INFO 13933 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=65226c2b-524f-3b14-8e17-9fdbc9f72d85
2018-02-09 10:31:10.471 INFO 13933 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$25380e89] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2018-02-09 10:31:10.688 INFO 13933 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 10001 (http)
2018-02-09 10:31:10.697 INFO 13933 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2018-02-09 10:31:10.698 INFO 13933 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet Engine: Apache Tomcat/8.5.27
2018-02-09 10:31:10.767 INFO 13933 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2018-02-09 10:31:10.768 INFO 13933 --- [ost-startStop-1] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 727 ms
2018-02-09 10:31:10.861 INFO 13933 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean : Mapping servlet: 'dispatcherServlet' to [/]
2018-02-09 10:31:10.864 INFO 13933 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'characterEncodingFilter' to: [/*]
2018-02-09 10:31:10.864 INFO 13933 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2018-02-09 10:31:10.864 INFO 13933 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2018-02-09 10:31:10.865 INFO 13933 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean : Mapping filter: 'requestContextFilter' to: [/*]
2018-02-09 10:31:10.895 WARN 13933 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configClientApplication': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'content' in value "${content}"
2018-02-09 10:31:10.896 INFO 13933 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2018-02-09 10:31:10.914 INFO 13933 --- [ main] utoConfigurationReportLoggingInitializer :
Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2018-02-09 10:31:10.923 ERROR 13933 --- [ main] o.s.boot.SpringApplication : Application startup failed
Apparently, the config server is wrong. However, the Spring Cloud Config Server is running at localhost:10000/ and application.yml of the project(config-client) is below. Why the spring.cloud.config.uri doesn't work?
application.yml [config-client]
server:
port: 10001
spring:
application:
name: config-client
cloud:
config:
label: master
profile: dev
uri: http://localhost:10000
Fur future readers, as answered here, when using Spring Cloud Config Server, we should specify basic bootstrap settings such as : spring.application.name and spring.cloud.config.uri inside bootstrap.yml (or "bootstrap.properties").
Upon startup, Spring Cloud makes an HTTP call to the config server with the name of the application and retrieves back that application's configuration.
That's said, since we're externalizing our settings using Spring Cloud Config Server, any default configurations defined in application.yml (or "application.properties") will be overridden during the bootstrap process upon startup.
IntelliJ Users: add the following override parameter in the run/Debug Configuration:
Name: spring.cloud.config.uri
Value: http://your-server-here/config-server
you can load configuration servers before starting the Application, using bootstrap.yml
just add configuration server and application name
spring:
application:
name: clientTest
cloud:
config:
uri: http://localhost:8889
enabled: true
fail-fast: true
if we are using bootstrap.properties. we have to include this dependency in pom for spring-2.4.0+
agregado para evitar un error al usar spring mayor que 2.4.0
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-bootstrap</artifactId>
</dependency>
in my case i was testing spring consul, which usually runs in 8500, i saw a different port in the log. Found that the different port is due to following deplendency of spring cloud. Hence i just have to remove it.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
I had a Turbine and Hystrix setup working, but decided to change it over to Turbine AMQP so I could aggregate multiple services into one stream/dashboard.
I have set up a Turbine AMQP server running on localhost:8989, but it doesn't appear to be getting Hystrix data from the client service. When I hit the Turbine server's IP in my browser, I see data: {"type":"Ping"} repeatedly, even while I am polling the URL of the Hystrix. If I attempt to show the Turbine AMQP stream in the Hystrix Dashboard, I get: Unable to connect to Command Metric Stream.
I have a default install of RabbitMQ running on port 5672.
My client service using Hystrix-AMQP has a application.yml file that looks like so:
spring:
application:
name: policy-service
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
spring:
rabbitmq:
addresses: ${vcap.services.${PREFIX:}rabbitmq.credentials.uri:amqp://${RABBITMQ_HOST:localhost}:${RABBITMQ_PORT:5672}}
The tail end of the startup log looks like this:
2015-09-14 16:31:13.030 INFO 52844 --- [ main] com.netflix.discovery.DiscoveryClient : Starting heartbeat executor: renew interval is: 30
2015-09-14 16:31:13.047 INFO 52844 --- [ main] c.n.e.EurekaDiscoveryClientConfiguration : Registering application policy-service with eureka with status UP
2015-09-14 16:31:13.194 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.i.channel.PublishSubscribeChannel : Channel 'policy-service:8088.errorChannel' has 1 subscriber(s).
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started _org.springframework.integration.errorLogger
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {filter} as a subscriber to the 'cloudBusOutboundFlow.channel#0' channel
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy- service:8088.cloudBusOutboundFlow.channel#0' has 1 subscriber(s).
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {filter} as a subscriber to the 'cloudBusInboundChannel' channel
2015-09-14 16:31:13.195 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.cloudBusInboundChannel' has 1 subscriber(s).
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#1
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {message-handler} as a subscriber to the 'cloudBusInboundFlow.channel#0' channel
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.cloudBusInboundFlow.channel#0' has 1 subscriber(s).
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#2
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter} as a subscriber to the 'cloudBusWiretapChannel' channel
2015-09-14 16:31:13.196 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.cloudBusWiretapChannel' has 1 subscriber(s).
2015-09-14 16:31:13.197 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#3
2015-09-14 16:31:13.197 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {amqp:outbound-channel-adapter} as a subscriber to the 'cloudBusOutboundChannel' channel
2015-09-14 16:31:13.197 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.cloudBusOutboundChannel' has 1 subscriber(s).
2015-09-14 16:31:13.197 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#4
2015-09-14 16:31:13.198 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {bridge} as a subscriber to the 'cloudBusAmqpInboundFlow.channel#0' channel
2015-09-14 16:31:13.198 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.cloudBusAmqpInboundFlow.channel#0' has 1 subscriber(s).
2015-09-14 16:31:13.198 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#5
2015-09-14 16:31:13.198 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {amqp:outbound-channel-adapter} as a subscriber to the 'hystrixStream' channel
2015-09-14 16:31:13.199 INFO 52844 --- [ main] o.s.integration.channel.DirectChannel : Channel 'policy-service:8088.hystrixStream' has 1 subscriber(s).
2015-09-14 16:31:13.199 INFO 52844 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#6
2015-09-14 16:31:13.219 INFO 52844 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 1073741823
2015-09-14 16:31:13.219 INFO 52844 --- [ main] ApplicationEventListeningMessageProducer : started org.springframework.integration.event.inbound.ApplicationEventListeningMessageProducer#0
2015-09-14 16:31:13.555 INFO 52844 --- [cTaskExecutor-1] o.s.amqp.rabbit.core.RabbitAdmin : Auto-declaring a non-durable, auto-delete, or exclusive Queue (4640c1c8-ff8f-45d7-8426-19d1b7a4cdb0) durable:false, auto-delete:true, exclusive:true. It will be redeclared if the broker stops and is restarted while the connection factory is alive, but all messages will be lost.
2015-09-14 16:31:13.572 INFO 52844 --- [ main] o.s.i.a.i.AmqpInboundChannelAdapter : started org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter#0
2015-09-14 16:31:13.573 INFO 52844 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 2147483647
2015-09-14 16:31:13.576 INFO 52844 --- [ main] c.n.h.c.m.e.HystrixMetricsPoller : Starting HystrixMetricsPoller
2015-09-14 16:31:13.609 INFO 52844 --- [ main] ration$HystrixMetricsPollerConfiguration : Starting poller
2015-09-14 16:31:13.803 INFO 52844 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8088 (http)
2015-09-14 16:31:13.805 INFO 52844 --- [ main] com.ml.springboot.PolicyService : Started PolicyService in 22.544 seconds (JVM running for 23.564)
So it looks like PolicyService successfully connects to the message broker.
The Turbine AMQP server's end of log:
2015-09-14 16:58:05.887 INFO 51944 --- [ main] i.reactivex.netty.server.AbstractServer : Rx server started at port: 8989
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.i.channel.PublishSubscribeChannel : Channel 'bootstrap:-1.errorChannel' has 1 subscriber(s).
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started _org.springframework.integration.errorLogger
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.i.endpoint.EventDrivenConsumer : Adding {bridge} as a subscriber to the 'hystrixStreamAggregatorInboundFlow.channel#0' channel
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.integration.channel.DirectChannel : Channel 'bootstrap:-1.hystrixStreamAggregatorInboundFlow.channel#0' has 1 subscriber(s).
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.i.endpoint.EventDrivenConsumer : started org.springframework.integration.config.ConsumerEndpointFactoryBean#0
2015-09-14 16:58:05.991 INFO 51944 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 1073741823
2015-09-14 16:58:06.238 INFO 51944 --- [cTaskExecutor-1] o.s.amqp.rabbit.core.RabbitAdmin : Auto-declaring a non-durable, auto-delete, or exclusive Queue (spring.cloud.hystrix.stream) durable:false, auto-delete:false, exclusive:false. It will be redeclared if the broker stops and is restarted while the connection factory is alive, but all messages will be lost.
2015-09-14 16:58:06.289 INFO 51944 --- [ main] o.s.i.a.i.AmqpInboundChannelAdapter : started org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter#0
2015-09-14 16:58:06.290 INFO 51944 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 2147483647
2015-09-14 16:58:06.434 INFO 51944 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): -1 (http)
Any ideas why the Turbine AMQP server is not receiving communication from the Hystrix AMQP client?
EDIT: Turbine-AMQP main looks like:
package com.turbine.amqp;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.netflix.turbine.amqp.EnableTurbineAmqp;
import org.springframework.context.annotation.Configuration;
#Configuration
#EnableAutoConfiguration
#EnableTurbineAmqp
#EnableDiscoveryClient
public class TurbineAmqpApplication {
public static void main(String[] args) {
SpringApplication.run(TurbineAmqpApplication.class, args);
}
}
Here's its application.yml:
server:
port: 8989
spring:
rabbitmq:
addresses: ${vcap.services.${PREFIX:}rabbitmq.credentials.uri:amqp://${RABBITMQ_HOST:localhost}:${RABBITMQ_PORT:5672}}
Hitting http://localhost:8989/turbine.stream produces a repeating stream of data: {"type":"Ping"}
and shows this in console:
2015-09-15 08:54:37.960 INFO 83480 --- [o-eventloop-3-1] o.s.c.n.t.amqp.TurbineAmqpConfiguration : SSE Request Received
2015-09-15 08:54:38.025 INFO 83480 --- [o-eventloop-3-1] o.s.c.n.t.amqp.TurbineAmqpConfiguration : Starting aggregation
EDIT: The below exception is thrown when I stop listening to the turbine stream, not when I try to listen with the dashboard.
2015-09-15 08:56:47.934 INFO 83480 --- [o-eventloop-3-3] o.s.c.n.t.amqp.TurbineAmqpConfiguration : SSE Request Received
2015-09-15 08:56:47.946 WARN 83480 --- [o-eventloop-3-3] io.netty.channel.DefaultChannelPipeline : An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.NoSuchMethodError: rx.Observable.collect(Lrx/functions/Func0;Lrx/functions/Action2;)Lrx/Observable;
at com.netflix.turbine.aggregator.StreamAggregator.lambda$null$36(StreamAggregator.java:89)
at rx.internal.operators.OnSubscribeMulticastSelector.call(OnSubscribeMulticastSelector.java:60)
at rx.internal.operators.OnSubscribeMulticastSelector.call(OnSubscribeMulticastSelector.java:40)
at rx.Observable.unsafeSubscribe(Observable.java:8591)
at rx.internal.operators.OperatorMerge$MergeSubscriber.handleNewSource(OperatorMerge.java:190)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:160)
at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:96)
at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54)
at rx.internal.operators.OperatorGroupBy$GroupBySubscriber.onNext(OperatorGroupBy.java:173)
at rx.subjects.SubjectSubscriptionManager$SubjectObserver.onNext(SubjectSubscriptionManager.java:224)
at rx.subjects.PublishSubject.onNext(PublishSubject.java:101)
at org.springframework.cloud.netflix.turbine.amqp.Aggregator.handle(Aggregator.java:53)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:112)
at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:102)
at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:342)
at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:131)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:330)
at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:164)
at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:276)
at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:142)
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:75)
at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:71)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:99)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:277)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:239)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:95)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:248)
at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:171)
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:119)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:105)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:78)
at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:101)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:277)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:239)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:95)
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:101)
at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter.access$400(AmqpInboundChannelAdapter.java:45)
at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$1.onMessage(AmqpInboundChannelAdapter.java:93)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:756)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1241)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:660)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1005)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:989)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1103)
at java.lang.Thread.run(Thread.java:745)
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: GroupedObservable.class
at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:98)
at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:56)
... 58 common frames omitted
My dependencies for turbine-amqp are as follows:
dependencies {
compile('org.springframework.cloud:spring-cloud-starter-turbine-amqp:1.0.3.RELEASE')
compile 'org.springframework.boot:spring-boot-starter-web:1.2.5.RELEASE'
compile 'org.springframework.boot:spring-boot-starter-actuator:1.2.5.RELEASE'
testCompile("org.springframework.boot:spring-boot-starter-test")
}
dependencyManagement {
imports {
mavenBom 'org.springframework.cloud:spring-cloud-starter-parent:1.0.2.RELEASE'
}
}
It is so hard to find a solution.
Using Spring cloud 2.1.4.RELEASE I faced with similar problem.
The main cause is the incompatibility [exchanges] name in rabbitMQ between:
spring-cloud-netflix-hystrix-stream and spring-cloud-starter-netflix-turbine-stream.
So solve it:
See the name created exchange name when you start the service componente {the same that declare hystrix-stream}
on the componente that declare {turbine-stream}
update the property
turbine.stream.destination=
in my case
turbine.stream.destination=hystrixStreamOutput
I faced with similar problem and I find a solution.
My Spring Cloud version is 2.1.0.RELEASE
The solution:
add property
spring.cloud.stream.bindings.turbineStreamInput.destination: hystrixStreamOutput
turbine.stream.enabled: false
add auto configuration
#EnableBinding(TurbineStreamClient.class)
public class TurbineStreamAutoConfiguration {
#Autowired
private BindingServiceProperties bindings;
#Autowired
private TurbineStreamProperties properties;
#PostConstruct
public void init() {
BindingProperties inputBinding = this.bindings.getBindings()
.get(TurbineStreamClient.INPUT);
if (inputBinding == null) {
this.bindings.getBindings().put(TurbineStreamClient.INPUT,
new BindingProperties());
}
BindingProperties input = this.bindings.getBindings()
.get(TurbineStreamClient.INPUT);
if (input.getDestination() == null) {
input.setDestination(this.properties.getDestination());
}
if (input.getContentType() == null) {
input.setContentType(this.properties.getContentType());
}
}
#Bean
public HystrixStreamAggregator hystrixStreamAggregator(ObjectMapper mapper,
PublishSubject<Map<String, Object>> publisher) {
return new HystrixStreamAggregator(mapper, publisher);
}
}