Docker - drone.io drone-agent installation getting errors - drone.io

I am new to Drone ci/cd while installing getting the below issues
and I am stuck at starting point. See below the installation steps I followed....
I've installed docker and docker compose. Below is the docker file which I used to configure cat docker-compose.yml version: '3'.
All this is configured at my Oracle VM Virtual Box.
services:
drone-server:
image: drone/drone:0.5
ports:
- 127.0.0.1:8000:8000
volumes:
- /var/lib/drone:/var/lib/drone
restart: always
env_file:
- /etc/drone-ci/server.env
drone-agent:
image: drone/drone:0.5
command: agent
depends_on:
- drone-server
volumes:
- /var/run/docker.sock:/var/run/docker.sock
restart: always
env_file:
- /etc/drone-ci/agent.env
Drone agent.env config file
DRONE_SECRET=5Kt8P0I4uLuzdVTEpC4TxzwineaM8EBxrIZVXriENxNiXvPnsoWm9AMD4Slfybbof
DRONE_SERVER=ws://192.168.200.180/ws/broker
server.env
DRONE_SECRET=5Kt8P0I4uLuzdVTEpC4TxzwineaM8EBxrIZVXriENxNiXvPnsoWm9AMD4Slfybbof
DRONE_HOST=http://192.168.200.180
DRONE_OPEN=true
DRONE_GITHUB=true
DRONE_GITHUB_CLIENT=93fc7a0e4f098b289de8
DRONE_GITHUB_SECRET=e2ce2dbbdd10d6166ffd6eac99b6818a33d53d8d
DRONE_ADMIN=sravan1265
##########################################################################
While i ran this got the below output:
docker-compose up
Recreating root_drone-server_1 ... done
Recreating root_drone-agent_1 ... done
Attaching to root_drone-server_1, root_drone-agent_1
drone-server_1 | time="2018-03-07T11:49:41Z" level=warning msg="agents can connect with token
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0ZXh0IjoiNUt0OFAwSTR1THV6ZFZURXBDNFR4endpbmVhTThFQnhySVpWWHJpRU54TmlYdlBuc29XbTlBTUQ0U2xmeWJib2YiLCJ0eXBlIjoiYWdlbnQifQ.nOOmxh7yttAAyq_NiDc182ZQKbQXWCYo8bbgIJ-0h20"
drone-server_1 | [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
drone-server_1 | - using env: export GIN_MODE=release
drone-server_1 | - using code: gin.SetMode(gin.ReleaseMode)
drone-server_1 |
drone-server_1 | [GIN-debug] POST /api/user/token --> github.com/drone/drone/server.PostToken (16 handlers)
drone-server_1 | [GIN-debug] DELETE /api/user/token --> github.com/drone/drone/server.DeleteToken (16 handlers)
drone-server_1 | [GIN-debug] GET /api/users --> github.com/drone/drone/server.GetUsers (16 handlers)
drone-server_1 | [GIN-debug] POST /api/users --> github.com/drone/drone/server.PostUser (16 handlers)
drone-server_1 | [GIN-debug] GET /api/users/:login --> github.com/drone/drone/server.GetUser (16 handlers)
drone-server_1 | [GIN-debug] PATCH /api/users/:login --> github.com/drone/drone/server.PatchUser (16 handlers)
github.com/drone/drone/server.PostTeamSecret (16 handlers)
drone-server_1 | [GIN-debug] DELETE /api/teams/:team/secrets/:secret --> github.com/drone/drone/server.DeleteTeamSecret (16 handlers)
drone-server_1 | [GIN-debug] GET /api/global/secrets --> github.com/drone/drone/server.GetGlobalSecrets (16 handlers)
drone-server_1 | [GIN-debug] POST /api/global/secrets --> github.com/drone/drone/server.PostGlobalSecret (16 handlers)
...................................../api/repos/:owner/:name/builds/:number --> github.com/drone/drone/server.GetBuild (18 handlers)
drone-server_1 | [GIN-debug] GET /api/repos/:owner/:name/logs/:number/:job --> github.com/drone/drone/server.GetBuildLogs (18 handlers)
drone-server_1 | [GIN-debug] POST /api/repos/:owner/:name/sign --> github.com/drone/drone/server.Sign (19 handlers)
drone-server_1 | [GIN-debug] GET /api/repos/:owner/:name/secrets --> github.com/drone/drone/server.GetSecrets (19 handlers)
drone-server_1 | [GIN-debug] POST /api/repos/:owner/:name/builds/:number --> github.com/drone/drone/server.PostBuild (19 handlers)
drone-server_1 | [GIN-debug] DELETE /api/repos/:owner/:name/builds/:number/:job --> github.com/drone/drone/server.DeleteBuild (19 handlers)
drone-server_1 | [GIN-debug] GET /api/badges/:owner/:name/status.svg --> github.com/drone/drone/server.GetBadge (15 handlers)
drone-server_1 | [GIN-debug] GET /api/badges/:owner/:name/cc.xml --> github.com/drone/drone/server.GetCC (15 handlers)
drone-server_1 | [GIN-debug] POST /hook --> github.com/drone/drone/server.PostHook (15 handlers)
drone-server_1 | [GIN-debug] POST /api/hook --> github.com/drone/drone/server.PostHook (15 handlers)
drone-server_1 | [GIN-debug] GET /ws/broker --> github.com/drone/drone/server.Broker (15 handlers)
drone-server_1 | [GIN-debug] GET /ws/feed --> github.com/drone/drone/server.EventStream (15 handlers)
drone-server_1 | [GIN-debug] GET /ws/logs/:owner/:name/:build/:number --> github.com/drone/drone/server.LogStream (18 handlers)
drone-server_1 | [GIN-debug] GET /authorize --> github.com/drone/drone/server.GetLogin (15 handlers)
drone-server_1 | [GIN-debug] POST /authorize --> github.com/drone/drone/server.GetLogin (15 handlers)
drone-server_1 | [GIN-debug] POST /authorize/token --> github.com/drone/drone/server.GetLoginToken (15 handlers)
drone-server_1 | [GIN-debug] GET /api/builds --> github.com/drone/drone/server.GetBuildQueue (16 handlers)
drone-server_1 | [GIN-debug] GET /api/debug/pprof/ --> github.com/drone/drone/server/debug.IndexHandler.func1 (16 handlers)
drone-server_1 | [GIN-debug] GET /api/debug/pprof/heap -->
drone-agent_1 | 1:M 07 Mar 11:49:41.579 * connecting to server ws://192.168.200.180/ws/broker
drone-agent_1 | 1:M 07 Mar 11:49:41.580 # connection failed, retry in 15s. websocket.Dial ws://192.168.200.180/ws/broker: dial tcp 192.168.200.180:80: getsockopt: connection refused
drone-agent_1 | 1:M 07 Mar 11:49:56.580 # connection failed, retry in 15s. websocket.Dial ws://192.168.200.180/ws/broker: dial tcp 192.168.200.180:80: getsockopt: connection refused
drone-agent_1 | 1:M 07 Mar 11:50:11.582 # connection failed, retry in 15s. websocket.Dial ws://192.168.200.180/ws/broker: dial tcp 1`enter code here`92.168.200.180:80: getsockopt: connection refused.

Related

java.io.IOException: Failed to bind error on ActiveMQ service start

I am configuring a new install of ActiveMQ 5.15.10 on a RHEL 7.7 AWS instance. When the process starts I get this error:
[ec2-user#ip-***-***-***-*** activemq]$ more data/activemq.log
2019-10-30 17:36:45,784 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1#4667ae56: startup date [Wed Oct 30 17:36:45 UTC 2019]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2019-10-30 17:36:46,568 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/opt/activemq/data/kahadb] | org.apache.activemq.broker.BrokerService | main
2019-10-30 17:36:46,628 | INFO | KahaDB is version 6 | org.apache.activemq.store.kahadb.MessageDatabase | main
2019-10-30 17:36:46,652 | INFO | PListStore:[/opt/activemq/data/***-***-***-***/tmp_storage] started | org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2019-10-30 17:36:46,821 | INFO | Apache ActiveMQ 5.15.10 (***-***-***-***, ID:ip-***-***-***-***.ec2.internal-36686-1572457006663-0:1) is starting | org.apache.activemq.broker.BrokerService | main
2019-10-30 17:36:46,849 | INFO | Listening for connections at: tcp://ip-***-***-***-***.ec2.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2019-10-30 17:36:46,851 | INFO | Connector openwire started | org.apache.activemq.broker.TransportConnector | main
2019-10-30 17:36:46,855 | INFO | Listening for connections at: amqp://ip-***-***-***-***.ec2.internal:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2019-10-30 17:36:46,856 | INFO | Connector amqp started | org.apache.activemq.broker.TransportConnector | main
2019-10-30 17:36:46,859 | INFO | Listening for connections at: stomp://ip-***-***-***-***.ec2.internal:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2019-10-30 17:36:46,861 | INFO | Connector stomp started | org.apache.activemq.broker.TransportConnector | main
2019-10-30 17:36:46,864 | INFO | Listening for connections at: mqtt://ip-***-***-***-***.ec2.internal:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2019-10-30 17:36:46,865 | INFO | Connector mqtt started | org.apache.activemq.broker.TransportConnector | main
2019-10-30 17:36:46,870 | INFO | Starting Jetty server | org.apache.activemq.transport.WebTransportServerSupport | main
2019-10-30 17:36:46,951 | INFO | Creating Jetty connector | org.apache.activemq.transport.WebTransportServerSupport | main
2019-10-30 17:36:47,043 | WARN | ServletContext#o.e.j.s.ServletContextHandler#7b420819{/,null,STARTING} has uncovered http methods for path: / | org.eclipse.jetty.security.SecurityHandler | main
2019-10-30 17:36:47,091 | INFO | Listening for connections at ws://ip-***-***-***-***.ec2.internal:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.ws.WSTransportServer | main
2019-10-30 17:36:47,092 | INFO | Connector ws started | org.apache.activemq.broker.TransportConnector | main
2019-10-30 17:36:47,093 | INFO | Apache ActiveMQ 5.15.10 (***-***-***-***, ID:ip-***-***-***-***.ec2.internal-36686-1572457006663-0:1) started | org.apache.activemq.broker.BrokerService | main
2019-10-30 17:36:47,094 | INFO | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2019-10-30 17:36:47,095 | WARN | Store limit is 102400 mb (current store usage is 0 mb). The data directory: /opt/activemq/data/kahadb only has 47858 mb of usable space. - resetting to maximum available disk space: 47858 mb | org.apache.activemq.broker.BrokerService | main
2019-10-30 17:36:47,096 | WARN | Temporary Store limit is 51200 mb (current store usage is 0 mb). The data directory: /opt/activemq/data only has 47858 mb of usable space. - resetting to maximum available disk space: 47858 mb | org.apache.activemq.broker.BrokerService | main
2019-10-30 17:36:47,726 | INFO | ActiveMQ WebConsole available at http://***-***-***-***:8161/ | org.apache.activemq.web.WebConsoleStarter | main
2019-10-30 17:36:47,726 | INFO | ActiveMQ Jolokia REST API available at http://***-***-***-***:8161/api/jolokia/ | org.apache.activemq.web.WebConsoleStarter | main
2019-10-30 17:36:48,001 | WARN | Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.io.IOException: Failed to bind to /***-***-***-***:8161 | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2019-10-30 17:36:48,013 | INFO | Apache ActiveMQ 5.15.10 (***-***-***-***, ID:ip-***-***-***-***.ec2.internal-36686-1572457006663-0:1) is shutting down | org.apache.activemq.broker.BrokerService | main
2019-10-30 17:36:48,014 | INFO | Connector openwire stopped | org.apache.activemq.broker.TransportConnector | main
2019-10-30 17:36:48,015 | INFO | Connector amqp stopped | org.apache.activemq.broker.TransportConnector | main
2019-10-30 17:36:48,016 | INFO | Connector stomp stopped | org.apache.activemq.broker.TransportConnector | main
2019-10-30 17:36:48,017 | INFO | Connector mqtt stopped | org.apache.activemq.broker.TransportConnector | main
2019-10-30 17:36:48,021 | INFO | Connector ws stopped | org.apache.activemq.broker.TransportConnector | main
2019-10-30 17:36:48,024 | INFO | PListStore:[/opt/activemq/data/***-***-***-***/tmp_storage] stopped | org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2019-10-30 17:36:48,025 | INFO | Stopping async queue tasks | org.apache.activemq.store.kahadb.KahaDBStore | main
2019-10-30 17:36:48,025 | INFO | Stopping async topic tasks | org.apache.activemq.store.kahadb.KahaDBStore | main
2019-10-30 17:36:48,026 | INFO | Stopped KahaDB | org.apache.activemq.store.kahadb.KahaDBStore | main
2019-10-30 17:36:48,046 | INFO | Apache ActiveMQ 5.15.10 (***-***-***-***, ID:ip-***-***-***-***.ec2.internal-36686-1572457006663-0:1) uptime 1.495 seconds | org.apache.activemq.broker.BrokerService | main
2019-10-30 17:36:48,047 | INFO | Apache ActiveMQ 5.15.10 (***-***-***-***, ID:ip-***-***-***-***.ec2.internal-36686-1572457006663-0:1) is shutdown | org.apache.activemq.broker.BrokerService | main
2019-10-30 17:36:48,048 | INFO | Closing org.apache.activemq.xbean.XBeanBrokerFactory$1#4667ae56: startup date [Wed Oct 30 17:36:45 UTC 2019]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2019-10-30 17:36:48,049 | ERROR | Failed to load: class path resource [activemq.xml], reason: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.io.IOException: Failed to bind to /***-***-***-***:8161 | org.apache.activemq.xbean.XBeanBrokerFactory | main
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'invokeStart' defined in class path resource [jetty.xml]: Invocation of init method failed; nested exception is java.io.IOException: Failed to bind to /3.231.235.30:8161
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1630)[spring-beans-4.3.24.RELEASE.jar:4.3.24.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)[spring-beans-4.3.24.RELEASE.jar:4.3.24.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)[spring-beans-4.3.24.RELEASE.jar:4.3.24.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)[spring-beans-4.3.24.RELEASE.jar:4.3.24.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.3.24.RELEASE.jar:4.3.24.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)[spring-beans-4.3.24.RELEASE.jar:4.3.24.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)[spring-beans-4.3.24.RELEASE.jar:4.3.24.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:737)[spring-beans-4.3.24.RELEASE.jar:4.3.24.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)[spring-context-4.3.24.RELEASE.jar:4.3.24.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)[spring-context-4.3.24.RELEASE.jar:4.3.24.RELEASE]
at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-4.14.jar:4.14]
at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-4.14.jar:4.14]
at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.15.10.jar:5.15.10]
at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.15.10.jar:5.15.10]
at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.15.10.jar:5.15.10]
at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.15.10.jar:5.15.10]
at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.15.10.jar:5.15.10]
at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.15.10.jar:5.15.10]
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.10.jar:5.15.10]
at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.10.jar:5.15.10]
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.10.jar:5.15.10]
at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.10.jar:5.15.10]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_191]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_191]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_191]
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.10]
at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.10]
It looks like the web console is logged as available, and then it tries to bind the web console port again? This is on a fresh instance with no other services or java processes running.
Please advise.

Order dependency of ActiveMQ in case of ServiceMix-Shutdown

It seems, that in case using an ActiveMQ-Consumer the depending components are not shut down is the correct order.
As far as I can see, the JMSSessionPool is shutdown before the depending routes were closed.
This causes JMSEsception and each client routes will run into timeouts, before they are forced to shutdown.
Also the amq-broker ist removed before its depending routes.
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
<camelContext id="ObjectServerConnection" xmlns="http://camel.apache.org/schema/blueprint" >
<route id="test-aqm-shutdown-route">
<from uri="activemq:buffered-route?concurrentConsumers=5&asyncConsumer=5" />
<!--transacted ref="required" /-->
<convertBodyTo type="java.lang.String" />
<to uri="file:/out.txt" />
</route>
</camelContext>
</blueprint>
As I can see in logfile:
...
2016-12-08 11:48:53,003 | INFO | FelixShutdown | DefaultShutdownStrategy | 151 - org.apache.camel.camel-core - 2.16.4 | Starting to graceful shutdown 1 routes (timeout 300 seconds)
2016-12-08 11:49:02,336 | ERROR | [buffered-route] | faultJmsMessageListenerContainer | 145 - org.apache.servicemix.bundles.spring-jms - 3.2.17.RELEASE_1 | Could not refresh JMS Connection for destination 'buffered-route' - retrying in 5000 ms. Cause: Error while attempting to add new Connection to the pool; nested exception is javax.jms.JMSException: Could not create Transport. Reason: java.io.IOException: Broker named 'amq-broker' does not exist.
2016-12-08 11:49:07,339 | INFO | 5 - ShutdownTask | DefaultShutdownStrategy | 151 - org.apache.camel.camel-core - 2.16.4 | Route: test-aqm-shutdown-route shutdown complete, was consuming from: Endpoint[activemq://buffered-route?asyncConsumer=5&concurrentConsumers=5]
2016-12-08 11:49:07,340 | INFO | FelixShutdown | DefaultShutdownStrategy | 151 - org.apache.camel.camel-core - 2.16.4 | Graceful shutdown of 1 routes completed in 14 seconds
...
2016-12-08 11:47:09,064 | DEBUG | m://amq-broker#0 | AdvisoryConsumer | 122 - org.apache.activemq.activemq-osgi - 5.12.3 | Failed to send remove command: javax.jms.JMSException: Peer (vm://amq-broker#1) disposed.
javax.jms.JMSException: Peer (vm://amq-broker#1) disposed.
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1314)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1306)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.AdvisoryConsumer.dispose(AdvisoryConsumer.java:57)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.ActiveMQConnection.close(ActiveMQConnection.java:655)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.jms.pool.ConnectionPool.close(ConnectionPool.java:168)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.jms.pool.ConnectionPool.onException(ConnectionPool.java:379)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:1982)[122:org.apache.activemq.activemq-osgi:5.12.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_102]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_102]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
Caused by: org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://amq-broker#1) disposed.
at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:91)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1312)[122:org.apache.activemq.activemq-osgi:5.12.3]
... 9 more
Does anyone have an idea how to solve this?
Update the start-level of the bundle for your route. The container will shutdown bundles in reverse order. It appears your bundle is at the same level (or lower) as the broker bundle.
Also.. if you are using the deploy/ directory, you can update the setting:
etc/org.apache.felix.fileinstall-deploy.cfg:
felix.fileinstall.start.level=XX
ref: http://felix.apache.org/documentation/subprojects/apache-felix-file-install.html
For those that might be encountering this in Karaf/Felix, we got round the issue by writing a Karaf console script which stops our Camel contexts first before issuing the 'shutdown' command.
The context stop command(s) will gracefully stop all routes in that context before it returns.
The context stops take about 15 seconds (about 300 routes), and then the shutdown completes in about 5 seconds (we have over 350 bundles).
So the script looks something like this:
camel:context-stop context1
camel:context-stop context2
camel:context-stop context3
camel:context-stop context4
shutdown -f
(Context names have been changed to protect the innocent)

SSL Renegotiation failures with Jetty HttpClient

I have a proxy servlet based on Jetty's ProxyServlet that is seeing intermittent 502 responses due to SSL renegotiation failures in the proxy's HttpClient when attempting to proxy the request to the remote server. Wireshark traces show that the SSL handshake has completed, but then the HttpClient starts the negotiation over again by sending another Client Hello packet. The remote server (an F5 in this case) is configured to not allow SSL renegotiation and so it shuts down the connection, causing the proxied request to fail.
I tried calling SslContextFactory.setRenegotiationAllowed(false) when configuring the proxy's HttpClient, but this just causes the request to fail internally within the proxy. Debug level logging yields the output shown below. Note the "Renegotiation Denied" message, which causes the stream to be closed, resulting in a Connection Closed exception when subsequently attempting to write the proxied request to the output stream.
So what could be causing the HttpClient to think that it needs to perform SSL renegotiation, and what can I do to work around this issue? Changing the configuration of the F5 to allow SSL renegotiation is not an option. The problem is intermittent and the reproducability is variable, suggesting there may be a timing component.
I'm using Jetty 9.2.13.v20150730 on Java 1.8.0_66.
2015-10-26 15:23:04,987 | DEBUG | vletModel-46-263 | SslConnection | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | SslConnection#276888f4{NEED_WRAP,eio=-1/-1,di=-1} -> HttpConnectionOverHTTP#76f2815f(l:/9.32.133.96:51386 <-> r:mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443,closed=false)[HttpChannelOverHTTP#44d24828(exchange=HttpExchange#3284d378 req=TERMINATED/null#null res=PENDING/null#null)[send=HttpSenderOverHTTP#74d58ca9(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#501e585d(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]]] fill enter
2015-10-26 15:23:04,987 | DEBUG | vletModel-46-263 | ChannelEndPoint | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | filled 1006 SelectChannelEndPoint#57eceb70{mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443<->51386,Open,in,out,-,-,15/30000,SslConnection}{io=0,kio=0,kro=1}
2015-10-26 15:23:04,987 | DEBUG | vletModel-46-263 | SslConnection | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | SslConnection#276888f4{NEED_WRAP,eio=1006/-1,di=0} -> HttpConnectionOverHTTP#76f2815f(l:/9.32.133.96:51386 <-> r:mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443,closed=false)[HttpChannelOverHTTP#44d24828(exchange=HttpExchange#3284d378 req=TERMINATED/null#null res=PENDING/null#null)[send=HttpSenderOverHTTP#74d58ca9(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#501e585d(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]]] filled 1006 encrypted bytes
2015-10-26 15:23:04,987 | DEBUG | vletModel-46-263 | SslConnection | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | SslConnection#276888f4{NEED_WRAP,eio=0/-1,di=977} -> HttpConnectionOverHTTP#76f2815f(l:/9.32.133.96:51386 <-> r:mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443,closed=false)[HttpChannelOverHTTP#44d24828(exchange=HttpExchange#3284d378 req=TERMINATED/null#null res=PENDING/null#null)[send=HttpSenderOverHTTP#74d58ca9(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#501e585d(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]]] unwrap Status = OK HandshakeStatus = NEED_WRAP
bytesConsumed = 1006 bytesProduced = 977
2015-10-26 15:23:04,988 | DEBUG | vletModel-46-263 | SslConnection | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | SslConnection#276888f4{NEED_WRAP,eio=0/-1,di=977} -> HttpConnectionOverHTTP#76f2815f(l:/9.32.133.96:51386 <-> r:mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443,closed=false)[HttpChannelOverHTTP#44d24828(exchange=HttpExchange#3284d378 req=TERMINATED/null#null res=PENDING/null#null)[send=HttpSenderOverHTTP#74d58ca9(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#501e585d(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]]] renegotiation denied
2015-10-26 15:23:04,988 | DEBUG | vletModel-46-263 | SslConnection | 73 - org.eclipse.jetty.util - 9.2.13.v20150730 | SslConnection#276888f4{NEED_WRAP,eio=-1/-1,di=977} -> HttpConnectionOverHTTP#76f2815f(l:/9.32.133.96:51386 <-> r:mail.notes.collabservdaily.swg.usma.ibm.com/9.70.230.131:443,closed=false)[HttpChannelOverHTTP#44d24828(exchange=HttpExchange#3284d378 req=TERMINATED/null#null res=PENDING/null#null)[send=HttpSenderOverHTTP#74d58ca9(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator{s=START}],recv=HttpReceiverOverHTTP#501e585d(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]]] fill exit

Master/slave using Oracle

we're using the so called JDBC Master Slave architecture with Oracle DB. We have 2 nodes and each one has one Broker started. We start the Broker1 (on node1) and it becomes the MASTER obtaining the lock over the tables. Then we start the Broker2 on the node2 and this one starts as SLAVE. WE can see in the log of Slave broker that it's trying to obtain the lock every 10sec, but it fails:
2013-06-12 19:32:38,714 | INFO | Default failed to acquire lease. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
2013-06-12 19:32:48,720 | INFO | Default Lease held by Default till Wed Jun 12 19:32:57 UTC 2013 | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
Everything works fine and then in one moment in SLAVE's log we see that it suddenly becomes the MASTER:
2013-06-13 00:38:11,262 | INFO | Default Lease held by Default till Thu Jun 13 00:38:17 UTC 2013 | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
2013-06-13 00:38:11,262 | INFO | Default failed to acquire lease. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
...
2013-06-13 00:38:21,314 | INFO | Default, becoming the master on dataSource: org.apache.commons.dbcp.BasicDataSource#9c6a99d | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
2013-06-13 00:38:21,576 | INFO | Apache ActiveMQ 5.8.0 (Default, ID:corerec3-49774-1371083901328-0:1) is starting | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:21,692 | WARN | Failed to start jmx connector: Cannot bind to URL [rmi://localhost:1616/jmxrmi]: javax.naming.NameAlreadyBoundException: jmxrmi [Root exception is java.rmi.AlreadyBoundException: jmxrmi]. Will restart management to re-create jmx connector, trying to remedy this issue. | org.apache.activemq.broker.jmx.ManagementContext | JMX connector
2013-06-13 00:38:21,700 | INFO | Listening for connections at: tcp://corerec3:61617?transport.closeAsync=false | org.apache.activemq.transport.TransportServerThreadSupport | main
2013-06-13 00:38:21,700 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
2013-06-13 00:38:21,701 | INFO | Apache ActiveMQ 5.8.0 (Default, ID:corerec3-49774-1371083901328-0:1) started | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:21,701 | INFO | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:21,701 | ERROR | Memory Usage for the Broker (512 mb) is more than the maximum available for the JVM: 245 mb | org.apache.activemq.broker.BrokerService | main
2013-06-13 00:38:22,157 | INFO | Web console type: embedded | org.apache.activemq.web.WebConsoleStarter | main
2013-06-13 00:38:22,292 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
2013-06-13 00:38:22,353 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
while the MASTER's log shows no change from what it usually outputs...
So, it seems that somehow SLAVE obtains the lock (due to hmm... for example connection loss between master and the DB), but if we don't restart the brokers we start losing messages...
The problem is that in the producers' log we can see that it successfully sends the messages to the QueueX, but we don't see the consumer's taking them from the queue...
If we go to the DB and query _ACTIVEMQ_MSGS_ table we see that the messages are unprocessed.
It looks as if the broker (Producers are connected to) has the lock and inserts the messages into the DB and the brokers Clients are consuming from doesn't have the lock and can't consult the tables...
I don't know if all this makes much sense, but I surely hope someone might shed some light upon this one...
I didn't want to saturate the post with the configuration details, but if you need specific details like failover config, IPs, ports etc. I will post it...

Why does ActiveMQ restart automatically and how do i prevent it?

We've been using AMQ 5.5.1 in production for several months. Occasionally, we observe that the broker decides to refresh itself with no outside trigger. When this happens, our queue senders fail until the broker is back online some 10 minutes later. I cannot find any information or settings that would cause this behavior .. and let me control it.
Is this normal for the broker to recycle on its own like this? If so, what things would cause it?
2012-12-11 11:02:11,603 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1#f6ac0b: startup date [Tue Dec 11 11:02:11 EST 2012]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | WrapperSimpleAppMain
2012-12-11 11:02:13,806 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | WrapperSimpleAppMain
2012-12-11 11:02:13,821 | INFO | PListStore:D:\Tools\ActiveMQ\apache-activemq-5.5.1\bin\win32\..\..\data\localhost\tmp_storage started | org.apache.activemq.store.kahadb.plist.PListStore | WrapperSimpleAppMain
2012-12-11 11:02:13,868 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[D:\Tools\ActiveMQ\apache-activemq-5.5.1\bin\win32\..\..\data\kahadb] | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2012-12-11 11:02:16,618 | INFO | KahaDB is version 3 | org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain
2012-12-11 11:02:16,697 | INFO | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | WrapperSimpleAppMain
I found that the wrapper exe process was forcing the restart.
I was able to see in the wrapper.log (windows service) that the process was being restarted because the JVM was not responding. So this is not an issue with the broker auto-restarting per se.. it was an issue with the broker JVM somehow becoming hung (separate problem).
Here are the wrapper log entries for those interested:
ERROR | wrapper | 2012/12/11 11:01:58 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR | wrapper | 2012/12/11 11:01:58 | JVM did not exit on request, terminated
STATUS | wrapper | 2012/12/11 11:02:04 | Launching a JVM...