Order dependency of ActiveMQ in case of ServiceMix-Shutdown - activemq

Question

It seems, that in case using an ActiveMQ-Consumer the depending components are not shut down is the correct order.
As far as I can see, the JMSSessionPool is shutdown before the depending routes were closed.
This causes JMSEsception and each client routes will run into timeouts, before they are forced to shutdown.
Also the amq-broker ist removed before its depending routes.
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
<camelContext id="ObjectServerConnection" xmlns="http://camel.apache.org/schema/blueprint" >
<route id="test-aqm-shutdown-route">
<from uri="activemq:buffered-route?concurrentConsumers=5&asyncConsumer=5" />
<!--transacted ref="required" /-->
<convertBodyTo type="java.lang.String" />
<to uri="file:/out.txt" />
</route>
</camelContext>
</blueprint>
As I can see in logfile:
...
2016-12-08 11:48:53,003 | INFO | FelixShutdown | DefaultShutdownStrategy | 151 - org.apache.camel.camel-core - 2.16.4 | Starting to graceful shutdown 1 routes (timeout 300 seconds)
2016-12-08 11:49:02,336 | ERROR | [buffered-route] | faultJmsMessageListenerContainer | 145 - org.apache.servicemix.bundles.spring-jms - 3.2.17.RELEASE_1 | Could not refresh JMS Connection for destination 'buffered-route' - retrying in 5000 ms. Cause: Error while attempting to add new Connection to the pool; nested exception is javax.jms.JMSException: Could not create Transport. Reason: java.io.IOException: Broker named 'amq-broker' does not exist.
2016-12-08 11:49:07,339 | INFO | 5 - ShutdownTask | DefaultShutdownStrategy | 151 - org.apache.camel.camel-core - 2.16.4 | Route: test-aqm-shutdown-route shutdown complete, was consuming from: Endpoint[activemq://buffered-route?asyncConsumer=5&concurrentConsumers=5]
2016-12-08 11:49:07,340 | INFO | FelixShutdown | DefaultShutdownStrategy | 151 - org.apache.camel.camel-core - 2.16.4 | Graceful shutdown of 1 routes completed in 14 seconds
...
2016-12-08 11:47:09,064 | DEBUG | m://amq-broker#0 | AdvisoryConsumer | 122 - org.apache.activemq.activemq-osgi - 5.12.3 | Failed to send remove command: javax.jms.JMSException: Peer (vm://amq-broker#1) disposed.
javax.jms.JMSException: Peer (vm://amq-broker#1) disposed.
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1314)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1306)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.AdvisoryConsumer.dispose(AdvisoryConsumer.java:57)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.ActiveMQConnection.close(ActiveMQConnection.java:655)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.jms.pool.ConnectionPool.close(ConnectionPool.java:168)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.jms.pool.ConnectionPool.onException(ConnectionPool.java:379)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:1982)[122:org.apache.activemq.activemq-osgi:5.12.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_102]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_102]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_102]
Caused by: org.apache.activemq.transport.TransportDisposedIOException: Peer (vm://amq-broker#1) disposed.
at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:91)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)[122:org.apache.activemq.activemq-osgi:5.12.3]
at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1312)[122:org.apache.activemq.activemq-osgi:5.12.3]
... 9 more
Does anyone have an idea how to solve this?

Answer 1

Update the start-level of the bundle for your route. The container will shutdown bundles in reverse order. It appears your bundle is at the same level (or lower) as the broker bundle.
Also.. if you are using the deploy/ directory, you can update the setting:
etc/org.apache.felix.fileinstall-deploy.cfg:
felix.fileinstall.start.level=XX
ref: http://felix.apache.org/documentation/subprojects/apache-felix-file-install.html

Answer 2

For those that might be encountering this in Karaf/Felix, we got round the issue by writing a Karaf console script which stops our Camel contexts first before issuing the 'shutdown' command.
The context stop command(s) will gracefully stop all routes in that context before it returns.
The context stops take about 15 seconds (about 300 routes), and then the shutdown completes in about 5 seconds (we have over 350 bundles).
So the script looks something like this:
camel:context-stop context1
camel:context-stop context2
camel:context-stop context3
camel:context-stop context4
shutdown -f
(Context names have been changed to protect the innocent)