ActiveMQ remote connections refused despite 0.0.0.0 in broker URL - activemq

Question

I have an ActiveMQ v5.7.0 broker, running in Karaf v2.3.3, that I want to enable for remote connections. I've set the broker URL to 0.0.0.0:61616, to enable it to listen to network traffic. I've opened the firewall to allow the traffic from the client machines. However, all remote connections are being refused. A quick netstat seems to tell me that the broker isn't listening outside of localhost.
jeremy#server:~$ netstat -pan | grep 61616
tcp6 0 0 127.0.0.1:61616 :::* LISTEN -
Looking at the broker via Hawtio tells me that the URL looks as it should.
Transport connectors Openwire: tcp://0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600
The firewall is definitely OK, as the connections are being refused rather than just being dropped.
The broker is responding correctly to connections from localhost.
2013-10-14 17:34:29 Connected to localhost:61613
This is the sort of error I get from remote connections:-
Error connecting to xxx.xxx.xxx.xxx:61613: IO::Socket::INET: connect: Connection refused at /usr/local/share/perl/5.14.2/Net/Stomp.pm line 102.
EDIT: telnet output added
Localhost port 61613
jeremy#server:~$ telnet localhost 61613
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Remote connection port 61613
jeremy#other-server:~$ telnet xxx.xxx.xxx.xxx 61613
Trying xxx.xxx.xxx.xxx...
telnet: Unable to connect to remote host: Connection refused
Localhost connection port 61616 (this one is interesting)
jeremy#server:~$ telnet localhost 61616
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
ðActiveMQ Þ
MaxFrameSizÿÿÿ CacheSize
CacheEnabledSizePrefixDisabled MaxInactivityDurationInitalDelay'TcpNoDelayEnabledMaxInactivityDurationu0TightEncodingEnabledStackTraceEnabledPuTTYConnection closed by foreign host.
Remote connection port 61616
jeremy#other-server:~$ telnet xxx.xxx.xxx.xxx 61616
Trying xxx.xxx.xxx.xxx...
telnet: Unable to connect to remote host: Connection refused
EDIT: remote server karaf log output added
2013-10-15 19:00:46,599 | ERROR | c.event.invited] | faultJmsMessageListenerContainer | .DefaultMessageListenerContainer 909 | 69 - org.springframework.jms - 3.2.4.RELEASE | Could not refresh JMS Connection for destination 'Consumer.notifications.VirtualTopic.event.invited' - retrying in 5000 ms. Cause: Error while attempting to add new Connection to the pool; nested exception is javax.jms.JMSException: Could not connect to broker URL: tcp://xxx.xxx.xxx.xxx:61616. Reason: java.net.ConnectException: Connection refused
Here's the broker.xml.
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0"
xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0"
xmlns:amq="http://activemq.apache.org/schema/core">
<ext:property-placeholder />
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="jellyfish-messaging"
dataDirectory="${karaf.data}/activemq/localhost"
useShutdownHook="false"
persistent="true"
schedulerSupport="true"
startAsync="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<persistenceAdapter>
<kahaDB directory="${karaf.data}/activemq/localhost/kahadb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="64 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!-- The transport connectors ActiveMQ will listen to -->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireformat.maxFrameSize=104857600"/>
</transportConnectors>
</broker>
<bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="tcp://0.0.0.0:61616" />
</bean>
<bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
<property name="maxConnections" value="8" />
<property name="maximumActive" value="500" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>
<bean id="resourceManager" class="org.apache.activemq.pool.ActiveMQResourceManager" init-method="recoverResource">
<property name="transactionManager" ref="transactionManager" />
<property name="connectionFactory" ref="jmsConnectionFactory" />
<property name="resourceName" value="activemq.localhost" />
</bean>
<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory" />
<property name="transacted" value="false" />
<property name="concurrentConsumers" value="10" />
</bean>
<bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration" ref="jmsConfig" />
</bean>
<reference id="transactionManager" interface="javax.transaction.TransactionManager" />
<service ref="pooledConnectionFactory" interface="javax.jms.ConnectionFactory">
<service-properties>
<entry key="name" value="localhost"/>
</service-properties>
</service>
</blueprint>
Can anyone tell me what I'm missing?
Thanks,
J.

Answer 1

I've solved this. It was neither a problem with the firewall, nor with the ActiveMQ configuration.
The Karaf kar file in which the ActiveMQ broker was defined included the activemq-web-console feature. We've not been using this feature, as we're fans of Hawtio, so had never configured it.
As per this blog post, the console was coming up with default settings, including listening on port 61616. This meant that two brokers were in a race condition on start-up and the webconsole-defined one was generally winning. Since by default it isn't configured for remote access, it was locking the port for localhost connections only.
The giveaway was a directory called ${activemq.data} (literally) within the Karaf home directory, containing a second Kahadb repository. All of our broker config was set to use the data directory and we've never specifically set the ActiveMQ environment variables, so this led us to look for where a second broker might have come from.
Might have spotted it more quickly had we done activemq:list inside a Karaf session, as it was listing two brokers.
Simple solution - delete activemq-web-console from the features XML.