jmx doesnt seem to be working with activeMQ - activemq

I'm trying to use JMX with activeMQ for monitoring so far I've been using this and this as a reference but so far I'm unable to connect to jmx remotely and also I don't see any mention of jmx url in activemq logs. I'm wondering if there is another way to make sure jmx is working? is it supposed to be indicated in activemq logs?
PS I'm using jdk1.7 and activeMQ 5.14.2.
Thanks in advance!
EDIT
I set useJmx="true" in my activemq.xml file:
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="primary" useJmx="true" dataDirectory="${activemq.data}">
I tried two steps:
FIRST
I tried changing management context from createConnector="false" to :
<managementContext>
<managementContext createConnector="true" connectorPort="1099"/>
</managementContext>
FOR FIRST TIME THE PORT IS OPEN AND ACTIVEMQ RUNS FINE AND JMX URL GETS REPORTED IN LOGS ALTHOUGH I CAN NOT CONNECT IT TO IT REMOTLEY BUT IM ASSUMING ITS WORKING
SECOND
I reverted back the changes I made for managmentContext and I tried setting:
ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_BASE}/jmx.password -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_BASE}/jmx.access"
in bin/activemq script and I set a username in conf/jmx.access file as:
admin readwrite
And also have set a password in conf/jmx.password:
admin activemq
NOW ACTIVEMQ IS NOT RUNNINT AT ALL BUT IT WILL RUN IF I SET
AUTHENTICATE=FALSE AND DELETE JMX.ACCESS AND JMX.PASSWORD
CONFIGURATION IN BIN/ACTIVEMQ FILE BUT I NEED USER NAME AND PASSWORD
FOR SECURITY REASONS
I found this post which has the exact same issue as mine. any ideas?

Password authentication for remote monitoring is enabled by default. To disable it, set the following system property when you start the JVM:
-Dcom.sun.management.jmxremote.authenticate=false like you done in second test but you need to add system property -Dcom.sun.management.jmxremote
Try to add these jvm param to env file and update host ip
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=X.X.X.X
UPDATE
SO, to resume, i think that the FIRST step you tried is the best, for making it working these are the steps :
revert all jmx env file changes, like this :
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099 "
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote"
<broker useJmx="true" ...
<managementContext>
<managementContext createConnector="true" connectorPort="1099" />
</managementContext>
verify that in AMQ logs you have
INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi |
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
NOTE : Assuming that 10.10.10.16 is the IP of AMQ host.
try to connect with jconsole from another machine than AMQ host with url "service:jmx:rmi:///jndi/rmi://10.10.10.16:1099/jmxrmi" without user/pwd.
if you cannot connect, try like this :
<managementContext>
<managementContext createConnector="true" connectorPort="1099" connectorHost="10.10.10.16" />
</managementContext>
verify that in AMQ logs you have
INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://10.10.10.16:1099/jmxrmi |
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
retry to connect, step 4
at this step normally you can connect with jconsole.
if you want to add security and authorizations, use this :
<managementContext>
<managementContext createConnector="true" connectorPort="1099" connectorHost="10.10.10.16" >
<property xmlns="http://www.springframework.org/schema/beans" name="environment">
<map xmlns="http://www.springframework.org/schema/beans">
<entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.password.file"
value="${activemq.conf}/jmx.password"/>
<entry xmlns="http://www.springframework.org/schema/beans" key="jmx.remote.x.access.file"
value="${activemq.conf}/jmx.access"/>
</map>
</property>
</managementContext>
</managementContext>
Please try these steps and let me know in which one you fails to connect and provide error message from jconsole.

A couple troubleshooting steps:
Start jconsole or visualvm on the same system and connect using the "pid" attach method. Browse the MBeans and confirm org.apache.activemq beans are present
Run netstat -na and confirm ports 1099 (and 44444) are in LISTEN
Look at logs and confirm you do not have any "java.net.BindException: Address already in use.." messages that indicate a port conflict with an already running Java process.
Edit bin/env to configure JMX (this disables requiring SSL, sets the port to 1099 and disables requiring username and password.
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.port=1099 "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false "
ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote "
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"`
# ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"

Related

ActiveMQ doesn't recognize Stomp: Transport scheme NOT recognized: [stomp+ssl]

I'm trying to set up mcollective/activemq on a puppetmaster (open source puppet). I am having a problem where ActiveMQ does not recognize the Stomp protocol. Here is the relevant snippet in my /etc/activemq/instances-enabled/activemq/activemq.xml file that should enable stomp+ssl:
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
<transportConnector name="stomp+ssl" uri="stomp+ssl://0.0.0.0:61614?needClientAuth=true&transport.enabledProtocols=TLSv1,TLSv1.1,TLSv1.2"/>
</transportConnectors>
When I start ActiveMQ via service activemq start, I notice that the daemon doesn't end up running (I don't see it as a process). Then I tried running service activemq console activemq, and it looks like the problem is that it can't find the stomp Transport scheme. Here is the first error that I see in the output (and the error persists throughout the output):
ERROR | Failed to start Apache
ActiveMQ ([localhost,
ID:my-servers-hostname.example.com-40447-1475514312306-0:1], java.io.IOException: Transport
Connector could not be registered in
JMX: java.io.IOException: Transport
scheme NOT recognized: [stomp+ssl])
ActiveMQ recognizes openwire just fine. When using openwire+ssl only, without using stomp+ssl, the ActiveMQ daemon starts fine with no errors. However, when I try running mco find, I get an error because it seems that mco is still trying to use stomp+ssl (and ActiveMQ only has openwire+ssl enabled):
error 2016/10/03 17:26:59: activemq.rb:149:in `on_ssl_connectfail' SSL session creation with stomp+ssl://mcollective#localhost:61614 failed: Connection refused - connect(2) for "localhost" port 61614
Perhaps I need to adjust my mco config to use openwire instead of stomp? I wasn't sure where or what file that configuration would be in. Not sure why it doesn't recognize stomp, but I was wondering what my options are here. Is it even possible to use MCollective/ActiveMQ using only openwire+ssl, or is using stomp a requirement if I want to use mco? I don't think this is a port issue, as the relevant ports are open on the server I believe.
Here are the relevant packages/versions installed on my machine:
OS: Ubuntu 16.04 (xenial)
puppet: 4.7.0
ActiveMQ: 5.13.2
ruby-stomp: 1.3.5-1
MCollective (mco) version: 2.9.0
I've run into the same problem with the embedded ActiveMQ server in my project. Turns out I needed to add the following dependencies to my pom.
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-stomp</artifactId>
<version>5.15.0</version>
</dependency>
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-kahadb-store</artifactId>
<version>5.15.0</version>
</dependency>
In your case, I think you need to get hold of those 2 jars and add them to your ActiveMQ installation.
In activemq package provided by ubuntu 16+ library stomp transport was not included. I do not know why.
Yoy can download it manually and place in /usr/share/activemq/lib:
cd /usr/share/activemq/lib
# check your actviemq version before (apt-cache policy activemq) and use the relevant version of library.
wget https://repository.apache.org/content/repositories/releases/org/apache/activemq/activemq-stomp/5.13.5/activemq-stomp-5.13.5.jar
service activemq restart

Connecting JConsole to remote glassfish 4.1 / 4.1.1 via JMX

GlassFish running on a host with a public ip address and jmx port 8986
On startup
JMXStartupService has started JMXConnector on JMXService URL service:jmx:rmi://test.anahata.net.au:8986/jndi/rmi://test.anahata.net.au:8986/jmxrmi]]
When I try to connect via JConsole using test.anahata.net.au:8986 + user + password from another box, I get:
non-JRMP server at remote endpoint
If I try to connect using service:jmx:rmi://test.anahata.net.au:8986/jndi/rmi://test.anahata.net.au:8986/jmxrmi
I get:
connection refused
domain.xml
> <admin-service system-jmx-connector-name="system"
> type="das-and-server">
> <jmx-connector address="0.0.0.0" port="8986" name="system" auth-realm-name="admin-realm">
> <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
> client-auth="want" cert-nickname="s1as"></ssl>
> </jmx-connector>
> <property name="adminConsoleContextRoot" value="/admin"></property>
> <property name="adminConsoleDownloadLocation" value="${com.sun.aas.installRoot}/lib/install/applications/admingui.war"></property>
> <property name="ipsRoot" value="${com.sun.aas.installRoot}/.."></property>
> <das-config></das-config>
> </admin-service>
>
> ...
> <jvm-options>-Djava.rmi.server.hostname=test.anahata.net.au</jvm-options>
I have a smilar setup on a different vm running gf 4.1.1 and a real SSL certificate and I can connect to the it using host:port but If I try to connect using
service:jmx:rmi://host.com:8886/jndi/rmi://host.com:8886/jmxrmi
I get
java.rmi.ConnectIOException: non-JRMP server at remote endpoint
Has anyone got his head around this?
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.port=8686
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=myremoteip
Since glassfish 4 this is the only way.
Though I did managed to connect to glassfish's own jmx only before running asadmin enable-secure-admin.
turn on logging to FINE see if you got "LoginException during JMX auth"
javax.enterprise.system.tools.admin.security
Anyway my conclusion was lot of headache, just use the jvm jmx above.
When you get this error:
java.rmi.ConnectIOException: non-JRMP server at remote endpoint
you are missing the keystore.jks file from glassfish/payara domain.

ActiveMQ 5.8 network of broker with custom jmx port

I am trying to run a network of broker with 2 brokers on the same network but on 2 different virtual machines.
Because of some internal constraints I have to use a custom jmx port.
I am using the Tanuki wrapper to launch ActiveMQ on an Ubuntu server.
Here is the relevant part of my activemq.xml
<broker xmlns="http://activemq.apache.org/schema/core" advisorySupport="false" useJmx="true" brokerName="test1" dataDirectory=".../data/activemq">
<networkConnectors>
<networkConnector uri="multicast://1.2.3.4:101234?group=test"
dynamicOnly="true"
networkTTL="3"
prefetchSize="1"
decreaseNetworkConsumerPriority="true"
userName="user"
password="password"/>
</networkConnectors>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616" rebalanceClusterClients="true" updateClusterClients="true" />
<transportConnector name="nio" uri="nio://0.0.0.0:61617" rebalanceClusterClients="true" updateClusterClients="true" discoveryUri="multicast://1.2.3.4:101234?group=test" />
</transportConnectors>
...
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
...
</broker>
Here is the relevant part of wrapper.conf:
# Uncomment to enable jmx
wrapper.java.additional.1=-Dcom.sun.management.jmxremote
wrapper.java.additional.2=-Dcom.sun.management.jmxremote.port=4321
wrapper.java.additional.3=-Dcom.sun.management.jmxremote.authenticate=false
wrapper.java.additional.4=-Dcom.sun.management.jmxremote.ssl=false
When running ActiveMQ on both broker, I see the process with the expected options:
activemq 30682 30680 3 13:27 ? 00:00:30 java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=4321 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djavax.net.ssl.keyStore=../../conf/broker.ks -Djavax.net.ssl.trustStore=../../conf/broker.ts -Dcom.sun.management.jmxremote -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dactivemq.conf=../../conf -Dactivemq.data=../../data -Xms2048m -Xmx2048m -Djava.library.path=../../bin/linux-x86-64/ -classpath ../../bin/wrapper.jar:../../bin/activemq.jar -Dwrapper.key=y4TuwO32Hj6kN7w8 -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=30680 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp org.apache.activemq.console.Main start
The port is open on the running shorewall.
The network of broker is up but I cannot connect to the jmx using jvisualvm with server_dns:4321. It returns the error
"cannot connect to server_dns:4321 using service:jmx:rmi:///jndi/rmi://server_dns:4321/jmxrmi
I can see some traffic on the port via tcpdump.
Could anybody tell me what I am doing wrong or how I should use ActiveMQ as a Network Of Broker with a custom jmx port?
JMX needs 2 open ports. An extra one is necessary for rmi.
I figured it out thanks to this post: Apache ActiveMQ browser can't connect to JMX console
In my case the fix is to change the configuration of my wrapper to expose the rmi port and open the port on the firewall
# Uncomment to enable jmx
wrapper.java.additional.1=-Dcom.sun.management.jmxremote
wrapper.java.additional.2=-Dcom.sun.management.jmxremote.port=4321
wrapper.java.additional.3=-Dcom.sun.management.jmxremote.port=8765
wrapper.java.additional.4=-Dcom.sun.management.jmxremote.authenticate=false
wrapper.java.additional.5=-Dcom.sun.management.jmxremote.ssl=false

configure JMX for ActiveMQ for remoting access

Anyone can give the detailed steps on how to enable JMX (can be access remotely) on a newly installed 5.5.0 version?
In your activemq.xml file, you need make sure useJmx is true on your broker element:
<broker xmlns="http://activemq.org/config/1.0" brokerName="localhost" useJmx="true">
and ensure that you have a management context
<managementContext>
<managementContext createConnector="true" connectorPort="1099"/>
</managementContext>
From there it is just a matter of making sure you can connect over TCP to your broker on port 1099 or whatever port you specify. This doesn't work quite so straightforward on services like EC2 or anything that does some heavy NAT'ing: http://jmsbrdy.com/monitoring-java-applications-running-on-ec2-i

Active MQ JMX SSL

I'm trying to use SSL with the JMX connector that Active MQ creates, but with no success. I'm able to get SSL working with the JVM platform JMX connector, but that requires storing keystore and truststore passwords plaintext, which is a no-go for our project.
Using the instructions here, I set up managementContext in activemq.xml as follows:
<managementContext>
<managementContext createConnector="true">
<property xmlns="http://www.springframework.org/schema/beans" name="environment">
<map xmlns="http://www.springframework.org/schema/beans">
<entry xmlns="http://www.springframework.org/schema/beans"
key="javax.net.ssl.keyStore"
value="${activemq.base}/conf/keystore.jks"/>
<entry xmlns="http://www.springframework.org/schema/beans"
key="javax.net.ssl.keyStorePassword"
value="${keystore.password}"/>
<entry xmlns="http://www.springframework.org/schema/beans"
key="javax.net.ssl.trustStore"
value="${activemq.base}/conf/truststore.jks"/>
<entry xmlns="http://www.springframework.org/schema/beans"
key="javax.net.ssl.trustStorePassword"
value="${truststore.password}"/>
</map>
</property>
</managementContext>
</managementContext>
This section seems to be completely ignored when the connector starts up. I can connect without credentials. I also tried using username and password authentication instead of ssl for JMX, as seen here, and that worked fine.
Has anyone seen this before? Any ideas? Thanks!
Have you enabled jmx ssl in the activemq launch scripts? On windows in the activemq-admin or activemq batch files, uncomment and modify the SUNJMX settings.
JMX authentiation is independent of whether ssl is used. It is controlled by the authenticate attribute. By default it will use the jmx access files in your jre, so re-point them with the system properties shown below. You may get an error message stating that the files themselves must be access controlled, so set them with chmod on unix or cacls on windows. I would suggest even turning off the ssl and getting the authentication to work first. You can test with jconsole with a remote connection to confirm that it wants credentials. Then follow-up with the ssl stuff.
set SUNJMX=-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=1199 -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.password.file=%ACTIVEMQ_BASE%/conf/access/jmx.password -Dcom.sun.management.jmxremote.access.file=%ACTIVEMQ_BASE%/conf/access/jmx.access
I had the same issue regarding the ActiveMQ SSL configuration (keystore & password) in the XML not working.
My requirement was to enable remote JMX monitoring of ActiveMQ with SSL and authentication through a firewall.
I resolved it using a custom JMX connector (via a Java Agent), rather than using the JMX connector that Active MQ creates.
see: JMX connectivity through a firewall for an example (JMXAgent.java)
The important entries for configuring SSL in the JMXAgent.java are:
Map<String, Object> env = new HashMap<String, Object>();
SslRMIClientSocketFactory csf = new SslRMIClientSocketFactory();
SslRMIServerSocketFactory ssf = new SslRMIServerSocketFactory();
env.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, csf);
env.put(RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE, ssf);
You can also specify your authentication files in the env Map:
env.put("jmx.remote.x.password.file", System.getProperty("password.file","<default_path>"));
env.put("jmx.remote.x.access.file", System.getProperty("access.file","<default_path>"));
The Java Agent needs to be compiled and put into a jar with a valid manifest file as described here
Add the following to the activemq launch configuration (depending on activemq version/ environment and run ActiveMQ:
-javaagent:<full_path_to_agent_jar_file> \
-Dpassword.file=<full_path_to_jmx.password_file> \
-Daccess.file=<full_path_to_jmx.access_file> \
-Djavax.net.ssl.keyStore=<full_path_to_keystore_file> \
-Djavax.net.ssl.keyStorePassword=<password>
You should then be able to connect through jconsole (with correct security parameters)
The remote JMX connection URL will be something like:
service:jmx:rmi://<host>:<rmi_server_port>/jndi/rmi://<host>:<port>/jmxrmi
Note - ports can be configured in the Java Agent.