I have ingestion pipeline source --> ActiveMQ --> Storm --> HDFS.
Now I want to monitor ActiveMQ queue so as to detect failures and send email to concerned person.
I haven't used it before so I'm looking at options that can be used for monitoring these queues.
Can somebody suggest suitable tools ?
See http://activemq.apache.org/how-can-i-monitor-activemq.html. But here you only see information how to access the data.
With the jolokia interface you have a REST api which access all jmx values.
Check for existing monitoring application you already have and let's see if they can do REST requests or jmx. E.g. you could use Graphite, Kibana, monitis.
Related
IN the following link:
http://www.mulesoft.org/documentation/display/current/Mule+ESB+3.4.0+Release+Notes
I see the following
EE-3141 When using a Throttling policy with throttling statics enabled, limit headers are swapped.
However, I can find no example of throttling policies within Mule ESB, but there is possibly a throttling policy within the Anypoint API Manager
Could someone please provide a link to how to use a Throttling policy within Mule ESB?
Thanks
To achieve correct Throttling behaviour you can follow below steps:
Configure a queue (for example, a persistent VM or a JMS queue to avoid message lost during the Mule server crash) after your inbound endpoint.
Configure scheduled delay, for example AMQ_SCHEDULED_DELAY in case of ActiveMQ, to a desired value. If the queue does not support scheduled delay, then we need to find our way to achieve the delay, probably using a Java component.
Finally, configure the outbound endpoint.
The Throttling module (which can be configured as throttler or rate limiter) comes out of the box with any Mule API Gateway distribution. Mule EE comes with a light weight version of it. If you are using Anypoint API Platform, then you don't need to pay attention to the internals of how it is being done: Simply apply/unapply the policy to your managed API and it will work like a charm.
Even i tried implementing the Throttling concept in mule flows. There is no exact way of implementation for this , but i was able get that nature into the flows using the receiver thread profiling at inbounds and dispatch threading profiles on outbound connectors.
I am running a performance test using JMeter for our application and the there is some asynchronous processing in the form of events on an ActiveMQ queue. I want to wait for the ActiveMQ queue to be empty before recording the statistics for my test. Is there a good way to do that?
I have explored the JMS Producer/Consumers in JMeter 2.10 but they consume messages off the queue which is not what I want as it modifies the original flow of the application. Is there a way to monitor the draining of the queue without consuming the messages of ActiveMQ?
I am using ActiveMQ 5.8 and JMeter 2.10
I was able to monitor ActiveMQ using the HTTP Request to poll the ActiveMQ web console and get the state of all the queues in XML format. After that I used XPATH to extract the size of the queue I was interested in. The snapshots below show the configuration which I was finally able to use. The XPATH expression which I used was
/queues/queue[#name='${queueName}']/stats/#size
One additional thing which I has to do was to setup basic HTTP authentication to be able to connect to the ActiveMQ web console.
The MBean solution by Mahesh should also work if JMX is enabled on the server but it is not enabled by default.
I have documented it in detail here
You can get the pending messages in that queue using the MBean
"org.apache.activemq:BrokerName=host1,Type=Queue,Destination=dest1"
attribute: "QueueSize"
After checking once every few seconds and the value being not more than zero, you can start recording the statistics.
You can create a simple Java class that consumes all messages from the queue. JMeter can run it before tests.
I read in ActiveMQ page, using JMX we can monitor queues in activemq. How can we get notified if a queue has messages (depth high) or service interval is high in ActiveMQ. Without using any shell scripts in unix environment. Is it possible through Java program? If yes, give me some ideas to get this done.
For a JMX-free approach, you can also use the XML feed served by the activemq console page. The XML feed is hosted at http://ip:port/admin/xml/queues.jsp
This will have tags similar to this for each queue:
<queue name="your queue">
<stats size="0" consumerCount="1" enqueueCount="0" dequeueCount="0"/>
....
</queue>
Just parse this XML in your code and you are good to go.
you can use Java via JMX APIs to periodically poll for queue stats (see this guide)
for the notification approach, you'd need to use advisory messages to monitor messages delivered to a queue (see this guide)
Yes it is possible in Java.
Starting from version 5.8 of ActiveMQ jolokia agent comes embedded. So it is possible for you to get all stats that JMX can pull using HTTP request which will retuen you stats as JSON and then you can check current values and raise Email alert using SMTP if values go beyond threshold you have decided.
Lets say you want to pull Broker stats using Jolokia hit below URL in your browser enter AMQ console username and password which is admin by default
http://servername.com:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=localhost
Or if you dont want to go through all this trouble , You can use ready made Python script which I have created to Monitor AMQ Heap, Queue params and Broker availability.You can take a look , it may help you in developing your custiom script or program
AMQMonitor and Alerting script
I am a looking for proven tools to monitor performance on ActiveMQ 5.5. I come from an environment which used Glassfish and JMQ that can tell me rate of messages produced and consumed on any given destination using "imqcmd". Is there a like tool for ActiveMQ or a different way to go about it?
I see that there is a project at http://activemq.apache.org/activemq-performance-module-users-manual.html that will do some sort of performance reporting but it seems to be no more than a SNAPSHOT version that I cannot get to operate.
Any input would be appreciated.
there are several options for this: JMX, AMQ webconsole, other options
here are my notes on this...I opted to go with JMX and built a simple web app (JSP, jQuery, Google Charts, etc) to interface with JMX to gather queue stats, manage queues, etc...
http://www.consulting-notes.com/2010/08/monitoring-and-managing-activemq-with.html
I have tried and tested the JMX API and it is pretty simple to use and provides a vast number of statistics required for monitoring ActiveMQ.
But the problem is, i dont want to monitor my ActiveMQ remotely and also i dont want to use another API.To be more precise, i want to use the JMS API itself to get statistics related to various destinations and the broker itself.
Advisory messages seem to be an alternative but they provide limited Amount of Administrative Messages to monitor.
Any input is highly appreciated...
There is no built-in support for this. But you can implement a JMS topic which publishes the monitoring data every few seconds. Make the connection non-persistent so that it doesn't pile up when there are no listeners or when they loose connection.
Now you can write a client that connects to this topic and it will receive updates.
AMQ-2379 resulted in a broker plugin for grabbing statistics from destinations by sending a simple JMS message. Check out the docs that show how to use it here:
http://activemq.apache.org/statisticsplugin.html
The statistics plugin is available in the 5.3 release.
You can checkout this http://issues.apache.org/activemq/browse/AMQ-2379, it will be avaiable in upcoming 5.3.0 release
There's a blog post queued up to go on http://issues.apache.org/activemq/browse/AMQ-2379 - will post it in a couple of days or so