Using Spring actuator and enabled Prometheus configuration to monitor RabbitMQ message stats but always showing the message count as zero even if there is a message in queue or consumed. IS there any config required to read the message where in case other jvm or cpu usage are properly monitored.
http://localhost:8080/actuator/prometheus
HELP rabbitmq_rejected_total
# TYPE rabbitmq_rejected_total counter
rabbitmq_rejected_total{name="rabbit",} 0.0
# HELP rabbitmq_published_total
# TYPE rabbitmq_published_total counter
rabbitmq_published_total{name="rabbit",} 0.0
# HELP rabbitmq_consumed_total
# TYPE rabbitmq_consumed_total counter
rabbitmq_consumed_total{name="rabbit",} 0.0
I would strongly suggest using this Prometheus Plugin , it runs natively in ErLang runtime and will save you the complexity around configuration and As of 3.8.0, RabbitMQ ships with built-in Prometheus & Grafana support.
Related
I have a challenge - build and release monitoring system in consist of RabbitMQ cluster (with 3 nodes) and standalone Grafana server for visualisation metrics.
I have found in official documentation of prometheus plugin for RabbitMQ (documentation) next section:
This plugin is new as of RabbitMQ 3.8.0.
But i have cluster of version 3.6.8 and when i run the next command
rabbitmq-plugins enable rabbitmq_prometheus
The output is:
Error: The following plugins could not be found:
rabbitmq_prometheus
Upgrade the cluster now is not possible and my question is:
How do i may configure monitoring of the cluster without upgrade it and with prometheus (preferred option) and grafana?
Thanks in advance!
The Prometheus plugin is not the only way to monitor a RabbitMQ cluster.
You can also use the rabbitmq exporter in sidecar. If you are not on a docker platform, you can download the exporter from the release assets and install it as a service somewhere.
It would be best to install the exporter on every server hosting the RabbitMQ node because:
you will need to have as many install as there are nodes (Prometheus is service oriented monitoring)
from the settings, the exporter is accessing the management plugin interface of RabbitMQ; it should stay bound to localhost to reduce attack surface
If your hands are really tied, you can deploy them anywhere (let say on the same server) and point each exporter to a different RabbitMQ node. Prometheus configuration can then identify the underlying service.
- job_name: rabbitmq
honor_labels: true
static_configs:
- targets: ['monitoring-server:97001']
labels:
instance: 'rabbitmq_node_A'
- targets: ['monitoring-server:97002']
labels:
instance: 'rabbitmq_node_B'
# or play with relabeling to acchieve the same.
An important drawback is that there are more cases where the exporter may not be able to access RabbitMQ and you end up alert on events not impacting your RabbitMQ cluster.
My mule application is comprised of 2 nodes running in a cluster, and it listens to IBM MQ Cluster (basically connecting to 2 MQ via queue manager). There are situations where one mule node pulls or takes more than 80% of message from MQ cluster and another mule node picks rest 20%. This is causing CPU performance issues.
We have double checked that all load balancing is proper, and very few times we get CPU performance problem. Please can anybody give some ideas what could be possible reason for it.
Example: last scenario was created where there are 200000 messages in queue, and node2 mule server picked 92% of message from queue within few minutes.
This issue has been fixed now. Got into the root cause - our mule application running on MULE_NODE01 reads/writes to WMQ_NODE01, and similarly for node 2. One of the mule node (lets say MULE_NODE02) reads from linux/windows file system and puts huge messages to its corresponding WMQ_NODE02. Now, its IBM MQ which tries to push maximum load to other WMQ node to balance the work load. That's why MULE_NODE01 reads all those loaded files from WMQ_NODE01 and causes CPU usage alerts.
#JoshMc your clue helped a lot in understanding the issues, thanks a lot for helping.
Its WMQ node in a cluster which tries to push maximum load to other WMQ node, seems like this is how MQ works internally.
To solve this, we are now connecting our mule node to MQ gateway, rather making 1-to-1 connectivity
This could be solved by avoiding the racing condition caused by multiple listeners. Configure the listener in the cluster to the primary node only.
republish the message to a persistent VM queue.
move the logic to another flow that could be triggered via a VM listener and let the Mule cluster do the load balancing.
Rabbitmq beam.smp process utilize most of the Memory size for no reason
RabbitMQ version: 3.7
erlang 22
I don't have any special configuration or anything
I don't use celery or anything except Rabbitmq.
I searched for this issue and all I found is something related to celery!
What's the problem with this RabbitMQ .. it can't stand for couple of days without issues!
CPU utilization could be Erratic with large number of mirrored queues.
Please mention the following RabbitMQ deployment details:
RabbitMQ v3.7
Erlang/OTP v22
Ubuntu 16.04.5 LTS
Linux 4.15.0-32-generic x86_64
mention dstat --cpu check for RabbitMQ nodes utilization w.r.t CPU (user + system)
Additionally u can do what
It is possible to make schedulers that currently do not have work to do using the +sbwt flag:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+sbwt none"
The value of none can reduce CPU usage on systems that have a large number of mostly idle connections.
Several reasons can increase the CPU usage, you are not providing enough information.
What you should do is:
Check the rabbitmq logs, to see if there is some error
Check if you have some publisher that are trolling the server
Check the number of the queues/binding, maybe you are creating too many queues/binding
you can also enable this rabbitmq plugin https://github.com/rabbitmq/rabbitmq-top to see with process is using all the CPU
I am running a performance test using JMeter for our application and the there is some asynchronous processing in the form of events on an ActiveMQ queue. I want to wait for the ActiveMQ queue to be empty before recording the statistics for my test. Is there a good way to do that?
I have explored the JMS Producer/Consumers in JMeter 2.10 but they consume messages off the queue which is not what I want as it modifies the original flow of the application. Is there a way to monitor the draining of the queue without consuming the messages of ActiveMQ?
I am using ActiveMQ 5.8 and JMeter 2.10
I was able to monitor ActiveMQ using the HTTP Request to poll the ActiveMQ web console and get the state of all the queues in XML format. After that I used XPATH to extract the size of the queue I was interested in. The snapshots below show the configuration which I was finally able to use. The XPATH expression which I used was
/queues/queue[#name='${queueName}']/stats/#size
One additional thing which I has to do was to setup basic HTTP authentication to be able to connect to the ActiveMQ web console.
The MBean solution by Mahesh should also work if JMX is enabled on the server but it is not enabled by default.
I have documented it in detail here
You can get the pending messages in that queue using the MBean
"org.apache.activemq:BrokerName=host1,Type=Queue,Destination=dest1"
attribute: "QueueSize"
After checking once every few seconds and the value being not more than zero, you can start recording the statistics.
You can create a simple Java class that consumes all messages from the queue. JMeter can run it before tests.
I have the following situation that is affecting our ActiveMQ 5.8 broker.
Several Perl scripts on a Windows workstation connected to ActiveMQ using STOMP and subscribed (nondurable) to various topics. The power failed on the Workstation.
Using the Web console, I can see that ActiveMQ still thinks these subscribers are connected, based on the number of consumers shown and on the high temp message store being used. I had set for no producer flow control and set memory limits, so what I believe I am seeing is that ActiveMQ is spooling all messages to disk because it thinks the long dead subscribers are still connected and might eventually read them. It's been 30 days, and ActiveMQ still doesn't realize that these subscribers are no longer connected.
It there a way to configure ActiveMQ so that "undead" subscriber connections like these are eventually cleared automatically?
While the previous answer is basically correct, ActiveMQ does provide solutions for STOMP transports on the Broker to heart-beat connections, even if the client connects with STOMP v1.0. I blogged about this some time ago when ActiveMQ v5.6 was released, see the section on STOMP 1.0 default heartbeat configuration. Another option is to set tcp keepAlive on for the transport and tune your OS to use a shorter default check interval, the default is usually around two hours.
Though Stomp 1.1+ supports Heartbeating, Active MQ currently doesnt support inactive consumer detection for Stomp. (usually achieved with wireFormat.maxInactivityDuration).
Be Careful:
These values are currently not supported but are planned for a later release
ActiveMQ supports it for Openwire though. i,e after the configured duration the consumer would be considered DEAD !