Recently I installed Rabbit MQ in Centos8 for my company. We also using Splunk Enterprise so we wants to integrate our Rabbit MQ to Splunk and we wants to see, search, check our logs which is coming from Rabbit MQ to in Splunk . How can I do that I don't know. I google it but I didn't get info about it. May anybody help to me for this goal ? Thank you
The scripts at https://github.com/jerrykuch/rabbitmq-splunk are somewhat old, but should still be functional. It can leverage the HTTP API to pull in the relevant data. That page also lists the recommended files to monitor
Always check Splunkbase when looking for ingesting data types - often there exist apps and add-ons that will do what you're looking for
Here are two related to RabbiMQ:
JMS Messaging Modular Input - https://splunkbase.splunk.com/app/1317/#/details
AMQP Messaging Modular Input - https://splunkbase.splunk.com/app/1812/#/details
Related
I am NOT talking about Cloudera or Yarn system level logs. I am talking about applications running on Cloudera/Yarn infrastructure.
We have tens of Java and Python applications running on our Cloudera Infra, and all of them generate application logs. I am looking for the best way to monitor these logs for any errors and warnings. If it is a pure stand alone Java application, traditionally we can use one of these log scraper tools that send emails based on an expression matching (to detect error/warning/any other special situation). I am looking for something similar, that can monitor our application logs and emails us in real time for better production application support.
If thinking about this like a traditional application log monitoring is not the right way, then I am happy to know if there are any better industry standard approaches. Thanks!
I guess the ElasticStack (https://www.elastic.co/de/) could be one approach to solve this. You could use FileBeats to send your application logs to Logstash which forwards it to ElasticSearch. You could then create a Watcher in Kibana which sends i.e. Emails based on some triggering condition (we use a webhook to send notifications into a MS Teams channel).
This solution should work at least in near-realtime (~1-2 minutes delay, but this also depends on your watcher configuration).
Being new to the RabbitMQ I was wondering how to deal with an offline target node.
As an example this scenario:
1 log recording application that stores logs to some persistent storage
N log publishing applications that want their logs to be written to the persistent storage via the log recording server.
There would be two options:
Each publishing application publishes it's log messages to it's local RabbitMQ instance and the log recording server must subscribe to each of these
The log recording application has it's local RabbitMQ instance on which each log publishing application delivers it's messages.
Option 1 would require me to reconfigure/recode/notify the recording application each time a new application appears or moves. Therefore I would think Option 2 is the right one, each new publishing application simply writes to the RabbitMQ Node of the recording application.
The only thing I am struggling with is how to deal with a situation in which the Node of the recording application is down. Do I need to build my own system to store the messages until it's back online or can I use some functionality of RabbitMQ to deal with that? I.e. could the local RabbitMQ of each of the publishing applications just receive the messages and forward them to the recording application RabbitMQ as soon as it's back online?
I found something about the Federated plugin be couldn't understand if that's the solution. Maybe I need something different or maybe I have to write my own local queueing system (which I hope I don't have to) to queue messages when the target Node is offline.
Any links to architectural examples or solutions are more than welcome.
BTW: https://groups.google.com/forum/#!topic/easynetq/nILIKSjxyMg states that you shouldn't be installing a RabbitMQ Node for each application, so maybe I should resort to something like MSQM or ZeroMQ (?)
From experience in what sounds like a similar situation, I would suggest using something other than a queue to store the messages locally, when offline.
Years ago, I built a system that had to work offline - no network connection at all - and then had to push messages through a message queue to the central server, when the laptop was brought back to the office.
I solved this by using a local database (sqlite at the time) to store my messages when the message queue was not available.
You should do something similar. Use a local database or even a plain text file or CSV file to store your messages when RabbitMQ is offline. When it reconnects, read the messages from your local file system and send them through RabbitMQ.
This is a good strategy to use, even if you do not expect RabbitMQ to go offline. Frankly, it will go offline at some point and you will have to deal with it. You should be prepared for that situation, and having a local store for your messages will help that.
...
regarding rqm node per application: bad idea. this adds a ton of complexity to your system. You want as few RabbitMQ nodes as you can get away with. Meaning, 1 per system (a system being comprised of many applications) when possible... with the exception of RabbitMQ clusters for availability - but that's another line of questions and design, entirely.
...
I did an interview with Aria Stewart about designing for failure with RabbitMQ and messaging systems, and have a small excerpt where she talks about how networks fail.
The point is, the network or RabbitMQ or something will fail and you will need a solution like a local datastore so that you can recover when RabbitMQ comes back online.
I'm attempting to set up a pub/sub system. My technical director has suggested using Rabbitmq and STOMP for the project I've been assigned.
In this system I want exchanges to be created on the server side and passed to the user to subscribe to. I want a user to subscribe to an exchange and receive information as it becomes available. I do not want the client to have the ability to subscribe to arbitrary exchanges (or routing keys for topic exchanges, I'm not sure what system I want to use yet).
For example, if someone works for a company widgetInk, when they logged in to our website they would receive a connection to the widgetInk.whatever exchange and AllUsers.whatever exchange, but that's it. if the Client side attempted to subscribe to something else they'd get an error.
I've gone through all of the RabbitMQ tutorials and I've looked through their how to section. I found this basic article on access control, but having read it three times now I still don't know if rabbitmq is a good fit for my requirements.
Can Rabbitmq be configured for my requirements? What resources can I use to learn about Rabbitmq's permission system? Has anyone build or used a system like this? Would it be in my best interest to switch to ZeroMQ or ActiveMQ?
I'd like to mention another platform you might use: Autobahn (Open-Source) or WebMQ (based on Autobahn).
WebMQ provides these features (besides a lot of other out of the box):
PubSub over WebSocket (WAMP)
Authentication of client sessions (WAMP-CRA)
fine-grained configurable authorization for topics
If you want to go the Open-source/build-your-own road, here are some tutorials. If you want an integrated, commercially supported product with Web UI for configuration/administration, please get in contact.
Disclaimer: I am creator of Autobahn/WAMP and work for Tavendo.
I am a looking for proven tools to monitor performance on ActiveMQ 5.5. I come from an environment which used Glassfish and JMQ that can tell me rate of messages produced and consumed on any given destination using "imqcmd". Is there a like tool for ActiveMQ or a different way to go about it?
I see that there is a project at http://activemq.apache.org/activemq-performance-module-users-manual.html that will do some sort of performance reporting but it seems to be no more than a SNAPSHOT version that I cannot get to operate.
Any input would be appreciated.
there are several options for this: JMX, AMQ webconsole, other options
here are my notes on this...I opted to go with JMX and built a simple web app (JSP, jQuery, Google Charts, etc) to interface with JMX to gather queue stats, manage queues, etc...
http://www.consulting-notes.com/2010/08/monitoring-and-managing-activemq-with.html
I have tried and tested the JMX API and it is pretty simple to use and provides a vast number of statistics required for monitoring ActiveMQ.
But the problem is, i dont want to monitor my ActiveMQ remotely and also i dont want to use another API.To be more precise, i want to use the JMS API itself to get statistics related to various destinations and the broker itself.
Advisory messages seem to be an alternative but they provide limited Amount of Administrative Messages to monitor.
Any input is highly appreciated...
There is no built-in support for this. But you can implement a JMS topic which publishes the monitoring data every few seconds. Make the connection non-persistent so that it doesn't pile up when there are no listeners or when they loose connection.
Now you can write a client that connects to this topic and it will receive updates.
AMQ-2379 resulted in a broker plugin for grabbing statistics from destinations by sending a simple JMS message. Check out the docs that show how to use it here:
http://activemq.apache.org/statisticsplugin.html
The statistics plugin is available in the 5.3 release.
You can checkout this http://issues.apache.org/activemq/browse/AMQ-2379, it will be avaiable in upcoming 5.3.0 release
There's a blog post queued up to go on http://issues.apache.org/activemq/browse/AMQ-2379 - will post it in a couple of days or so