My task is actually for self learning, I know the basics of Splunk and how does it work, overall it needs logs then further analysis is future part.
Right now, I am working in a small company without specific firewall device, just the router is there, I am planning to keep real time track of internet speed + data usage by each hosts and similar things, overall say I want to monitor all the network usage in this small WAN environment.
My question is, as splunk need logs then how can I get the log of all the network flow when there is just a router but no external device to create logs, that can be feeded to splunk later?
I tried google but there I find only preset softwares that automatically capture logs and show analysis while I just need logs only.
What you need is a way to get logs off your router to somewhere Splunk can parse them
Most typically, this is done by sending the router's syslog messages to a syslog collector like SC4S.
From there, Splunk can then get the messages processed and ingested for anaylsis
Related
I am trying to graph the IncomingLogEvents or IncomingBytes or PutLogEvents or whatever metric that would help me understand which log streams are the ones sending the most logs to a specific log group in Cloudwatch
Did anyone run into this? I was able to graph the IncomingBytes metric for log groups but not for logstreams within those log groups
I have several containers in my environment and each container sends their logs through a separate log stream within the same log group
Suddenly costs started rising due to errors on the containers and was able to identify which log group is causing it but I cannot find a way to identify which logstream is
Docker does not help either and of course I can check container logs but I want to be able to alert on an increase, hence detect when a logstream is sending more logs than normal as it will alert me on cost increase as on errors
I know I can monitor log errors with other log centralization tools or even with CloudWatch but need to know from Cloudwatch which container is the one sending the highest amount of logs
There used to be a metric that got deprecated and I cannot find any documentation that would help me use whatever metric or solution they replaced it with
The metric was "storedBytes": 0 which from now on, and since deprecation it will always be 0
Thank you for any help that you can provide me with and hope this question help others achieve their goals too
Eu
Cloudwatch dashboard but can make it for log groups only
Is there a way to prevent BackgroundTransfer from trying indefinitely to upload a file. Let's say one of my user is trying to upload a movie from the phone to Facebook. Facebook Graph API doesn't accept byte-range/resume/etc. Let's say the network is slow, less than 50 kbps. Under 50 kbps, BackgroundTransferService will restart the upload.
That being said, when testing my app, I've noticed that the uploaded restarted 4-5 times under my very slow 3G wifi router (yeah... I'm a mix of the two cases).
Will this behavior happen on a GSM/3G/4G network?
What think is that this behavior is totally welcome, on a Wifi, but not on a phone network, as data costs more.
[Edit]
I forgot one important info: I don't have internet on my WP, so that's why I ignore the behavior of BTS on a phone network.
Yes, the agent will try and reattempt the transfer if the connection is dropped. This is one of the benefits of using the agent, you let it worry about reattempting and network conditions so you don't have to. The API does all you a level of control over usage of cellular data via the TransferPreferences property. You could set this if you're concerned. Alternatively, let the user set their own preferences about data usage via the built in settings on the phone.
There is more information at http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh202955(v=vs.105).aspx#BKMK_TimelinessofCompletion
I want to know if Graphite can pull log data from Splunk to draw Graphs. I know Graphite can read data from Nagios, but want to know if it can pull from Splunk also.
You can also pull data via one of the Splunk SDKs - http://dev.splunk.com/view/sdks/SP-CAAADP7
There is an example on the developer site that shows pulling data from splunk and pushing it to Leftronic - http://dev.splunk.com/view/SP-CAAADSR
There also are a number of visual examples in the JavaScript SDK showing how to pull data from Splunk and visualize with other libraries - http://dev.splunk.com/view/javascript-sdk/SP-CAAAECM
Here's an app I wrote for Splunk that does exactly this: https://github.com/OnBeep/splunk_graphite
If the goal is to chart the data in splunk you can use the chart or timechart command in splunk.
If the gloat is to chart the splunk data in carbon/graphite, depending on the data that you wish to pull out of spunk you should be able to;
Create a save search in splunk
Use the cli or rest api to execute & gather the results of the save search
parse the results then push it into carbon.
This is how it works:
Carbon listen to receive data.
Carbon receives data and stores it in whisper.
Graphite reads from whisper and carbon cache and shows graphs.
There's no pull at all. Submitting data to carbon it's damn easy. It has two ports, one for simple tcp connect and submit one metric per line (metric.name metric.value metric.timestamp), or have a pickle port too.
Usually you will use Logstash or logster to parse application logs with regular expresion and any of those will take care of submitting the resulting metrics to carbon.
Also, if you have a software been able to submit real time metrics by udp, you can use statsd which will listen on udp and on a configured interval sum or average and submit to carbon with a lot of nice settings (like get the 95th percentile, etc).
In summary, I bet whatever log Splunk leave you, you will be able for sure to submit data to graphite.
I'm writing an application for the Mac App Store in Obj-C/Cocoa. The app processes .html files and does not require an internet connection.
I was wondering, what would be the best way to collect statistics? All I'm interested in is the number of files processed.
That way, on the app's home page, I can display XXX,XXX files processed.
I was thinking that I would just post to a web server whenever a file was converted, but that would considerably slow down the app and wouldn't work if the user was not connected to the internet.
You could accumulate the stats internally to be uploaded only every so often (each day, perhaps). You'd save the accumulated number across restarts using NSUserDefaults.
You should ask the user for permission to upload data, even something so seemingly innocuous as a count of processed files.
You'd use a simple HTTP request to upload the data. (You know it will be vulnerable to spoofing, right?) You should use the network reachability API to check whether the system is network connected before trying, so you don't force a dial-up, for example. The reachability API can't tell you that your connection will for sure succeed, so you should handle failure to connect gracefully.
Well, I tried to ask this question as a comment on this question, but I thought that maybe no one will notice it, so I decided to ask it as a separate one.
The question is about how to do real-time GPS tracking system things; if we have the following scenario:
Rather than connecting a GPS receiver to a PC, the user will have a mobile device with an integrated GPS receiver.
Location data will be sent over mobile network using GPRS data connection to a server side.
The data will be processed and a KML path file will be created and updated on time intervals and used to track the user using Google Earth.
The question is: what is the best method to accomplish this scenario for the server side; is it a web service, a web application, a windows service, a windows application or what exactly? Taking into account that the system will serve a number of users simultaneously, and that more users may use the system in the future(scalability issues).
Thank you in advance and I highly appreciate any help :)
What kind of device are you using exactly, something like this or something more sophisticated / configurable? If we assume that the device sends its data over TCP, I would consider the following approach with separate input/output processes:
Input: a process listening specific TCP port and storing incoming coordinates to database with a device id. Preferably, your listening loop must be able to handle simultaneous connections without them blocking each other.
Output: web application reading coordinates from database for a given device id and displaying them through the Google Earth API.
Use whatever programming language(s) you are familiar with.
For me there is a technical limitation/risk here -> the mobile device, and its connectivity.
1) What are your requirements? Do you need to support various mobile devices or will you focus on only one platform ?
2) More importantly, you have to understand that GPRS data connections differ from a PC connected to the Internet. There are various connection restrictions imposed by different mobile operators.
If I was to design such a system in order to minimise those risks I would go with a web server running on port 80 which the mobile devices would upload their Long/Lat through POST (or even GET to simplify things).
EDIT: Regarding scalability, it would be very easy to scale things up in the future using tried&tested load-balancing techniques.
EDIT2: Whichever technology you decide to use, i would HIGHLY recommend that the first thing you do is to mock up a prototype. Those connection restrictions could be show-stoppers. Ideally you need to explore them before you have made any serious investment.