I've been looking at a recent event in Splunk with sourcetype WinHostMon, and I see two different values for StartTime and _time:
StartTime="20200427223006.448182-300"
_time is recorded as 2020-04-28T15:38:13.000-04:00
If the last part is timezone, there are two things that are strange about this:
The timezone for StartTime is in the middle of the Atlantic.
The times don't actually match.
Question: What is the actual time of this event, if such a thing can actually be determined, and what is causing the discrepancy between these two times?
(I tried to post this on Splunk Answers but they seem to have a labyrinth to stop people from signing up and I was unable to get an activated account.)
_time is the timestamp of the event, that is, when the event was generated or written to a log file. This is the field Splunk uses for default sorting and rendering in tables and time charts.
For WinHostMon events, most notably Process events, StartTime is when that process started.
Hence, it is not surprising that these events are significantly different. The process may have started at some point in the past, and then the WinHostMon input may generate a list of active processes every 5 minutes or so (or more or less)
_time is the timestamp of the event as defined in props.conf - or, if undefined, whenever Splunk receives the event (as often happens with untagged JSON)
The field StartTime is - so far as I can tell - not related to whatever is populating _time
If you open the Add-On's props.conf, you'll see how they're defining the timestamp and the field extraction for StartTime
Related
First Event
06:09:17:362 INFO com.x.y.ConnApp - Making a GET Request
Second Event
06:09:17:480 INFO com.a.b.Response - Output Status Code: 200
Now I want to calculate duration of these two events for every request. I went over the solutions on splunk and Stack Overflow, but still can't get the proper result.
The easy answer is the transaction command, although it has a couple of drawbacks. The first is the command can be a resource hog. The other is can be "greedy" in that multiple requests might be taken to be a single transaction. We'll take care of the second issue with the maxevents option. There's not much we can do about the first except avoid using transaction.
index=foo ("Making a GET Request" OR "Output Status Code:")
| transaction maxevents=2 startswith="Making a GET Request" endswith="Output Status Code:"
| table duration
Another option uses the streamstats command to calculate the difference between adjacent events. This should perform better than transaction.
index=foo ("Making a GET Request" OR "Output Status Code:")
| streamstats window=2 range(_time) as duration
``` Erase the duration field for start events. ```
| eval duration = if(searchmatch("Making a GET Request"),"", duration)
| table _raw duration
Both queries assume the start and end events for different requests are not intermingled.
With the current logging messages, it will be tricky to group logs who are linked by the same source (imagine multiple calls who generate successive Making a GET messages)
In this case, I suggest to spread a ‘correlation Id’ in the logging message
Then you can identify exactly the messages who are triggered by the same source
This involve a change of the app logging function (you can search the following libs: log4/mcd/sleuth)
I am building an event reminder page where people can set a reminder for certain events. There is an option for the user to set the amount of time before they need to be notified. It is stored in notification_time and notification_unit. notification_time keeps track of the time before they want to be notified and notification_unit keeps track of the PHP date format in which they selected the time, eg. i for minutes, H for hours.
Eg. notification_time - 2 and notification_unit - H means they need to be notified 2 hours before.
I have Cron jobs running in the background for handling the notification. This function is being hit once every minute.
Reminder::where(function ($query) {
$query->where('event_time', '>=', now()->subMinutes(Carbon::createFromFormat('i', 60)->diffInMinutes() - 1)->format('H:i:s'));
$query->where('event_time', '<=', now()->subMinutes(Carbon::createFromFormat('i', 60)->diffInMinutes())->format('H:i:s'));
})
In this function, I am hard coding the 'i', 60 while it should be fetched from the database. event_time is also part of the same table
The table looks something like this -
id event_time ... notification_unit notification_time created_at updated_at
Is there any way to solve this issue? Is it possible to do the same logic with SQL instead?
A direct answer to this question is not possible. I found 2 ways to resolve my issue.
First solution
Mysql has DATEDIFF and DATE_SUB to get timestamp difference and subtract certain intervals from a timestamp. In my case, the function runs every minute. To use them, I have to refactor my database to store the time and unit in seconds in the database. Then do the calculation. I chose not to use this way because both operations are a bit heavy on the server-side since I am running the function every minute.
Second Solution
This is the solution that I personally did in my case. Here I did the calculations while storing it in the database. Meaning? Let me explain. I created a new table notification_settings which is linked to the reminder (one-one relation). The table looks like this
id, unit, time, notify_at, repeating, created_at, updated_at
The unit and time columns are only used while displaying the reminder. What I did is, I calculated when to be notified in the notify_at column. So in the event scheduler, I need to check for the reminders at present (since I am running it every minute). The repeating column is there to keep track of whether the reminder is repeating or not. If it is repeating I re-calculate the notify_at column at the time of scheduling. Once the user is notified notify_at is set to null.
Is there an TSI-Endpoint that provides the most recent message of a time series that arrived from e.g. an iot-hub? In my current situation I have to poll a certain period of time (for example now to 30 seconds in the past) and I wonder if there is a better way to do this?
Unfortunately there is no endpoint that lets you query only the last event. You can use GetEvents and query over the last 30 seconds, like you said, or peek the last message from the Event Hub/IoT Hub.
I am trying to create a Splunk alert that will be triggered if two events do not occur in a certain time window. The two events will be linked by a GUID and there may be multiple events occurring with different GUIDs simultaneously.
Can someone indicate where to start?
There probably are a few ways to do this. The worst one is by using the transaction command because it's very slow.
Try using stats to find the time span of the events and alert on those that are too slow (5 minutes in this example).
... | stats range(_time) as duration by GUID | where duration > 300
I am sending events with logstash to an elasticsearch database. An event is structured like this:
timestamp:2014-04-04 12:00:00 name:'leo' time:10
timestamp:2014-04-04 12:00:30 name:'john' time:15
...
...
In the Kibana interface, I am able to display some graphs; for example, the mean time of the timestamp.
Since logstash is continuously sending events, I would like to display the latest event sent in real time. Is it possible to write a query which will only return the latest event, using the timestamp field? I don't want to touch Kibana's "time filter"
Thanks in advance for your help
The easiest way to see the latest entry is to sort by timestamp, descending, in the 'Events' section of the Kibana interface. If you really only wanted to see one result, you could adjust the paging settings to only return one page consisting of one entry, and set the 'Auto-refresh' to a short interval (the shortest is 5s, I believe.)
Here's the Kibana documentation - might help.