First Event
06:09:17:362 INFO com.x.y.ConnApp - Making a GET Request
Second Event
06:09:17:480 INFO com.a.b.Response - Output Status Code: 200
Now I want to calculate duration of these two events for every request. I went over the solutions on splunk and Stack Overflow, but still can't get the proper result.
The easy answer is the transaction command, although it has a couple of drawbacks. The first is the command can be a resource hog. The other is can be "greedy" in that multiple requests might be taken to be a single transaction. We'll take care of the second issue with the maxevents option. There's not much we can do about the first except avoid using transaction.
index=foo ("Making a GET Request" OR "Output Status Code:")
| transaction maxevents=2 startswith="Making a GET Request" endswith="Output Status Code:"
| table duration
Another option uses the streamstats command to calculate the difference between adjacent events. This should perform better than transaction.
index=foo ("Making a GET Request" OR "Output Status Code:")
| streamstats window=2 range(_time) as duration
``` Erase the duration field for start events. ```
| eval duration = if(searchmatch("Making a GET Request"),"", duration)
| table _raw duration
Both queries assume the start and end events for different requests are not intermingled.
With the current logging messages, it will be tricky to group logs who are linked by the same source (imagine multiple calls who generate successive Making a GET messages)
In this case, I suggest to spread a ‘correlation Id’ in the logging message
Then you can identify exactly the messages who are triggered by the same source
This involve a change of the app logging function (you can search the following libs: log4/mcd/sleuth)
Related
What's the most efficient way to perform the following search?
Event occurs on index A at X time
Take X time and use it as a start point in index B
Search all occurrences of a field within index B, with additional filters, 5 minutes after that initial event time that occurred from index A
Example using Windows logs: after every successful login via event ID 4624 (index="security") for a particular user on a host, search all Sysmon event ID 1 (index="sysmon") process creation events on that specific host that occurred in a 5 minute window after the login event. My vision is to examine user logins on a particular host and correlate subsequent process creation events over a short period of time.
I've been trying to play with join, stats min(_time), and eval starttimeu, but haven't had any success. Any help/pointers would be greatly appreciated!
Have you tried map? The map command runs a search for each result of another search. For example:
index=security sourcetype=wineventlog EventCode=4624
```Set the latest time for the map to event time + 5 minutes (300 seconds)```
| eval latest=_time+300
| map search="search index=sysmon host=$host$ earliest=$_time$ latest=$latest$"
Names within $ are field names from the main search.
I've been looking at a recent event in Splunk with sourcetype WinHostMon, and I see two different values for StartTime and _time:
StartTime="20200427223006.448182-300"
_time is recorded as 2020-04-28T15:38:13.000-04:00
If the last part is timezone, there are two things that are strange about this:
The timezone for StartTime is in the middle of the Atlantic.
The times don't actually match.
Question: What is the actual time of this event, if such a thing can actually be determined, and what is causing the discrepancy between these two times?
(I tried to post this on Splunk Answers but they seem to have a labyrinth to stop people from signing up and I was unable to get an activated account.)
_time is the timestamp of the event, that is, when the event was generated or written to a log file. This is the field Splunk uses for default sorting and rendering in tables and time charts.
For WinHostMon events, most notably Process events, StartTime is when that process started.
Hence, it is not surprising that these events are significantly different. The process may have started at some point in the past, and then the WinHostMon input may generate a list of active processes every 5 minutes or so (or more or less)
_time is the timestamp of the event as defined in props.conf - or, if undefined, whenever Splunk receives the event (as often happens with untagged JSON)
The field StartTime is - so far as I can tell - not related to whatever is populating _time
If you open the Add-On's props.conf, you'll see how they're defining the timestamp and the field extraction for StartTime
I am trying to create a Splunk alert that will be triggered if two events do not occur in a certain time window. The two events will be linked by a GUID and there may be multiple events occurring with different GUIDs simultaneously.
Can someone indicate where to start?
There probably are a few ways to do this. The worst one is by using the transaction command because it's very slow.
Try using stats to find the time span of the events and alert on those that are too slow (5 minutes in this example).
... | stats range(_time) as duration by GUID | where duration > 300
I'm very new to Splunk and wanted to know if the following was possible: I'm trying to set up a dashboard of how many times we had to retry a call to a service. I am currently logging the following text:
number of retries required 0
The number of retries required can vary from 0 to 3
Is there an easy way to query this and display how many times it was either 0, 1, 2 or 3?
Thanks.
The gist of it is that you need to extract that piece of information into a field and than analyze that field according to your wishes (i.e. via timechart, chart, stats, etc.) Here are two different ways:
you can use the Field Extractor to extract and create a new field from the retries count. This is the recommended long-term option.
use the rex command to extract and define a new field inline.
search * | rex field=_raw ".+retries required (?<retries>\d)$"
Then you can chart them over time by appending | timechart retries or use the stats command to do some other calculations.
The intent here is to use the Splunk
transaction startswith
in conjunction with a query that specifies a pair of fields and a free form search string. What is the correct structuring/ordering of the query?
To start with: the following query returns a few records:
index=dev sourcetype="alstest-tuning--1-0-50--*" Error
Now we want to view some records following each of the results. Here are a couple of attempts:
transaction startswith("index=dev sourcetype=\"alstest-tuning--1-0-50--*\" Error") maxevents=20
and
index=dev sourcetype="alstest-tuning--1-0-50--*" transaction startswith("Error") maxevents=20
Neither of those approaches returns anything. The first one also surfaces a related question: how to escape the quotes for the fields. Pointers appreciated.
The way of using transaction is different. Let me brief on Splunk transaction command:
A transaction is any group of related events that span time
Events can come from multiple applications or hosts
Events related to single purchase from an online store can span across an application server, database, and e-commerce engine
One email message can create multiple events as it travels through various queues
Each event in the network traffic logs represents a single user generating a single http request
Visiting a single website normally generates multiple http requests
Syntax of transaction command
transaction [field-list] [name=transaction-name] [transaction_definition-opt]*
The events are grouped into transactions based on the values of this field list
If a quoted list of fields is specified, events are grouped together if they have the same value for each of the fields
Common constraints maxspan | maxpause | maxevetns | startswith | endswith
How to use transaction command:
combine all the events by SESSIONID
index=index_name sourcetype=some-source-type | transaction SESSIONID
With the transaction command, we can also use tables to easily view the information that we want: index=* | transaction SESSIONID | table SESSIONID, action, product_name
transaction command: startswith / endswith : To form transactions based on terms, field values or evaluations, use startswith & endswith options
Example: the first event in the transaction includes addtocart & the last event includes purchase
index=* sourcetype=access* | transaction clientip \
startswith=action="addtocart" endswith=action="purchase"
In your case, you need to use the last shown example.
NOTE: Use transaction when you need to see events correlated together and also must define event grouping based on start / end values.