My bar chart is built from this query:
index=$index$ namespace=$namespace$ host="*$host$*"
| timechart span=1h
eval(distinct_count(field1) - distinct_count(field2)) as "Re-processing"
count(eval(field3="entry" AND (field4="someValue" OR field4="someOtherValue"))) as "Input"
count(eval(field3="entry" AND (field4="object-notifications"))) as "Input2"
count(eval(executionStatus="FAILURE")) as "Failure"
| appendcols
[ search index=$index$ namespace=$namespace$ host="*$host$*" level=ERROR
| timechart span=1h
count(eval(like(log,"%Unhandled exception occurred%"))) as "Unhandled exceptions"
count(eval(like(log,"%These credentials do not authorize access%"))) as "Catalog Access Error"
count(eval(NOT like(log,"%Unhandled exception occurred%") AND NOT like(log,"%These credentials do not authorize access%"))) as "Errors" ]
I have a lot of series like "Input" and "Input2", so that the legend is so long I have two pages of it in a chart. I'd like to hide the series that are not shown in a current timespan on the chart from the legend. Curiously, another chart on the same dashboard is built from this:
index=$indexDas$ namespace=$namespaceDas$ host="$_env$-das-*"
event=PARTITION_COMPLETED
| timechart span=1h count by vendor
and it does that automatically. However, I can't eval all my series before timechart in the query in question, because there is eval(distinct_count(field1) - distinct_count(field2)) as "Re-processing" and appended columns. What would you suggest?
Related
First Event
17:09:05:362 INFO com.a.b.App - Making a GET Request and req-id: [123456]
Second Event
17:09:06:480 INFO com.a.b.App - Output Status Code: 200 req-id:"123456"
I tried to use index="xyz" container="service-name" | transaction "req-id" startswith="Making a GET Request" endswith="Output Status Code" | table duration but it is also not working.
I want to calculate duration of above two events for every request. I went over some solutions in splunk and Stack Overflow, but still can't get the proper result.
Try doing it with stats instead:
index=ndx sourcetype=srctp
| rex field=_raw "req\-id\D+(?<req_id>\d+)"
| rex field=_raw "(?<sequence>Making a GET Request)"
| rex field=_raw "(?<sequence>Output Status Code)"
| eval sequence=sequence+";"+_time
| stats values(sequence) as sequence by req_id
| mvexpand sequence
| rex field=sequence "(?<sequence>[^;]+);(?<time>\d+)"
| eval time=strftime(time,"%c")
This will extract the "req-id" into a field named req_id, and the start and end of the sequence into a field named sequence
Presuming the sample data you shared is correct, when you stats values(sequence) as sequence, it will put the "Making..." entry first and the "Output..." entry second
Because values() will do this, when you mvexpand and then split the values()'d field part into sequence and time, they'll be in the proper order
If the sample data is incomplete, you may need to tweak the regexes for populating sequence
It’s seem you’re going with my previously suggested approach 😉
Now you have 2 possibilities
1. SPL
Below the simplest query, only invoking 1 rex and assuming _time field correctly filled
index=<your_index> source=<your_source>
("*Making a GET Request*" OR "*Output Status Code*")
| rex field=_raw "req\-id\D+(?<req_id>\d+)"
| stats max(_time) as end, min(_time) as start by id
| eval duration = end - start
| table id duration
Note that depending the amount of data to scan, this one can be ressources consuming for your Splunk cluster
2. Log the response time directly in API (more efficient)
It seem you are working on an API. You must have capabilities to get the response time of each call and directly trace it in your log
Then you can exploit it easily in SPL without calculation
It always preferable to persist data at index time vs. operate systematic calculation at search time
I am able to get the multiple events (api's logs) in splunk dashboard like below
event-1:
{ "corrId":"12345", "traceId":"srh-1", "apiName":"api1" }
event-2:
{ "corrId":"69863", "traceId":"srh-2", "apiName":"api2" }
event-3:
{ "corrId":"12345", "traceId":"srh-3", "apiName":"api3" }
I want to retrieve corrId (ex:- "corrId":"12345") dynamically from one event (api log)by providing apiName and build splunk search query based on retrieved corrId value that means it will pull all the event logs which contains same corrId ("corrId":"12345").
Output
In above scenario expected results would be like below
event-1:
{ "corrId":"12345", "traceId":"srh-1", "apiName":"api1" }
event-3:
{ "corrId":"12345", "traceId":"srh-3", "apiName":"api3" }
I am new to splunk, please help me out here, how to fetch "corrId":"12345" dynamically by providing other field like apiName and build Splunk search query based on that.
I have tried out like below, but to no luck.
index = "test_srh source=policy.log [ search index = "test_srh source=policy.log | rex field=_raw "apiName":|s+"(?[^"]+)" | search name="api1" | table corrId]
This query gives event-1 log only but we need all other events which contain same corrId ("corrId":"12345"). Appreciate quick help here.
Given you're explicitly extracting the apiName field, I'll assume the corrId field is not automatically extracted, either. That means putting corrId="12345" in the base query won't work. Try index=test_srh source=policy.log corrId="12345" to verify that.
If the corrId field needs to be extracted then try this query.
index=test_srh source=policy.log
| rex "corrId\\":\\"(?<corrId>[^\\"]+)"
| where [ search index = "test_srh source=policy.log
| rex "apiName\":\"(?<name>[^\"]+)"
| search name="api1"
| rex "corrId\\":\\"(?<corrId>[^\\"]+)"
| fields corrId | format ]
Note: I also corrected the regex to properly extract the apiName field.
Full disclosure, I am very new Splunk so I may explain my question incorrectly.
I have two data sources and was given a query to pull data from them individually. I am trying to join this data together so I can create some type of chart, but I am unsure of this would be a join/search etc.
My initial query is as follows:
This allows me to search through the mail logs by sender address and show all emails with a bcSendAction=1, which is a successful send.
index=mail sourcetype=barracuda [search index=mail sourcetype=barracuda bcSender="someemail#domain.com" | table bcMsgId] bcSendAction=1
The result of this search is as follows:
Now, my other search is a log that shows all of the sender email addresses during a certain time period. I would like to use the result of this (the email value) in the first search so that I don't have to hard-code the bcSender, but rather have it use the results from the other source.
// Returns an email address
index=mail sourcetype=sendmail_syslog *#sfdc.net |
rex field=from "<(?<from>.*)>" |
table from | dedup from
I was able to parse the log and pull out just the email addresses that I want to use to plug into my first search.
I followed a few emails and tutorials, but a lot of the joins I was seeing only used two different sources/datasets and didn't use the search as I did in my first query.
My attempt at this was something like:
index=mail sourcetype=sendmail_syslog *#sfdc.net
| rex field=from "<(?<from>.*)>"
| table from | dedup from
| join from
[search index=mail sourcetype=barracuda [search index=mail sourcetype=barracuda bcSender=from | table bcMsgId] bcSendAction=1]
I don't know that I am referencing the email from the first result set correctly.
Can someone point me in the right direction with how to approach this search?
If I understand your request properly, then you need 3 steps:
get the sender addresses from index=mail sourcetype=sendmail_syslog
use these sender addresses to get a list of messageID's from index=mail sourcetype=barracuda
use these messageID's to finally get the events you are looking for
This sounds like you need a subsearch (for getting the sender addresses) inside of another subsearch (for getting the messageID's), meaning your own attempt was pointing in the right direction already.
Try something along these lines:
index=mail sourcetype=barracuda bcSendAction=1
[ search
index=mail sourcetype=barracuda
[ search
index=mail sourcetype=sendmail_syslog *#sfdc.net
| rex field=from "<(?<bcSender>.*)>"
| stats count by bcSender
| fields bcSender
| format
]
| stats count by bcMsgId
| fields bcMsgId
| format
]
I can not really verify it without having your data, but I'll try to explain what it's supposed to do. Let's start from the innermost subsearch.
Line 4 starts the innermost subsearch
Line 5 selects the events in from which you generate the address list
Line 6 extracts the addresses directly into the field bcSender. (We could extract it to the field from first and then rename it, but this is more direct.)
We need the fieldname to be bcSender for the outer search.
Line 7 is a different way to deduplicate by bcSender and at the same time reduce the amount of data which needs to be sent back from indexers to the searchhead (if you have a distributed environment).
Line 8 gets rid of all the fields we don't require. They would be problematic with the following format command.
Line 9 passes the results back to he enclosing search in a way so it can be used as part of the search string.
Line 10, of course, closes the innermost subsearch.
Now let's have a look at the outer subsearch.
Line 2 starts the subsearch.
Line 3 selects the events from which we can get the messageID's. This is, of cause, augmented by the enclosed subsearch we've just discussed.
Line 11 again is a way to dedup the messageID's.
Line 12 again limits things to the field we need.
Line 13 passes the found messageID's to the outermost (main) search in a such a way that they become part of the search string.
Line 14, you already know, closes the subsearch.
And the outermost search:
Line 1 selects the data you are targetting and is augmented by what the subsearches pass to it.
That one side of the join is a single field indicates it is a good candidate for a subsearch. Subsearches run first and their results then become part of the main search.
index=mail sourcetype=barracuda bcSendAction=1
[ search index=mail sourcetype=sendmail_syslog *#sfdc.net
| rex field=from "<(?<from>.*)>"
| fields from | rename from as bcSender | format ]
It's important that the result of the subsearch contain a field present in the main search. That's why I used rename.
After the subsearch runs, you get a search that's equivalent to this:
index=mail sourcetype=barracuda bcSendAction=1 (bcSender="someemail#domain.com" OR bcSender="anotheremail#domain.com")
I have been trying to display a chart in splunk. I uploaded my json data through Splunk HTTP Forwarder and running the query:
After I uploaded the json data, I have got fields such as
"message":{"acplbuild":"ACPL 1.20.1","coresyncbuild":"4.3.10.25","testregion":"EU_Stage","client":"EU_Mac","date":"2019-08-27","iteration":"20","localCreateTime":"6.672","createSyncTime":"135.768","createSearchTime":"0.679","filetype":"CPSD","filesize":"690_MB","filename":"690MB_NissPoetry.cpsd","operation":"upload","upload_DcxTime":"133.196","upload_manifest_time":"133.141","upload_journal_time":"1.753","upload_coresync_time":"135.225","upload_total_time":142.44},"severity":"info"}
I am trying to run the following query
index="coresync-ue1" host="acpsync_allacpl_7" message.testregion=EU_STAGE message.client=EU_Mac message.operation="upload" |eval roundVal = round(message.upload_total_time, 2) | chart median(roundVal) by message.acplbuild
I am getting no values. It should display rounded off median values as a chart. Can someone point me if I am doing anything wrong here.
I used the same data as specified by you and I faced an issue while rounding off the upload_total_time value. So, I first converted it to number, and then the Splunk search query worked.
Input Data Set
{"message":{"acplbuild":"ACPL 1.20.1","coresyncbuild":"4.3.10.25","testregion":"EU_Stage","client":"EU_Mac","date":"2019-08-27","iteration":"20","localCreateTime":"6.672","createSyncTime":"135.768","createSearchTime":"0.679","filetype":"CPSD","filesize":"690_MB","filename":"690MB_NissPoetry.cpsd","operation":"upload","upload_DcxTime":"133.196","upload_manifest_time":"133.141","upload_journal_time":"1.753","upload_coresync_time":"135.225","upload_total_time":142.44},"severity":"info"}
{ "message":{"acplbuild":"ACPL 1.20.2","coresyncbuild":"4.3.10.25","testregion":"EU_Stage","client":"EU_Mac","date":"2019-08-27","iteration":"20","localCreateTime":"6.672","createSyncTime":"135.768","createSearchTime":"0.679","filetype":"CPSD","filesize":"690_MB","filename":"690MB_NissPoetry.cpsd","operation":"upload","upload_DcxTime":"133.196","upload_manifest_time":"133.141","upload_journal_time":"1.753","upload_coresync_time":"135.225","upload_total_time":152.44123},"severity":"info"}
{ "message":{"acplbuild":"ACPL 1.20.3","coresyncbuild":"4.3.10.25","testregion":"EU_Stage","client":"EU_Mac","date":"2019-08-27","iteration":"20","localCreateTime":"6.672","createSyncTime":"135.768","createSearchTime":"0.679","filetype":"CPSD","filesize":"690_MB","filename":"690MB_NissPoetry.cpsd","operation":"upload","upload_DcxTime":"133.196","upload_manifest_time":"133.141","upload_journal_time":"1.753","upload_coresync_time":"135.225","upload_total_time":160.456},"severity":"info"}
Splunk Search Query
source="sample.json" index="splunk_answers" sourcetype="_json"
| convert num(message.upload_total_time) as total_upld_time
| eval roundVal = round(total_upld_time,2)
| chart median(roundVal) by message.acplbuild
Statistics View
Visualization View
I am gathering performance metrics for each each api that we have. With the below query I get results as
method response_time
Create Billing 2343.2323
index="dev-uw2" logger_name="*Aspect*" message="*ApiImpl*" | rex field=message "PerformanceMetrics - method='(?<method>.*)' execution_time=(?<response_time>.*)" | table method, response_time | replace "public com.xyz.services.billingservice.model.Billing com.xyz.services.billingservice.api.BillingApiImpl.createBilling(java.lang.String)” WITH "Create Billing” IN method
If the user clicks on each api text in table cell to drill down further it will open a new search with "Create Billing" obviosuly it will give zero results since we don't have any log with that string.
I want splunk to search with original text that was replaced earlier.
You can use click.value to get around this.
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Viz/tokens