I am new to Splunk dashboard. I need some help with this kind of data.
2020-09-22 11:14:33.328+0100 org{abc} INFO 3492 --- [hTaskExecutor-1] c.j.a.i.p.v.b.l.ReadFileStepListener : [] read-feed-file-step ended with status exitCode=COMPLETED;exitDescription= with compositeReadCount 1 and other count status as: BatchStatus(readCount=198, multiEntityAccountCount=0, readMultiAccountEntityAdjustment=0, accountFilterSkipCount=7, broadRidgeFilterSkipCount=189, writeCount=2, taskCreationCount=4)
I wanted to have statistics in a dashboard showing all the integer values in the above log.
Edit 1:
I tried this but not working.
index=abc xyz| rex field=string .*readCount=(?P<readCount>\d+) | table readCount
See if this run-anywhere example helps.
| makeresults
| eval _raw="2020-09-22 11:14:33.328+0100 org{abc} INFO 3492 --- [hTaskExecutor-1] c.j.a.i.p.v.b.l.ReadFileStepListener : [] read-feed-file-step ended with status exitCode=COMPLETED;exitDescription= with compositeReadCount 1 and other count status as: BatchStatus(readCount=198, multiEntityAccountCount=0, readMultiAccountEntityAdjustment=0, accountFilterSkipCount=7, broadRidgeFilterSkipCount=189, writeCount=2, taskCreationCount=4)"
`comment("Everything above just sets up test data")`
| extract kvdelim=",", pairdelim="="
| timechart span=1h max(*Count) as *Count
I solved this using
index=xyz |regex ".*fileName=(\s*([\S\s]+))" | rex field=string .*compositeReadCount=(?P<compositeReadCount>\d+) |regex ".*readCount=(?P<readCount>\d+)" | regex ".*multiEntityAccountCount=(?P<multiEntityAccountCount>\d+)" | regex ".*readMultiAccountEntityAdjustment=(?P<readMultiAccountEntityAdjustment>\d+)" | regex ".*accountFilterSkipCount=(?P<accountFilterSkipCount>\d+)" | regex ".*broadRidgeFilterSkipCount=(?P<broadRidgeFilterSkipCount>\d+)" | regex ".*writeCount=(?P<writeCount>\d+)" | regex ".*taskCreationCount=(?P<taskCreationCount>\d+)" | regex ".*status=(\s*([\S\s]+))" | table _time fileName compositeReadCount readCount multiEntityAccountCount readMultiAccountEntityAdjustment accountFilterSkipCount broadRidgeFilterSkipCount writeCount taskCreationCount status
Related
I have a table generated by chart that lists the results of a compliance scan
These results are typically Pass, Fail, and Error - but sometimes there is "Unknown" as a response
I want to show the percentage of each (Pass, Fail, Error, Unknown), so I do the following:
| fillnull value=0 Pass Fail Error Unknown
| eval _total=Pass+Fail+Error+Unknown
<calculate percentages for each field>
<append "%" to each value (Pass, Fail, Error, Unknown)>
What I want to do is eliminate a "totally" empty column, and only display it if it actually exists somewhere in the source data (not merely because of the fillnull command)
Is this possible?
I was thinking something like this, but cannot figure out the second step:
| eventstats max(Unknown) as _unk
| <if _unk is 0, drop the field>
edit
This could just as easily be reworded to:
if every entry for a given field is identical, remove it
Logically, this would look something like:
if(mvcount(values(fieldname))<2), fields - fieldname
Except, of course, that's not valid SPL
could you try that logic after the chart :
``` fill with null values ```
| fillnull value=null()
``` do 90° two time, droping empty/null ```
| transpose 0 include_empty=false | transpose 0 header_field=column | fields - column
[edit:] it is working when I do the following but not sure it is easy to make it working on all conditions
| stats count | eval keep=split("1 2 3 4 5"," ") | mvexpand keep
| table keep nokeep
| fillnull value=null()
| transpose 0 include_empty=false | transpose 0 header_field=column | fields - column
[edit2:] and if you need to add more null() could be done like that
| stats count | eval keep=split("1 2 3 4 5"," "), nokeep=0 | mvexpand keep
| table keep nokeep
| foreach nokeep [ eval nokeep=if(nokeep==0,null(),nokeep) ]
| transpose 0 include_empty=false | transpose 0 header_field=column | fields - column
I have used this query, but I cannot get the time chart to show the trend of the CPU. appears to be only showing the current cpu. my objective is to show the trend from each computer(host)
Telemetry_system
| where hostname_s contains "computer-A"
| where TimeGenerated > ago(5m)
| summarize
by
hostname,
callBackUrl,
cpu_d
| summarize Aggregatedcpu= avg(cpu_d) by hostname, callBackUrl
| render timechart
If I understand your intention correctly, try this:
Telemetry_system
| where hostname_s contains "computer-A"
| where TimeGenerated > ago(5m)
| summarize Aggregatedcpu = avg(cpu_d) by strcat(hostname, "_", callBackUrl), bin(TimeGenerated, 1s)
| render timechart
When I search "SearchText1" then lets say there are 20 records.
When I search "SearchText2" then there are 10 results
Then I need to display a single value "2" in the dashboard
How do I formulate the Splunk query?
I tried below query where the numerator count is evaluated correctly but something is wrong with the denominator count related part:
index=something "searchText1"
| stats count as NumeratorCount
| eval numerator=NumeratorCount
| append [ | search index=something "searchText2"
| stats count as DenominatorCount
| eval denominator=DenominatorCount ]
| eval result=round(if(denominator=0,0,numerator/denominator), 2)
| table result
When you remove the table command, you'll see the numerator and denominator are in separate results. The means the eval command computing 'result' is dividing numerator by NULL and NULL by denominator.
The fix is to combine the two rows using appendcols as in this example.
index=_internal "service_health_monitor"
| stats count as NumeratorCount
| eval numerator=NumeratorCount
| appendcols [ | search index=_internal "service_health_metrics_monitor"
| stats count as DenominatorCount
| eval denominator=DenominatorCount ]
| eval result=round(if(denominator=0,0,numerator/denominator), 2)
| table result
I have a Splunk Query to fetch top 5 API based on error percent. Below is the query for it
index=myaccount sourcetype=myaccountweb-master Response status=* url=* |
chart count over url by status | addtotals
| foreach * [
| eval <<FIELD>> = if('<<FIELD>>'==0,"-",'<<FIELD>>')
| eval p_<<MATCHSTR>> =
if(isnull(tonumber('<<FIELD>>')),'<<FIELD>>',round(('<<FIELD>>'/Total)*100,2))
| eval p_<<MATCHSTR>> = if('p_<<MATCHSTR>>'<1, "< 1",'p_<<MATCHSTR>>')
| eval <<FIELD>> = if("<<FIELD>>"=="Total",'<<FIELD>>', case('<<FIELD>>'=="-","-
",tonumber('<<FIELD>>')>1,'<<FIELD>>'." (".p_<<MATCHSTR>>."%)",1=1,'<<FIELD>>')) ]
| fields - p_* | eval url=lower(url) | rex mode=sed field=url
"s/account\/(\d+)\//account\/me\//" | rex mode=sed field=url
"s/\d+account.\w+|\d+fm|\d+fs\d+/*/g" | rex mode=sed field=url "s/..:..:..:..:..:../*/" | rex
mode=sed field=url "s/accounts\?ip=.*/accounts?ip=__/"| rex mode=sed field=url "s/[^\/]
{30,}/*/g" | rex mode=sed field=url "s/(\d|\.){8,}/*/g"
| rex field="500" "\d+\s\((?<perc>.*)%\)" | sort - perc | where perc>10 | head 5
I have URL's where userID comes in between and to replace those userID with * I have used rex commands and it works replacing the userID as *
But the issue is it counts them separately since userID differs for each hit made on the URL. Because of this my top5 API hits output differs.
Eg URL:/account/user/JHWERTYQMNVSJAIP/email where JHWERTYQMNVSJAIP is userID and its replaced by *
I am getting below output for the query
url 200 201 204 400 401 500
/account/user/*/email - - - - - 5 (100.00%)
/account/user/*/email - - - - - 4 (100.00%)
/account/user/*/email - - - - - 4 (100.00%)
Whereas all these URLs are actually one and the expected output should be like adding 5+4+4 and displaying once like this
url 200 201 204 400 401 500
/account/user/*/email - - - - - 13 (100.00%)
Since userID differs for each one, it take count separately. Any help on this would be appreciated. Thanks in advance
You have the right idea, but to get the numbers right normalization of the URL must be done before the numbers are calculated by the chart command.
I have a log which has below lines in it:
"Results":{"Elapsed":"0","Message":"No of Application to Obsolete in Teradata : 4","TraceLevel":"INFO"},"Security":{"Vendor":"CRAB"}}
"Results":{"Elapsed":"0","Message":"Total Application Asset in Teradata : 1696","TraceLevel":"INFO"},"Security":{"Vendor":"CRAB"}}
"Results":{"Elapsed":"0","Message":"Total Application count from SPAM : 1694","TraceLevel":"INFO"},"Security":{"Vendor":"CRAB"}}
"Results":{"Elapsed":"0","Message":" Application/s to Obsolete in Teradata : [PA00007618, PA00007617, PA00007619, PA00007620]","TraceLevel":"INFO"},"Security":{"Vendor":"CRAB"}}
I want the output to have the below fields like a summary and not like in 4 columns.
ExecutionDate Host Summary
02-24-2021 Production No of Application to Obsolete in Teradata : 4
Total Application Asset in Teradata : 1696
Total Application count from SPAM : 1694
Application/s to Obsolete in Teradata : [PA00007618,
PA00007617, PA00007619, PA00007620]
I have built below query but it's only giving me one record :
ExecutionDate Host Total Application count from SPAM : 1694
index=hdt sourcetype=Teradata_SPAM_logs | fields -_raw
| where match(_raw, "Host_cdc") and (match(_raw,"Total\sApplication\scount\sfrom\sSPAM\s*")
OR match(_raw,"Total\sApplication\sAsset\sin\sTeradata\s*")
OR match(_raw,"No\sof\sApplication\sto\sObsolete\sin\sTeradata\s*")
OR match(_raw,"List\sof\sApplications\sin\sTeradata\sto\sbe\smarked*")
)
| rex "(?<Summary>\"Message\":(.*\w+)\s:.*)"
| rex "(?<Host>\"Host\":(.*\",))"
| rex "(?<ExecutionDate>\d{4}\-\d{2}\-\d{2})"
| rex field=Summary mode=sed "s/\"Message\":\"/ /"
| rex field=Summary mode=sed "s/\"TraceLevel.*/ /"
| rex field=Summary mode=sed "s/\".*$//"
| rex field=Host mode=sed "s/\"Channel.*/ /"
| rex field=Host mode=sed "s/\"Host\":\"/ /"
| rex field=Host mode=sed "s/\/.*/ /"
| eval Host = replace(Host,"Host_cdc.cdc.CRAB.com", "PRODUCTION")
| eval Host = replace(Host,"Host_DEV.cdc.CRAB.com", "PROFILING")
| eval Host = replace(Host,"Host_PP.cdc.CRAB.com", "VALIDATION")
| stats values(Summary) as Summary by ExecutionDate, Host
| where isnotnull(Summary)
Can anyone tell me where is the problem here?
Based on your simplyfied example (i of course dont have all your other data so cannot run your full query but using the data you provided
"Results":{"Elapsed":"0","Message":"No of Application to Obsolete in Teradata : 4","TraceLevel":"INFO"},"Security":{"Vendor":"CRAB"}}
"Results":{"Elapsed":"0","Message":"Total Application Asset in Teradata : 1696","TraceLevel":"INFO"},"Security":{"Vendor":"CRAB"}}
"Results":{"Elapsed":"0","Message":"Total Application count from SPAM : 1694","TraceLevel":"INFO"},"Security":{"Vendor":"CRAB"}}
"Results":{"Elapsed":"0","Message":" Application/s to Obsolete in Teradata : [PA00007618, PA00007617, PA00007619, PA00007620]","TraceLevel":"INFO"},"Security":{"Vendor":"CRAB"}}
and the query
source="stack.csv" host="localhost" index="stack"
| rex "Message\":\"(?<message_value>[^\"]+)\""
| table message_value
i get the output
Application/s to Obsolete in Teradata : [PA00007618, PA00007617, PA00007619, PA00007620]
Total Application count from SPAM : 1694
Total Application Asset in Teradata : 1696
No of Application to Obsolete in Teradata : 4
you should then be able to just put that in your stats output with the vairable name message_value
This should work based on your example input. Of course, you may need to fiddle with it to adapt it to your event stream.
| makeresults
| eval myevent="\"Results\":{\"Elapsed\":\"0\",\"Message\":\"No of Application to Obsolete in Teradata : 4\",\"TraceLevel\":\"INFO\"},\"Security\":{\"Vendor\":\"CRAB\"}}
\"Results\":{\"Elapsed\":\"0\",\"Message\":\"Total Application Asset in Teradata : 1696\",\"TraceLevel\":\"INFO\"},\"Security\":{\"Vendor\":\"CRAB\"}}
\"Results\":{\"Elapsed\":\"0\",\"Message\":\"Total Application count from SPAM : 1694\",\"TraceLevel\":\"INFO\"},\"Security\":{\"Vendor\":\"CRAB\"}}
\"Results\":{\"Elapsed\":\"0\",\"Message\":\"Application/s to Obsolete in Teradata : [PA00007618, PA00007617, PA00007619, PA00007620]\",\"TraceLevel\":\"INFO\"},\"Security\":{\"Vendor\":\"CRAB\"}}"
| rex mode=sed field=myevent "s/([\r\n]+)/##LF##/g" | makemv myevent delim="##LF##" | rename myevent as myevent_new | mvexpand myevent_new
| rex field=myevent_new "Results\":(?<json_event>.*)"
| spath input=json_event
| eval ExecutionDate=strftime(now(),"%d-%m-%Y")
| eval Host="Production"
| stats values(Message) AS Summary by ExecutionDate Host
And the output: