I recreated the dashboard using the report query and have the search returning all of the table results. I have an input for the reference number as a text box. The token name is: purchCostReferenceToken
I want to limit the table results based on this token. This is the query:
<form>
<label>Thru Train Dashboard</label>
<fieldset submitButton="false" autoRun="true">
<input type="text" token="purchCostReferenceToken" searchWhenChanged="true">
<label>Enter a TMS Reference Number to Filter Table</label>
<default>*</default>
<initialValue>*</initialValue>
</input>
</fieldset>
<row>
<panel>
<title>Thru Train Data</title>
<table>
<search>
<query>index=... "<billingMethod>RULE</billingMethod>" "createMessage MsgSource" | xmlkv | rex max_match=0 "\<purchasedCostTripSegment\>(?P<segment>[^\<]+)" |eval Segments = mvrange(1,mvcount(mvindex(segment, 0, 2))+1,1) | rex max_match=0 "\<carrier\>(?P<Carriers>[^\<]+)" | rex max_match=0 "\<billingMethod\>(?P<BillingMethod>[^\<]+)" | rex max_match=0 "<purchasedCostTripSegment>[\s\S]*?<origin>\s*<ns2:numberCode>(?P<Origin>\d+)" | rex max_match=0 "<purchasedCostTripSegment>[\s\S]*?<destination>\s*<ns2:numberCode>(?P<Destination>\d+)" | rex max_match=0 "<purchasedCostTripSegment>[\s\S]*?<stopOff>\s*<ns2:stopOffLocation>\s*<ns2:numberCode>(?P<StopOffLocation>\d+)" | eval Time =_time | convert timeformat="%m-%d-%Y %H:%M:%S" ctime(Time) | table purchCostReference, eventType, Time, Segments, Carriers, BillingMethod, Origin, Destination, StopOffLocation | sort Time</query>
<earliest>-30d#d</earliest>
<latest>now</latest>
</search>
<option name="drilldown">none</option>
</table>
</panel>
</row>
</form>
Where do I add the token to limit the search?
I tried adding this to the end of the query before the table command:
... | eval Time =_time | convert timeformat="%m-%d-%Y %H:%M:%S" ctime(Time) purchCostReference=$purchCostReferenceToken$ | table purchCostReference, eventType, Time, Segments, Carriers, BillingMethod, Origin, Destination, StopOffLocation | sort Time
I get an error...error in convert command: the argument purchCostReference- is invalid
I would like to add filters in several of the table columns. The purchCostReference value is an extracted field in the query using xmlkv
Technically, a token can be placed anywhere in a query, but the query has to be valid when the token is replaced with its value. convert timeformat="%m-%d-%Y %H:%M:%S" ctime(Time) purchCostReference=4 is not valid SPL.
If the field referenced by the token is extracted automatically then it's usually best to put the token in the base search. That's not the case here.
You should use a search or where command to filter events based on the token value. Something like xmlkv | search purchCostReference=$purchCostReferenceToken$.
Related
I have multiple log messages each containing a list of JobIds -
IE -
1. `{"JobIds":["661ce07c-b5f3-4b37-8b4c-a0b76d890039","db7a18ae-ea59-4987-87d5-c80adefa4475"]}`
2. `{"JobIds":["661ce07c-b5f3-4b37-8b4c-a0b76d890040","db7a18ae-ea59-4987-87d5-c80adefa4489"]}`
3. `{"JobIds":["661ce07c-b5f3-4b37-8b4c-a0b76d890070"]}`
I have a rex to get those jobIds. Next I want to count the number of jobIds
My query looks like this -
| rex field=message "\"(?<job_ids>(?:\w+-\w+-\w+-\w+-\w+)+),?\""
| stats count(job_ids)
But this will only give me a count of 3 when I am looking for 5. How can I get a count of all jobIds? I am not sure if this is a splunk limitation or I am missing something in my regex.
Here is my regex - https://regex101.com/r/vqlq5j/1
Also with max-match=0 but with mvcount() instead of mvexpand():
| makeresults count=3 | streamstats count
| eval message=case(count=1, "{\"JobIds\":[\"a1a2a2-b23-b34-d4d4d4\", \"x1a2a2-y23-y34-z4z4z4\"]}", count=2, "{\"JobIds\":[\"a1a9a9-b93-b04-d4d4d4\", \"x1a9a9-y93-y34-z4z4z4\"]}", count=3, "{\"JobIds\":[\"a1a9a9-b93-b04-d14d14d14\"]}")
``` above is test data setup ```
``` below is the actual query ```
| rex field=message max_match=0 "\"(?<id>[\w\d]+\-[\w\d]+\-[\w\d]+\-[\w\d]+\")"
| eval cnt=mvcount(id)
| stats sum(cnt)
In Splunk, to capture multiple matches from a single event, you need to add max_match=0 to your rex, per docs.Splunk
But to get them then separated into a singlevalue field from the [potential] multivalue field job_ids that you made, you need to mvxepand or similar
So this should get you closer:
| rex field=message max_match=0 "\"(?<job_id>(?:\w+-\w+-\w+-\w+-\w+)+),?\""
| mvexpand job_id
| stats dc(job_id)
I also changed from count to dc, as it seems you're looking for a unique count of job IDs, and not just a count of how many in total you've seen
Note: if this is JSON data (and not JSON-inside-JSON) coming into Splunk, and the sourcetype is configured correctly, you shouldn't have to manually extract the multivalue field, as Splunk will do it automatically
Do you have a full set of sample data (a few entire events) you can share?
Say I have a query such as
index="example" source="example.log" host="example" "ERROR 1234"
| stats distinct_count by id
This will give me all the events with that error code per id.
I then want to combine this query to search the same log file for another string but only on the unique id's returned from the first search. Because the new string will appear on a separate event I can't just do an 'AND'.
There are a few ways to do that, including using subsearches, join, or append, but those require multiple passes through the data. Here is a way that makes a single pass through the index.
index=example source="example.log" ("ERROR 1234" OR "ERROR 5678")
``` Check for the presence of each string in the event ```
| eval string1=if(searchmatch("ERROR 1234"), 1, 0)
| eval string2=if(searchmatch("ERROR 5678"), 1, 0)
``` Count string occurrences by id ```
| stats sum(string1) as string1, sum(string2) as string2 by id
``` Keep only the ids that have both strings ```
| where (string1 > 0 AND string2 > 0)
You can search for "some other string" in subsearch and then join the queries on the id:
index="example" source="example.log" host="example" "ERROR 1234"
| join id [search index="example" source="example.log" host="example" "some other string" ]
| stats distinct_count by id
Presuming your id field is the same and available in both indices, this form should work:
(index=ndxA sourcetype=srctpA id=* source=example.log host=example "ERROR 1234") OR (index=ndxB sourcetype=srctpB id=* "some other string")
| rex field=_raw "(?<first_field>ERROR 1234)"
| rex field=_raw "(?<second_field>some other string)"
| fillnull value="-" first_field second_field
| stats count by id first_string second_string
| search NOT (first_string="-" OR second_string="-")
If your id field has a different name in the other index, do a rename like this before the stats line:
| rename otherIdFieldName as id
Advantages of this format:
you are not limited by subsearch constraints (search must finish in 60 seconds, no more than 50k rows)
the Search Peers (ie Indexers) will handle all of the overhead instead of having to wait on the Search Head that initiated the search to do lots of post-processing (all the SH is doing is sending the distributed search, then a post-stats filter to ensure both first_string and second_string have the values you are looking for)
I need a Splunk query to fetch the counts of each field used in my dashboard.
Splunk sample data for each search is like this
timestamp="2022-11-07 02:06:38.427"
loglevel="INFO" pid="1"
thread="http-nio-8080-exec-10"
appname="my-test-app"
URI="/testapp/v1/mytest-app/dashboard-service"
RequestPayload="{\"name\":\"test\",\"number\":\"\"}"
What would a search look like to print a table with the number of times the name and number is used to search data (at a time only either number/name data can be given by user).
Expected output in table format with counts for Name and Number
#Hanuman
Can you please try this? You can change regular expression as per your events and match with JSON data.
YOUR_SEARCH | rex field=_raw "RequestPayload=\"(?<data>.*[}])\""
| spath input=data
|table name number
My Sample Search:
| makeresults | eval _raw="*timestamp=\"2022-11-07 02:06:38.427\" loglevel=\"INFO\" pid=\"1\" thread=\"http-nio-8080-exec-10\" appname=\"my-test-app\" URI=\"/testapp/v1/mytest-app/dashboard-service\" RequestPayload=\"{\"name\":\"test\",\"number\":\"1\"}\"*"
| rex field=_raw "RequestPayload=\"(?<data>.*[}])\""
| spath input=data
|table name number
Screen
Thanks
Consider the following log data,
{"level":"info","msg":"Enqueued a recurring task","time":"2022-04-01T18:46:22.854684-07:00","uuid":"4e9fc098-f611-4128-ae0b-2e6b9cb232c8"}
{"level":"info","msg":"Reported a result","path":"/results","time":"2022-04-01T18:46:22.955999-07:00","uuid":"4e9fc098-f611-4128-ae0b-2e6b9cb232c8"}
{"level":"info","msg":"Reported a result","path":"/results","time":"2022-04-01T18:46:23.056295-07:00","uuid":"4e9fc098-f611-4128-ae0b-2e6b9cb232c8"}
{"level":"info","msg":"Enqueued a recurring task","time":"2022-04-01T18:46:23.056376-07:00","uuid":"28e9bea9-5d0c-4dd5-af4f-c22944fc4fcd"}
It represents enqueuing a recurring task with a certain uuid, the results of which can be reported multiple times (or not at all) with the same uuid. I'm interested in determining the interval elapsed between when the task was enqueued and when the result was reported for the first time. So far, I can display the results this outer join table,
msg="Enqueued a recurring task"
| join type=outer left=L right=R where L.uuid = R.uuid
[ search msg="Reported a result" | dedup uuid sortby +_time]
| fillnull value=""
| table _time, L.msg, L.uuid, R.msg, L.time, R.time
What I'd like for convenience is to add an additional column with the difference between R.time and L.time. As far as I can tell from how to calculate duration between two events Splunk, one way to do this is to use strptime to convert those time fields into time values and then determine their difference. However, the time of the events was already parsed when importing the data (as seen from the built-in _time field) so this seems inefficient to me. Is there a more efficient way?
Update
Even the 'inefficient' method using strptime is proving tricky; I've added eval fields t to each of the searches,
msg="Enqueued a recurring task"
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")
| join type=outer left=L right=R where L.uuid = R.uuid
[ search msg="Reported a result"
| dedup uuid sortby +_time
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")]
| fillnull value=""
| table _time, L.msg, L.uuid, R.msg, L.time, R.time, L.t, R.t, R.t-L.t
and it appears to parse the time correctly, yet the difference R.t-L.t shows up empty in the table:
Any idea why it's not showing the difference?
Update 2
I've tried both of RichG's answers, but they both lead to the following error in Splunk:
Error in 'eval' command: Type checking failed. '-' only takes numbers.
(See screenshots below). The way I'm running Splunk is with a Docker container,
docker run -p 8000:8000 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=supersecret" --name splunk splunk/splunk:latest
Update 3
I finally got this to work using RichG's updated answer, combining both an evaluated strftime field with a diff field where R.t and L.t are quoted:
msg="Enqueued a recurring task"
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")
| join type=outer left=L right=R where L.uuid = R.uuid
[ search msg="Reported a result"
| dedup uuid sortby +_time
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")]
| fillnull value=""
| eval diff='R.t'-'L.t'
| table _time, L.msg, L.uuid, R.msg, L.time, R.time, L.t, R.t, diff
The time difference is empty in the table because the table command does not perform arithmetic. You must calculate the difference in a separate eval and display the result in the table.
index=foo msg="Enqueued a recurring task"
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")
| join type=outer left=L right=R where L.uuid = R.uuid
[ search index=foo msg="Reported a result"
| dedup uuid sortby +_time
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")]
| fillnull value=""
| eval diff='R.t' - 'L.t'
| table _time, L.msg, L.uuid, R.msg, L.time, R.time, L.t, R.t, diff
You should be able to use each event's _time field to avoid having to parse the time field.
index=foo msg="Enqueued a recurring task"
| join type=outer left=L right=R where L.uuid = R.uuid
[ search index=foo msg="Reported a result"
| dedup uuid sortby +_time ]
| fillnull value=""
| eval diff='R._time' - 'L._time'
| table _time, L.msg, L.uuid, R.msg, L.time, R.time, L.t, R.t, diff
Fairly confident something akin to this should work (without using join:
index=ndx sourtype=srctp uuid=* msg=*
| stats min(_time) as first_time max(_time) as last_time earliest(msg) as first_msg latest(msg) as last_msg by uuid
| eval diff_seconds=last_time-first_time
| eval first_time=strftime(first_time,"%c"), last_time=strftime(last_time,"%c")
This approach will presume that _time has been set properly in the sourcetype's props.conf, but if it has, this gets you what you're looking for in one pass.
I have a log statement like 2017-06-21 12:53:48,426 INFO transaction.TransactionManager.Info:181 -{"message":{"TransactionStatus":true,"TransactioName":"removeLockedUser-1498029828160"}} .
How can i extract TransactionName and TranscationStatus and print in table form TransactionName and its count.
I tried below query but didn't get any success. It is always giving me 0.
sourcetype=10.240.204.69 "TransactionStatus" | rex field=_raw ".TransactionStatus (?.)" |stats count((status=true)) as success_count
Solved it with this :
| makeresults
| eval _raw="2017-06-21 12:53:48,426 INFO transaction.TransactionManager.Info:181 -{\"message\":{\"TransactionStatus\":true,\"TransactioName\":\"removeLockedUser-1498029828160\"}}"
| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"
| rex "{\"TransactionStatus\":(?[^,]),\"TransactioName\":\"(?[^\"])\""
| chart count OVER TransactioName BY TransactionStatus