I need a Splunk query to fetch the counts of each field used in my dashboard.
Splunk sample data for each search is like this
timestamp="2022-11-07 02:06:38.427"
loglevel="INFO" pid="1"
thread="http-nio-8080-exec-10"
appname="my-test-app"
URI="/testapp/v1/mytest-app/dashboard-service"
RequestPayload="{\"name\":\"test\",\"number\":\"\"}"
What would a search look like to print a table with the number of times the name and number is used to search data (at a time only either number/name data can be given by user).
Expected output in table format with counts for Name and Number
#Hanuman
Can you please try this? You can change regular expression as per your events and match with JSON data.
YOUR_SEARCH | rex field=_raw "RequestPayload=\"(?<data>.*[}])\""
| spath input=data
|table name number
My Sample Search:
| makeresults | eval _raw="*timestamp=\"2022-11-07 02:06:38.427\" loglevel=\"INFO\" pid=\"1\" thread=\"http-nio-8080-exec-10\" appname=\"my-test-app\" URI=\"/testapp/v1/mytest-app/dashboard-service\" RequestPayload=\"{\"name\":\"test\",\"number\":\"1\"}\"*"
| rex field=_raw "RequestPayload=\"(?<data>.*[}])\""
| spath input=data
|table name number
Screen
Thanks
Related
I have multiple log messages each containing a list of JobIds -
IE -
1. `{"JobIds":["661ce07c-b5f3-4b37-8b4c-a0b76d890039","db7a18ae-ea59-4987-87d5-c80adefa4475"]}`
2. `{"JobIds":["661ce07c-b5f3-4b37-8b4c-a0b76d890040","db7a18ae-ea59-4987-87d5-c80adefa4489"]}`
3. `{"JobIds":["661ce07c-b5f3-4b37-8b4c-a0b76d890070"]}`
I have a rex to get those jobIds. Next I want to count the number of jobIds
My query looks like this -
| rex field=message "\"(?<job_ids>(?:\w+-\w+-\w+-\w+-\w+)+),?\""
| stats count(job_ids)
But this will only give me a count of 3 when I am looking for 5. How can I get a count of all jobIds? I am not sure if this is a splunk limitation or I am missing something in my regex.
Here is my regex - https://regex101.com/r/vqlq5j/1
Also with max-match=0 but with mvcount() instead of mvexpand():
| makeresults count=3 | streamstats count
| eval message=case(count=1, "{\"JobIds\":[\"a1a2a2-b23-b34-d4d4d4\", \"x1a2a2-y23-y34-z4z4z4\"]}", count=2, "{\"JobIds\":[\"a1a9a9-b93-b04-d4d4d4\", \"x1a9a9-y93-y34-z4z4z4\"]}", count=3, "{\"JobIds\":[\"a1a9a9-b93-b04-d14d14d14\"]}")
``` above is test data setup ```
``` below is the actual query ```
| rex field=message max_match=0 "\"(?<id>[\w\d]+\-[\w\d]+\-[\w\d]+\-[\w\d]+\")"
| eval cnt=mvcount(id)
| stats sum(cnt)
In Splunk, to capture multiple matches from a single event, you need to add max_match=0 to your rex, per docs.Splunk
But to get them then separated into a singlevalue field from the [potential] multivalue field job_ids that you made, you need to mvxepand or similar
So this should get you closer:
| rex field=message max_match=0 "\"(?<job_id>(?:\w+-\w+-\w+-\w+-\w+)+),?\""
| mvexpand job_id
| stats dc(job_id)
I also changed from count to dc, as it seems you're looking for a unique count of job IDs, and not just a count of how many in total you've seen
Note: if this is JSON data (and not JSON-inside-JSON) coming into Splunk, and the sourcetype is configured correctly, you shouldn't have to manually extract the multivalue field, as Splunk will do it automatically
Do you have a full set of sample data (a few entire events) you can share?
<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version="2.0"><ProductInventoryInfo><Product>7781105882846</Product><EmployeeID>12151</EmployeeID><Quantity>28</Quantity><CenterID>167551</CenterID></ProductInventoryInfo></EmployeeInventory>
<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version="2.0"><ProductInventoryInfo><Product>1781305782846</Product><EmployeeID>12152</EmployeeID><Quantity>18</Quantity><CenterID>167552</CenterID></ProductInventoryInfo></EmployeeInventory>
How to write splunk query from above splunk log which will fetch table like this .
Product EmployeeID Quantity CenterID
7781105882846 12151 28 167551
1781305782846 12152 18 167552
It would help to know what you've tried so far and how those attempts failed to meet your needs.
The trick is extracting fields from the XML. You could use a series of rex commands, but spath is simpler.
| makeresults
| eval data="<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version=\"2.0\"><ProductInventoryInfo><Product>7781105882846</Product><EmployeeID>12151</EmployeeID><Quantity>28</Quantity><CenterID>167551</CenterID></ProductInventoryInfo></EmployeeInventory>;<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version=\"2.0\"><ProductInventoryInfo><Product>1781305782846</Product><EmployeeID>12152</EmployeeID><Quantity>18</Quantity><CenterID>167552</CenterID></ProductInventoryInfo></EmployeeInventory>"
| eval data=split(data,";")
| mvexpand data
```The above is just for setting up test data```
```Parse the data```
| spath input=data ```Replace "data" with the name of the field containing the data, perhaps "_raw"```
```Simplify the field names```
| rename EmployeeInventory.ProductInventoryInfo.* as *
```Display the data```
| table Product EmployeeID Quantity CenterID
In Splunk, I need to get the count of events from the below msg field value which matches factType=COMMERCIAL and has filters.
Using the basic Splunk query with wildcard does not work efficiently. Could you please assist
app_name="ABC" cf_space_name=prod msg="*/facts?factType=COMMERCIAL&sourceSystem=ADMIN&sourceOwner=ABC&filters*"
msg: abc.asia - [2021-08-23T00:27:08.152+0000] "GET
/facts?factType=COMMERCIAL&sourceSystem=ADMIN&sourceOwner=ABC&filters=%257B%2522stringMatchFilters%2522:%255B%257B%2522key%2522:%2522BFEESCE((json_data-%253E%253E'isNotSearchable')::boolean,%2520false)%2522,%2522value%2522:%2522false%2522,%2522operator%2522:%2522EQ%2522%257D%255D,%2522multiStringMatchFilters%2522:%255B%257B%2522key%2522:%2522json_data-%253E%253E'id'%2522,%2522values%2522:%255B%25224970111%2522%255D%257D%255D,%2522containmentFilters%2522:%255B%255D,%2522nestedMultiStringMatchFilter%2522:%255B%255D,%2522nestedStringMatchFilters%2522:%255B%255D%257D&sorts=%257B%2522sortOrders%2522:%255B%257B%2522key%2522:%2522id%2522,%2522order%2522:%2522DESC%2522%257D%255D%257D&pagination=null
Try this:
index=ndx sourcetype=srctp msg=*
| rex field=msg "factType=(?<facttype>\w+).(?<params>.+)"
| stats count by facttype params
| fields - count
| search facttype="commercial"
The rex will extract the facttype and any following parameters (note - if the URL is submitted with the arguments in a different order, you'll need to adjust the regular expression)
Then use a | stats count by to bin them together
Lastly, search only where there is both a facttype="commercial" and the URL has additional parameters
My splunk result looks like this:
9/1/20
5:00:14.487 PM
2020-09-01 16:00:14.487, 'TOTALITEMS'="Number of items registered in the last 2 hours ", COUNT(*)="1339"
I am trying to table the number that appears in the end in quotes.
index=my_db sourcetype=no_of_items_registered source=P_No_of_items_registered_2hours | rex field=_raw "\"Number of items registered in the last 2 hours \", COUNT(\*)=\"(?P<itm_ct>\d+)\"$" | table itm_ct
This displays a blank table without any numbers. The number of rows in the table however matches the the number of events.
Any help much appreciated
The regular expression doesn't match the sample data. Literal parentheses must be escaped in the regex. Try this:
index=my_db sourcetype=no_of_items_registered source=P_No_of_items_registered_2hours
| rex "COUNT\(\*\)="(?<itm_ct>\d+)" | table itm_c
I have 3 fields in my splunk result like message, id and docId.
Need to group the results by id and doc id which has specific messages
message="successfully added" id=1234 docId =1345
message="removed someUniqueId" id=1234 docId =1345
I have to group based on the results by both id's which has the specific message
search query | rex "message=(?<message[\S\s]*>)" | where message="successfully added"
which is giving result for the first search, when i tried to search for second search query which is not giving result due to the someUniqueId"
search query | rex "message=(?<message[\S\s]*>)" | where match(message, "removed *")
Could you pelase help me to filter the results which has the 2 messages and group by id and docID
The match function expects a regular expression, not a pattern, as the second argument. Try search query | rex "message=(?<message>[\S\s]*)" | where match(message, "removed .*").
BTW, the regex strings in the rex commands are invalid, but that may be a typing error in the question.