I am currently writing a Splunk Query to pull a report over the events and I am using this now to do it and it has to be using table only
index=1234 source="/apps/logs/*.log" AND "logType=API_RESPONSE"
| spath input=request
| spath input=response
| rename body.data.Item1 as Item1
| rename body.data.Item2 as Item2
| rename body.data.Item3 as Item3
| stats count by URI
| rename count as NumberofTimes_Called_URI, URI as URI_Called
| table Item1,Item2,Item3
Prerequisites
The API_RESPONSE is a JSON response
Item1, Item2, and Item3 are JSON fields in response.
Issue: Not able to render the Splunk table in the statistics for this part
| stats count by URI
| rename count as NumberofTimes_Called_URI, URI as URI_Called
individually the above is working but when i combine and render the table it's not working.
Please help me fix this problem.
"It's not working" is not a problem description, but I'm guessing you are getting all nulls in the results table. That's because stats is a transforming command so the only fields available after it are those used in it, namely count and URI.
Since table (also a transforming command) only displays Item1, Item2, and Item3, there is no need for stats or rename.
If you intend to add count and URI to the table then replace stats with eventstats, which is not transforming.
index=1234 source="/apps/logs/*.log" AND "logType=API_RESPONSE"
| spath input=request
| spath input=response
| rename body.data.Item1 as Item1
| rename body.data.Item2 as Item2
| rename body.data.Item3 as Item3
| eventstats count as NumberofTimes_Called_URI by URI
| rename URI as URI_Called
| table Item1,Item2,Item3, URI, NumberofTimes_Called_URI
Related
I am new to Splunk, trying to fetch the values from json request body. I am able to fetch values one by one by using
"json_extract(json,path)"
but I have more than 10 fields so I am trying to use
"json_extract(json,path1,path2..pathN)"
which is returning the json array.
But I am not getting how to read the values from json array and check if it is null or not.
eval keyValues ="json_extract(json,"firstname","lastname","dob")"
| table keyValues
output: ["testfirstname","testlastname","1/1/1999"]
["testfirstname","testlastname",null]
[null,"testlastname",null]
Can someone please help how to loop above json array and check the value, if it is null or not(eval isnotnull(firstname))
if your events are in JSON and it is easily accessible through field names then I suggest using the Spath command like below. Just replace your condition with YOUR_REQUIRED_CONDITION.
| makeresults
| eval json="{ \"firstname\": \"firstname123\", \"lastname\": \"lastname123\", \"dob\": \"1/1/1999\"}"
| spath input=json
| where YOUR_REQUIRED_CONDITION
You can also go with your approach but here you have to extract the same 10 fields and need to compare them as per your requirements, see the below code.
| makeresults
| eval json="{ \"firstname\": \"firstname123\", \"lastname\": \"lastname123\", \"dob\": \"1/1/1999\"}"
| eval keyValues =json_extract(json,"firstname","lastname","dob")
| table keyValues
| eval keyValues = json_array_to_mv(keyValues)
| eval firstname=mvindex(keyValues,0),lastname=mvindex(keyValues,1),dob=mvindex(keyValues,2)
| where YOUR_REQUIRED_CONDITION
If you share your SAMPLE events and the exact where conditions with us then we can provide the optimum solution.
Thanks
KV
You don't need to loop through the values, you need to treat them as multivalue fields, and expand/filter appropriately
For example:
index=ndx sourcetype=srctp fieldname.subname{}.value=*
| rename fieldname.subname{}.value as subname
| mvexpand subname
| stats count by subname
| fields - count
<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version="2.0"><ProductInventoryInfo><Product>7781105882846</Product><EmployeeID>12151</EmployeeID><Quantity>28</Quantity><CenterID>167551</CenterID></ProductInventoryInfo></EmployeeInventory>
<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version="2.0"><ProductInventoryInfo><Product>1781305782846</Product><EmployeeID>12152</EmployeeID><Quantity>18</Quantity><CenterID>167552</CenterID></ProductInventoryInfo></EmployeeInventory>
How to write splunk query from above splunk log which will fetch table like this .
Product EmployeeID Quantity CenterID
7781105882846 12151 28 167551
1781305782846 12152 18 167552
It would help to know what you've tried so far and how those attempts failed to meet your needs.
The trick is extracting fields from the XML. You could use a series of rex commands, but spath is simpler.
| makeresults
| eval data="<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version=\"2.0\"><ProductInventoryInfo><Product>7781105882846</Product><EmployeeID>12151</EmployeeID><Quantity>28</Quantity><CenterID>167551</CenterID></ProductInventoryInfo></EmployeeInventory>;<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version=\"2.0\"><ProductInventoryInfo><Product>1781305782846</Product><EmployeeID>12152</EmployeeID><Quantity>18</Quantity><CenterID>167552</CenterID></ProductInventoryInfo></EmployeeInventory>"
| eval data=split(data,";")
| mvexpand data
```The above is just for setting up test data```
```Parse the data```
| spath input=data ```Replace "data" with the name of the field containing the data, perhaps "_raw"```
```Simplify the field names```
| rename EmployeeInventory.ProductInventoryInfo.* as *
```Display the data```
| table Product EmployeeID Quantity CenterID
I've got an issue when trying to visualize in Google Data Studio some information from a denormalized table.
Context: I want to gather all the contact of a company and there related orders in a table in Big Query. Contacts can have no order or multiple orders. Following Big Query best practice, this table is denormalized and all the orders for a client are in arrays of struct. It looks like this:
Fields Examples:
+-------+------------+-------------+-----------+
| Row # | Contact_Id | Orders.date | Orders.id |
+-------+------------+-------------+-----------+
|- 1 | 23 | 2019-02-05 | CB1 |
| | | 2020-03-02 | CB293 |
|- 2 | 2321 | - | - |
|- 3 | 77 | 2010-09-03 | AX3 |
+-------+------------+-------------+-----------+
The issue is when I want to use this table as a data source in Data Studio.
For instance, if I build a table with Contact_Id as dimension, everything is fine and I can see all my contacts. However, if I add any dimensions from the Orders struct, all info from contact with no orders are not displayed. For instance, all info from Contact_Id 2321 is removed from the table.
Have you find any workaround to visualize these empty arrays (for instance as null values)?
The only solution I've found is to build an intermediary table with the orders unnested.
The way I've just discovered to work around this is to add an extra field in my DS-> BQ connector:
ARRAY_LENGTH(fields.orders) AS numberoforders
This will return zero if the array is empty - you can then create calculated fields within DataStudio - using the "numberoforders" field to force values to NULL or zero.
You can fix this behaviour by changing a little your query on the BigQuery connector.
Instead of doing this:
SELECT
Contact_id,
Orders
FROM myproject.mydataset.mytable
try this:
SELECT
Contact_id,
IF(ARRAY_LENGTH(Orders) > 0, Orders, [STRUCT(CAST(NULL AS DATE) AS date, CAST(NULL AS STRING) AS id)]) AS Orders
FROM myproject.mydataset.mytable
This way you are forcing your repeated field to have, at least, an array containing NULL values and hence Data Studio will represent those missing values.
Also, if you want to create new calculated fields using one of the nested fields, you should check before if the value is NULL to avoid filling all NULL values. For example, if you have a repeated and nested field which can be 1 or 0, and you want to create a calculated field swaping the value, you should do:
IF(myfield.key IS NOT NULL, IF(myfield.key = 1, 0, 1), NULL)
Here you can see what happens if you check before swaping and if you don't:
Original value No check Check
1 0 0
0 1 1
NULL 1 NULL
1 0 0
NULL 1 NULL
I have written a query to fetch the all java exception count wise in splunk. But this query fetch across all sourcetype.
java.*.*Exception NOT warn | rex "(?<rexexption>java*.*Exception)"| stats count by rexexption | table count,rexexption | sort count | reverse
Now I want all these exceptions per sourcetype wise.
Go to you splunk home\main page and click on data summary.
There you can see source , sourcetype and other fields. just select one sourcetype which you want to go for and edit your query.
Source => path where logs are stored
sourcetype => log files
if you know your sourcetype name then edit your search :
sourcetype = java_*log*.txt java.*.*Exception NOT warn | rex "(?<rexexption>java*.*Exception)"| stats count by rexexption | table count,rexexption | sort count | reverse
I am sending some data to splunk which looks like:
"Start|timestamp:1552607877702|type:counter|metricName:cache|count:34488378|End"
And then extracting the fields using a regex:
search "attrs.name"="service" | regex (Start)(.*)(End) | extract pairdelim="\"{|}" kvdelim=":"
After extraction, I can see the fields (type, metricName, count) under "INTERESTING FIELDS". How do I go about using these fields in a dashboard?
Thanks
search "attrs.name"="service" | regex (Start)(.*)(End) | extract pairdelim="\"{|}" kvdelim=":" | stats count by metricName
Or
search "attrs.name"="service" | regex (Start)(.*)(End) | extract pairdelim="\"{|}" kvdelim=":" | stats count by type
Or
search "attrs.name"="service" | regex (Start)(.*)(End) | extract pairdelim="\"{|}" kvdelim=":" | table type, metricName, count
should all give you a table, which can also be represented as a visualization. You can save any of these, or the original events, as a dashboard panel.
If you see a field listed in either the "Selected fields" or "Interesting fields" list then that means Splunk has extracted them and made them available for use. Use them by mentioning them by name in an SPL command such as table type, metricName, count or stats max(count) by metricName. Once you have the fields the rest is up to your imagination (and the rules of SPL).