I am new to Splunk, trying to fetch the values from json request body. I am able to fetch values one by one by using
"json_extract(json,path)"
but I have more than 10 fields so I am trying to use
"json_extract(json,path1,path2..pathN)"
which is returning the json array.
But I am not getting how to read the values from json array and check if it is null or not.
eval keyValues ="json_extract(json,"firstname","lastname","dob")"
| table keyValues
output: ["testfirstname","testlastname","1/1/1999"]
["testfirstname","testlastname",null]
[null,"testlastname",null]
Can someone please help how to loop above json array and check the value, if it is null or not(eval isnotnull(firstname))
if your events are in JSON and it is easily accessible through field names then I suggest using the Spath command like below. Just replace your condition with YOUR_REQUIRED_CONDITION.
| makeresults
| eval json="{ \"firstname\": \"firstname123\", \"lastname\": \"lastname123\", \"dob\": \"1/1/1999\"}"
| spath input=json
| where YOUR_REQUIRED_CONDITION
You can also go with your approach but here you have to extract the same 10 fields and need to compare them as per your requirements, see the below code.
| makeresults
| eval json="{ \"firstname\": \"firstname123\", \"lastname\": \"lastname123\", \"dob\": \"1/1/1999\"}"
| eval keyValues =json_extract(json,"firstname","lastname","dob")
| table keyValues
| eval keyValues = json_array_to_mv(keyValues)
| eval firstname=mvindex(keyValues,0),lastname=mvindex(keyValues,1),dob=mvindex(keyValues,2)
| where YOUR_REQUIRED_CONDITION
If you share your SAMPLE events and the exact where conditions with us then we can provide the optimum solution.
Thanks
KV
You don't need to loop through the values, you need to treat them as multivalue fields, and expand/filter appropriately
For example:
index=ndx sourcetype=srctp fieldname.subname{}.value=*
| rename fieldname.subname{}.value as subname
| mvexpand subname
| stats count by subname
| fields - count
Related
I need a Splunk query to fetch the counts of each field used in my dashboard.
Splunk sample data for each search is like this
timestamp="2022-11-07 02:06:38.427"
loglevel="INFO" pid="1"
thread="http-nio-8080-exec-10"
appname="my-test-app"
URI="/testapp/v1/mytest-app/dashboard-service"
RequestPayload="{\"name\":\"test\",\"number\":\"\"}"
What would a search look like to print a table with the number of times the name and number is used to search data (at a time only either number/name data can be given by user).
Expected output in table format with counts for Name and Number
#Hanuman
Can you please try this? You can change regular expression as per your events and match with JSON data.
YOUR_SEARCH | rex field=_raw "RequestPayload=\"(?<data>.*[}])\""
| spath input=data
|table name number
My Sample Search:
| makeresults | eval _raw="*timestamp=\"2022-11-07 02:06:38.427\" loglevel=\"INFO\" pid=\"1\" thread=\"http-nio-8080-exec-10\" appname=\"my-test-app\" URI=\"/testapp/v1/mytest-app/dashboard-service\" RequestPayload=\"{\"name\":\"test\",\"number\":\"1\"}\"*"
| rex field=_raw "RequestPayload=\"(?<data>.*[}])\""
| spath input=data
|table name number
Screen
Thanks
The objective of the query im trying to write is to take a count of raw data from the previous month and add that to a count from a lookup table (.csv)
What I have attempted to do is…
index=*** source=***
| stats count(_raw) as monthCount
| join
[ | inputlookup Log_Count_YTD.csv]
| eval countYTD = toNumber(monthCount) + toNumber(TOTAL_COUNT_YTD)
| table countYTD
This query doesn’t return any value on a table. The TOTAL_COUNT_YTD is the only field from the inputlookup file. Let me know if there is any other information you need to help me out with this one. Thanks!
The stats command transforms the data so it has only 1 field: monthCount. The inputlookup returns only the TOTAL_COUNT_YTD field. The join command works by comparing values of common fields between the main search and the subsearch. Since there are no common fields no events are joined.
There is no need for join in this case. The appendcols command will do, assuming the CSV contains a single field in a single row.
index=*** source=***
| stats count() as monthCount
| appendcols
[ | inputlookup Log_Count_YTD.csv]
| eval countYTD = toNumber(monthCount) + toNumber(TOTAL_COUNT_YTD)
| table countYTD
FWIW, the tonumber function is unnecessary, but doesn't hurt.
The following is my ideal final query to be used in a dashboard.
index=cdn_app httpMessage.host=taxes*
| eval _env=$env_host$
| eval _hostName=case(_env=="http:abc-h123-apps-prod","taxes.sf.com", _env=="http:abc-h123-apps-qa", "taxes-qa.sf.com")
| search httpMessage.host=_hostName
| spath output=status path=httpMessage.status
| eval status=case(like(status, "2%"),"2xx",like(status, "4%"),"4xx",like(status, "5%"),"5xx") | stats count by status
It seems _hostName is not resolving even when I hardcode and values like so.
index=cdn_app httpMessage.host=taxes*
| eval _env="taxes.sf.com"
| eval _hostName=case(_env=="http:abc-h123-apps-prod","taxes.sf.com", _env=="http:abc-h123-apps-qa", "taxes-qa.sf.com")
| search httpMessage.host=_hostName
| stats count by httpMessage.host
I'm sure its with my eval case because this works just fine.
index=cdn_app httpMessage.host=taxes*
| search httpMessage.host="taxes.sf.com"
| stats count by httpMessage.host
Open to any suggestions. Thanks!
Avoid leading underscores in field names as they can be problematic.
The case function is missing a default clause so any value of env not listed will set hostName to null. The stats command will ignore all null values of hostName.
index=cdn_app httpMessage.host=taxes*
| eval env="taxes.sf.com"
| eval hostName=case(env=="http:abc-h123-apps-prod","taxes.sf.com",
env=="http:abc-h123-apps-qa", "taxes-qa.sf.com",
1==1, "???")
| search httpMessage.host=hostName
| stats count by httpMessage.host
<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version="2.0"><ProductInventoryInfo><Product>7781105882846</Product><EmployeeID>12151</EmployeeID><Quantity>28</Quantity><CenterID>167551</CenterID></ProductInventoryInfo></EmployeeInventory>
<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version="2.0"><ProductInventoryInfo><Product>1781305782846</Product><EmployeeID>12152</EmployeeID><Quantity>18</Quantity><CenterID>167552</CenterID></ProductInventoryInfo></EmployeeInventory>
How to write splunk query from above splunk log which will fetch table like this .
Product EmployeeID Quantity CenterID
7781105882846 12151 28 167551
1781305782846 12152 18 167552
It would help to know what you've tried so far and how those attempts failed to meet your needs.
The trick is extracting fields from the XML. You could use a series of rex commands, but spath is simpler.
| makeresults
| eval data="<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version=\"2.0\"><ProductInventoryInfo><Product>7781105882846</Product><EmployeeID>12151</EmployeeID><Quantity>28</Quantity><CenterID>167551</CenterID></ProductInventoryInfo></EmployeeInventory>;<!DOCTYPE EmployeeInventory SYSTEM 'EmployeeInventory.dtd'><EmployeeInventory version=\"2.0\"><ProductInventoryInfo><Product>1781305782846</Product><EmployeeID>12152</EmployeeID><Quantity>18</Quantity><CenterID>167552</CenterID></ProductInventoryInfo></EmployeeInventory>"
| eval data=split(data,";")
| mvexpand data
```The above is just for setting up test data```
```Parse the data```
| spath input=data ```Replace "data" with the name of the field containing the data, perhaps "_raw"```
```Simplify the field names```
| rename EmployeeInventory.ProductInventoryInfo.* as *
```Display the data```
| table Product EmployeeID Quantity CenterID
I have the following table:
postgres=# \d so_rum;
Table "public.so_rum"
Column | Type | Collation | Nullable | Default
-----------+-------------------------+-----------+----------+---------
id | integer | | |
title | character varying(1000) | | |
posts | text | | |
body | tsvector | | |
parent_id | integer | | |
Indexes:
"so_rum_body_idx" rum (body)
I wanted to do phrase search query, so I came up with the below query, for example:
select id from so_rum
where body ## phraseto_tsquery('english','Is it possible to toggle the visibility');
This gives me the results, which only match's the entire text. However, there are documents, where the distance between lexmes are more and the above query doesn't gives me back those data. For example: 'it is something possible to do toggle between the. . . visibility' doesn't get returned. I know I can get it returned with <2> (for example) distance operator by giving in the to_tsquery, manually.
But I wanted to understand, how to do this in my sql statement itself, so that I get the results first with distance of 1 and then 2 and so on (may be till 6-7). Finally append results with the actual count of the search words like the following query:
select count(id) from so_rum
where body ## to_tsquery('english','string & string . . . ')
Is it possible to do in a single query with good performance?
I don't see a canned solution to this. It sounds like you need to use plainto_tsquery to get all the results with all the lexemes, and then implement your own custom ranking function to rank them by distance between the lexemes, and maybe filter out ones with the wrong order.