Assign Nested Value to Variable in Splunk - splunk

I have an event log in Splunk that looks like this:
{
"event": {
"Id":"12345",
"Name": "My Event",
"Inputs": {
"Param1":"some value",
"Param2":"another value"
},
"Result": {
"statusCode":"304"
}
}
}
I need to get the value of the statusCode from the Result to determine what kind of error I received. Currently, I'm using the following Splunk query:
index="myIndex"
sourcetype="*"
| spath=event
| fields
_time
Name
Result.statusCode
| eval _status="tbd"
| eval _code=statusCode
| eval _code=case(statusCode>=200 AND statusCode<300, "OK", statusCode>=300 AND statusCode<400, "Redirected", statusCode>=400 AND statusCode<500, "User Error", statusCode>500, "Server Error")
| rename
Name as RequestName
_code as StatusCode
_status as Status
| table
_time
RequestName
Status
StatusCode
Result.statusCode
The above is a port of the actual query in an effort to isolate the issue. Still, the issue is when I run my query, I can see:
_time
RequestName
Status
Result.statusCode
Oddly, and the part that is confusing me is, I cannot see StatusCode. I need a variable to do additional processing which is why I have the eval _code statement. However, I'm not having any using Result.statusCode as a variable. What am I missing?

Avoid leading underscores in field ("variable") names as they are hidden by default. Some can only be used after assigning their values to another field.
Also, creating a field and then renaming it is unnecessary unless the final field name will contain spaces or special characters.
It looks like something is missing from the query since only the _time and Result.statusCode fields exist, but statusCode is used often. The case function will return null if statusCode does not exist. The Name field also doesn't exist so I don't understand how you can see RequestName.
index="myIndex"
sourcetype="*"
| spath event
| fields
_time
Result.statusCode
| eval Status="tbd", statusCode='Result.statusCode'
| eval StatusCode=case(statusCode>=200 AND statusCode<300, "OK",
statusCode>=300 AND statusCode<400, "Redirected",
statusCode>=400 AND statusCode<500, "User Error",
statusCode>500, "Server Error",
1==1, statusCode)
| rename
Name as RequestName
| table
_time
RequestName
Status
StatusCode
Result.statusCode

Related

How to extract the data present in {} in Splunk Search

If the data present in json format {[]} get extracted, however when data present in {} as shown below doesn't behave same. How fields and values can be extracted from data in {}
_raw data:
{"AlertEntityId": "abc#domai.com", "AlertId": "21-3-1-2-4--12", "AlertType": "System", "Comments": "New alert", "CreationTime": "2022-06-08T16:52:51", "Data": "{\"etype\":\"User\",\"eid\":\"abc#domai.com\",\"op\":\"UserSubmission\",\"tdc\":\"1\",\"suid\":\"abc#domai.com\",\"ut\":\"Regular\",\"ssic\":\"0\",\"tsd\":\"Jeff Nichols <jeff#Nichols.com>\",\"sip\":\"1.2.3.4\",\"srt\":\"1\",\"trc\":\"abc#domai.com\",\"ms\":\"Grok - AI/ML summary, case study, datasheet\",\"lon\":\"UserSubmission\"}"}
When I perform query "| table Data", I get the below result, But how to get values of "eid", "tsd".
{"etype":"User","eid":"abc#domai.com","op":"UserSubmission","tdc":"1","suid":"abc#domai.com","ut":"Regular","ssic":"0","tsd":"Jeff Nichols <jeff#Nichols.com>","sip":"1.2.3.4","srt":"1","trc":"abc#domai.com","ms":"Grok - AI/ML summary, case study, datasheet","lon":"UserSubmission"}
| spath
by default this will parse the _raw field if the data is in the field "Data"
| spath input=Data
After which eid and tsd will be in fields of the same name.
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Spath

how to extract value from splunk and generate line graph

My log messages
.o.s.c.PaymentMethodInstrumentController : Exiting ServiceController.getMyServiceDetails() : elapsedTime(ms):34, xrfRequestId:c3b5878d-8795-49cb-b6a7-51ab02789f46, xCorrelationId:786d68ea-ze46-42b9-966f-124f2eb444f6, xForwardedFor:10.242.79.96
.o.s.c.PaymentMethodInstrumentController : Exiting ServiceController.getMyServiceDetails() : elapsedTime(ms):39, xrfRequestId:c3b2c08d-6c6d-49cb-b6a7-51a89897446, xCorrelationId:78676yt64-ze46-42b9-966f-124f2eb444f6, xForwardedFor:10.242.79.96
I am looking to extract elapsedTime(ms):34 and generate the line graph of these values.
Assuming you already have _time, something like that:
<your search>
| rex "elapsedTime(ms):(?<elapsedTime>\d+),"
| table _time elapsedTime

Splunk extract a value from string which begins with a particular value

Could you help me extract file name in table format.
Here the below field just before file name is always constant. "Put File /test/abc/test/test/test to /test/test/test/test/test/test/test/test/test/test destFolderPath: /test/test/test/test/test/test/test/abc/def/hij"
This is an event from splunk
2021-04-08T01:03:40.155069+00:00 somedata||someotherdata||..|||Put File /test/abc/test/test/test to /test/test/test/test/test/test/test/test/test/test destFolderPath: /test/test/test/test/test/test/test/abc/def/hij/CHARGEBACK_20210407_060334_customer.csv
Result should be in table format: (font / format doesnt matter)
File Name
CHARGEBACK_20210407_060334_customer.csv
Assuming the original event/field ends with the file name, you should use this regular expression:
(?<file_name>[^\/]+)$
This will extract the text between the last "/" and the end of the event/field ("$").
You can test it here: https://regex101.com/r/J6bU3m/1
Now you can use Splunk's rex command to extract fields at search-time:
| makeresults
| eval _raw="2021-04-08T01:03:40.155069+00:00 somedata||someotherdata||..|||Put File /test/abc/test/test/test to /test/test/test/test/test/test/test/test/test/test destFolderPath: /test/test/test/test/test/test/test/abc/def/hij/CHARGEBACK_20210407_060334_customer.csv"
| fields - _time
| rex field=_raw "(?<file_name>[^\/]+)$"
Alternatively, you could also use this regular expression since you mentioned that the file path is always the same:
| rex field=_raw "abc\/def\/hij\/(?<file_name>.+)"

How to reference an eval variable in query

I am trying to access a variable (in this example; sampleFromDate and sampleToDate) from a sub-query. I have defined the variables with syntax eval variableName = value and would like to access with syntax filterName=$variableName$. See the example below where I am trying to access values using earliest=$sampleFromDate$ latest=$sampleToDate$
index=*
earliest=-8d latest=-1d
| eval sampleToDate=now()
| eval sampleFromDate=relative_time(now(), "-1d")
| appendcols [
search (index=*)
earliest=$sampleFromDate$ latest=$sampleToDate$
]
This produces the error:
Invalid value "$sampleFromDate$" for time term 'earliest'
The value of sampleFromDate is in the format seconds since epoch time, e.g.
1612251236.000000
I know I can do earliest=-d latest=now() - but I don't want to do this because I want to reference the variables in several locations and output them at the end.
Why are you trying to eval those time values?
Just do:
index=* earliest=-8d latest=-1d
| <rest of search>
| appendcols [
search (index=*) earliest=-1d
| <rest of appended search>
]
There's no need to explicitly set latest unless you want something other than now()

Accessing values in JSON array

I am following the instruction in the documentation for how to access JSON values in CloudWatch Insights where the recomendation is as follows
JSON arrays are flattened into a list of field names and values. For example, to specify the value of instanceId for the first item in requestParameters.instancesSet, use requestParameters.instancesSet.items.0.instanceId.
ref
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html
I am trying the following and getting nothing in return. The intellisense autofills up to processList.0 but no further
fields processList.0.vss
| sort #timestamp desc
| limit 1
The JSON I am woking with is
"processList": [
{
"vss": xxxxx,
"name": "aurora",
"tgid": xxxx,
"vmlimit": "unlimited",
"parentID": 1,
"memoryUsedPc": 16.01,
"cpuUsedPc": 0.01,
"id": xxxxx,
"rss": xxxxx
},
{
"vss": xxxx,
"name": "aurora",
"tgid": xxxxxx,
"vmlimit": "unlimited",
"parentID": 1,
"memoryUsedPc": 16.01,
"cpuUsedPc": 0.06,
"id": xxxxx,
"rss": xxxxx
}]
Have you tried the following?
fields ##timestamp, #processList.0.vss
| sort ##timestamp desc
| limit 5
It may be a syntax error. If not, please post a couple of records worth of the overall structure, with #timestamp included.
The reference link that you have posted also states the following.
CloudWatch Logs Insights can extract a maximum of 100 log event fields
from a JSON log. For extra fields that are not extracted, you can use
the parse command to parse these fields from the raw unparsed log
event in the message field.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html
For very large JSON messages, Insights intellisense may not be parsing all the fields into named fields. So, the solution is to use parse on the complete JSON string in the field where you expect your data field to be present. In your example and mine it is processList.
I was able to extract the value of specific cpuUsedPc under processList by using a query like the following.
fields #timestamp, cpuUtilization.total, processList
| parse processList /"name":"RDS processes","tgid":.*?,"parentID":.*?,"memoryUsedPc":.*?,"cpuUsedPc":(?<RDSProcessesCPUUsedPc>.*?),/
| sort #timestamp asc
| display #timestamp, cpuUtilization.total, RDSProcessesCPUUsedPc