I receive JSON from API in the following format:
[
{
"scId": "000DD2",
"sensorId": 2,
"metrics": [
{
"s": 5414,
"dateTime": "2018-02-02T13:03:30+01:00"
},
{
"s": 5526,
"dateTime": "2018-02-02T13:04:56+01:00"
},
{
"s": 5631,
"dateTime": "2018-02-02T13:06:22+01:00"
}
}, .... ]
Currently trying to display these metrics on the linear chart with dateTime for the X-axis and "s" for Y.
I use the following search query:
index="main" source="rest://test3" | spath input=metrics{}.s| mvexpand metrics{}.s
| mvexpand metrics{}.dateTime | rename metrics{}.s as s
| rename metrics{}.dateTime as dateTime| table s,dateTime
And I receive the data in the following format which is not applicable for linear chart. The point is - how to correctly parse the JSON to apply date-time from dateTime field in JSON to _time in Splunk.
Query results
#Max Zhylochkin,
Can you please try following search?
index="main" source="rest://test3"
| spath input=metrics{}.s
| mvexpand metrics{}.s
| mvexpand metrics{}.dateTime
| rename metrics{}.s as s
| rename metrics{}.dateTime as dateTime
| table s,dateTime
| eval _time = strptime(dateTime,"%Y-%m-%dT%H:%M:%S.%3N")
Thanks
Related
I've a Splunk json event like this:
{
name: "my-name"
tasks: [
{
id: 1,
value: 1
},
{
id: 2,
value: 2
},
]
}
How to write a SPL command to return output with 2 records like:
{name: "my-name", id: 1, value: 1}
{name: "my-name", id: 2, value: 3}
Please help me, thank you guys !
Assuming you actually have ingested a valid JSON object, (as copied here your "json" event is not actually JSON, See the spec at json.org ) but it's possible you copied the prettified version instead of the raw event.
And assuming you're on a new enough version of Splunk to have JSON Functions with eval...
<your search>
| fields name
| spath tasks{} output=task
| mvexpand task
| eval _raw=json_set(task,"name",name)
I simulated an event with this:
| makeresults | eval _raw=json_object("name", "my-name", "tasks", json_array(json_object("id", 1, "value", 1), json_object("id", 2, "value", 2))) | spath
Admittedly you could also use | windbag | head 1 instead of | makeresults for simulation but that gets a bit into the obscure undocumented testing commands that happen to ship with the product.
I have below two JSON events where under "appliedConditionalAccessPolicies", in one event policy1 has results =failure and policy2 has results=notApplied. In the other event the values are reversed.
Now I'm trying to get the event where the policy1 has the status="failure", it gives both the events
index=test
| spath path="appliedConditionalAccessPolicies{}" | search "appliedConditionalAccessPolicies{}.displayName"="policy1" "appliedConditionalAccessPolicies{}.result"="failure"
It looks like Its searching within all the elements in the array.
How can I ensure It searches both the conditions on each element of the array and return the event which has the element satisfying both the conditions.
Events :
appDisplayName: App1
appId: aaaa-1111-111aeff-aad222221111
appliedConditionalAccessPolicies: [
{
displayName: policy1
enforcedGrantControls: [
Block
]
enforcedSessionControls: [
SignInFrequency
ContinuousAccessEvaluation
]
id: f111113-111-400c-a251-2123bbe4233e1
result: failure
}
{ [-]
displayName: policy2
enforcedGrantControls: [ [-]
Block
]
enforcedSessionControls: [ [-]
]
id: sdsds-8c92-45ef-sdsds-c0b2e006d39b
result: notApplied
}
]
appDisplayName: App1
appId: aaaa-1111-111aeff-aad222221111
appliedConditionalAccessPolicies: [
{
displayName: policy1
enforcedGrantControls: [
Block
]
enforcedSessionControls: [
SignInFrequency
ContinuousAccessEvaluation
]
id: f111113-111-400c-a251-2123bbe4233e1
result: notApplied
}
{ [-]
displayName: policy2
enforcedGrantControls: [ [-]
Block
]
enforcedSessionControls: [ [-]
]
id: sdsds-8c92-45ef-sdsds-c0b2e006d39b
result: failure
}
]
The problem is that appliedConditionalAccessPolicies{}.displayName and appliedConditionalAccessPolicies{}.result are multi-value fields so you need to do something that determines if the search matches the same index of both multi-value fields.
Here is a way using mvfind:
And mvfind gives you the multi-value field index so you can compare them, but from my testing mvfind hates field names like appliedConditionalAccessPolicies{}.displayName and appliedConditionalAccessPolicies{}.result so you need to rename them before you can use them with mvfind. This works for me:
| rename "appliedConditionalAccessPolicies{}.displayName" as displayName
| rename "appliedConditionalAccessPolicies{}.result" as result
| where mvfind(displayName,"policy1")=mvfind(result,"failure")
Here is a full example that you can play with:
| makeresults
| eval data="
{\"appDisplayName\":\"App1\",\"appId\":\"aaaa-1111-111aeff-aad222221111\",\"appliedConditionalAccessPolicies\":[{\"displayName\":\"policy1\",\"enforcedGrantControls\":[\"Block1\"],\"enforcedSessionControls\":[\"SignInFrequency\",\"ContinuousAccessEvaluation\"],\"id\":\"f111113-111-400c-a251-2123bbe4233e1\",\"result\":\"failure\"},{\"displayName\":\"policy2\",\"enforcedGrantControls\":[\"Block2\"],\"enforcedSessionControls\":[],\"id\":\"sdsds-8c92-45ef-sdsds-c0b2e006d39b\",\"result\":\"notApplied\"}]}
###
{\"appDisplayName\":\"App2\",\"appId\":\"aaaa-1111-111aeff-aad222221112\",\"appliedConditionalAccessPolicies\":[{\"displayName\":\"policy1\",\"enforcedGrantControls\":[\"Block1\"],\"enforcedSessionControls\":[\"SignInFrequency\",\"ContinuousAccessEvaluation\"],\"id\":\"f111113-111-400c-a251-2123bbe4233e1\",\"result\":\"notApplied\"},{\"displayName\":\"policy2\",\"enforcedGrantControls\":[\"Block2\"],\"enforcedSessionControls\":[],\"id\":\"sdsds-8c92-45ef-sdsds-c0b2e006d39b\",\"result\":\"failure\"}]}
"
| makemv data delim="###"
| mvexpand data
| spath input=data
| fields - data
| rename "appliedConditionalAccessPolicies{}.displayName" as displayName
| rename "appliedConditionalAccessPolicies{}.result" as result
| where mvfind(displayName,"policy1")=mvfind(result,"failure")
Here is a way using mvzip: (thanks to #warren)
You can join the multi-value fields together nad then just search for the string that contains both values. It looks like mvzip also hates field names like appliedConditionalAccessPolicies{}.displayName and appliedConditionalAccessPolicies{}.result so you need to rename them before you can use them with mvzip. This works for me:
| rename "appliedConditionalAccessPolicies{}.displayName" as displayName
| rename "appliedConditionalAccessPolicies{}.result" as result
| where mvzip(displayName,result)="policy1,failure"
Here is a full example that you can play with:
| makeresults
| eval data="
{\"appDisplayName\":\"App1\",\"appId\":\"aaaa-1111-111aeff-aad222221111\",\"appliedConditionalAccessPolicies\":[{\"displayName\":\"policy1\",\"enforcedGrantControls\":[\"Block1\"],\"enforcedSessionControls\":[\"SignInFrequency\",\"ContinuousAccessEvaluation\"],\"id\":\"f111113-111-400c-a251-2123bbe4233e1\",\"result\":\"failure\"},{\"displayName\":\"policy2\",\"enforcedGrantControls\":[\"Block2\"],\"enforcedSessionControls\":[],\"id\":\"sdsds-8c92-45ef-sdsds-c0b2e006d39b\",\"result\":\"notApplied\"}]}
###
{\"appDisplayName\":\"App2\",\"appId\":\"aaaa-1111-111aeff-aad222221112\",\"appliedConditionalAccessPolicies\":[{\"displayName\":\"policy1\",\"enforcedGrantControls\":[\"Block1\"],\"enforcedSessionControls\":[\"SignInFrequency\",\"ContinuousAccessEvaluation\"],\"id\":\"f111113-111-400c-a251-2123bbe4233e1\",\"result\":\"notApplied\"},{\"displayName\":\"policy2\",\"enforcedGrantControls\":[\"Block2\"],\"enforcedSessionControls\":[],\"id\":\"sdsds-8c92-45ef-sdsds-c0b2e006d39b\",\"result\":\"failure\"}]}
"
| makemv data delim="###"
| mvexpand data
| spath input=data
| fields - data
| rename "appliedConditionalAccessPolicies{}.displayName" as displayName
| rename "appliedConditionalAccessPolicies{}.result" as result
| where mvzip(displayName,result)="policy1,failure"
In Splunk, I'm trying to extract the key value pairs inside that "tags" element of the JSON structure so each one of the become a separate column so I can search through them.
for example :
| spath data | rename data.tags.EmailAddress AS Email
This does not help though and Email field comes as empty.I'm trying to do this for all the tags. Any thoughts/pointers?
{
"timestamp": "2021-10-26T18:23:05.180707Z",
"data": {
"tags": [
{
"key": "Email",
"value": "john.doe#example.com"
},
{
"key": "ProjectCode",
"value": "ABCD"
},
{
"key": "Owner",
"value": "John Doe"
}
]
},
"field1": "random1",
"field2": "random2"
}
I think does what you want:
| spath data.tags{}
| mvexpand data.tags{}
| spath input=data.tags{}
| table key value
| transpose header_field=key
| fields - column
How it works:
| spath data.tags{} takes the json and creates a multi value field that contains each item in the tags array
| mvexpand data.tags{} splits the multi value field into individual events - each one contains one of the items in the tags array
| spath input=data.tags{} takes the json in each event and makes a field for each KVP in that item (key and value in this case)
| table key value limits further commands to these two fields
| transpose header_field=key makes a field for each value of the key field (including one for the field named column)`
| fields - column removes the column field from the output
Here is a fully runnable example:
| makeresults
| eval _raw="
{
\"timestamp\": \"2021-10-26T18:23:05.180707Z\",
\"data\": {
\"tags\": [
{\"key\": \"Email\", \"value\": \"john.doe#example.com\"},
{\"key\": \"ProjectCode\", \"value\": \"ABCD\"},
{\"key\": \"Owner\", \"value\": \"John Doe\"}
]
},
\"field1\": \"random1\",
\"field2\": \"random2\"
}
"
| spath data.tags{}
| mvexpand data.tags{}
| spath input=data.tags{}
| table key value
| transpose header_field=key
It creates this output:
+----------------------+-------------+----------+
| Email | ProjectCode | Owner |
+----------------------+-------------+----------+
| john.doe#example.com | ABCD | John Doe |
+----------------------+-------------+----------+
Trying to pull out text value out of column with json using varchar but get an invalid argument error on snowflake while running on mode. This json has a bit of different structure that what I'm used to seeing.
Have tried these to pull out the text:
changes:comment:new_value::varchar
changes:new_value::varchar
changes:comment::varchar
JSON looks like this:
{
"comment":
{
"new_value": "Hello there. Welcome to our facility.",
"old_value": ""
}
}
Wish to pull out the data in this column so the output reads:
Hello there. Welcome to our facility.
You can't extract fields from VARCHAR. If your string is JSON, you have to convert it to the VARIANT type, e.g. through PARSE_JSON function.
Example below:
create or replace table x(v varchar) as select * from values('{
"comment":
{
"new_value": "Hello there. Welcome to our facility.",
"old_value": ""
}
}');
select v, parse_json(v):comment.new_value::varchar from x;
--------------------------------------------------------------+------------------------------------------+
V | PARSE_JSON(V):COMMENT.NEW_VALUE::VARCHAR |
--------------------------------------------------------------+------------------------------------------+
{ | Hello there. Welcome to our facility. |
"comment": | |
{ | |
"new_value": "Hello there. Welcome to our facility.", | |
"old_value": "" | |
} | |
} | |
--------------------------------------------------------------+------------------------------------------+
I have the following JSON payload stored in a single string column in a BQ table.
{
"customer" : "ABC Ltd",
"custom_fields" : [
{
"name" : "DOB",
"value" : "2000-01-01"
},
{
"name" : "Account_Open_Date",
"value" : "2019-01-01"
}
]
}
I am trying to figure out how I can extract the custom_fields name value pairs as columns?
Something like follows.
| Customer.name | Customer.DOB | Customer.Account_Open_Date |
| ABC Ltd | 2000-01-01 | 2019-01-01 |
You can use json-functions , such as
JSON_EXTRACT(json_string_expr, json_path_string_literal)
In your case will be
SELECT
JSON_EXTRACT(json_text, '$.customer') as Customer.Name,
JSON_EXTRACT(json_text, '$.custom_fields[0].value') as Customer.DOB,
JSON_EXTRACT(json_text, '$.custom_fields[1].value') as Customer.Account_Open_Date