I have below two JSON events where under "appliedConditionalAccessPolicies", in one event policy1 has results =failure and policy2 has results=notApplied. In the other event the values are reversed.
Now I'm trying to get the event where the policy1 has the status="failure", it gives both the events
index=test
| spath path="appliedConditionalAccessPolicies{}" | search "appliedConditionalAccessPolicies{}.displayName"="policy1" "appliedConditionalAccessPolicies{}.result"="failure"
It looks like Its searching within all the elements in the array.
How can I ensure It searches both the conditions on each element of the array and return the event which has the element satisfying both the conditions.
Events :
appDisplayName: App1
appId: aaaa-1111-111aeff-aad222221111
appliedConditionalAccessPolicies: [
{
displayName: policy1
enforcedGrantControls: [
Block
]
enforcedSessionControls: [
SignInFrequency
ContinuousAccessEvaluation
]
id: f111113-111-400c-a251-2123bbe4233e1
result: failure
}
{ [-]
displayName: policy2
enforcedGrantControls: [ [-]
Block
]
enforcedSessionControls: [ [-]
]
id: sdsds-8c92-45ef-sdsds-c0b2e006d39b
result: notApplied
}
]
appDisplayName: App1
appId: aaaa-1111-111aeff-aad222221111
appliedConditionalAccessPolicies: [
{
displayName: policy1
enforcedGrantControls: [
Block
]
enforcedSessionControls: [
SignInFrequency
ContinuousAccessEvaluation
]
id: f111113-111-400c-a251-2123bbe4233e1
result: notApplied
}
{ [-]
displayName: policy2
enforcedGrantControls: [ [-]
Block
]
enforcedSessionControls: [ [-]
]
id: sdsds-8c92-45ef-sdsds-c0b2e006d39b
result: failure
}
]
The problem is that appliedConditionalAccessPolicies{}.displayName and appliedConditionalAccessPolicies{}.result are multi-value fields so you need to do something that determines if the search matches the same index of both multi-value fields.
Here is a way using mvfind:
And mvfind gives you the multi-value field index so you can compare them, but from my testing mvfind hates field names like appliedConditionalAccessPolicies{}.displayName and appliedConditionalAccessPolicies{}.result so you need to rename them before you can use them with mvfind. This works for me:
| rename "appliedConditionalAccessPolicies{}.displayName" as displayName
| rename "appliedConditionalAccessPolicies{}.result" as result
| where mvfind(displayName,"policy1")=mvfind(result,"failure")
Here is a full example that you can play with:
| makeresults
| eval data="
{\"appDisplayName\":\"App1\",\"appId\":\"aaaa-1111-111aeff-aad222221111\",\"appliedConditionalAccessPolicies\":[{\"displayName\":\"policy1\",\"enforcedGrantControls\":[\"Block1\"],\"enforcedSessionControls\":[\"SignInFrequency\",\"ContinuousAccessEvaluation\"],\"id\":\"f111113-111-400c-a251-2123bbe4233e1\",\"result\":\"failure\"},{\"displayName\":\"policy2\",\"enforcedGrantControls\":[\"Block2\"],\"enforcedSessionControls\":[],\"id\":\"sdsds-8c92-45ef-sdsds-c0b2e006d39b\",\"result\":\"notApplied\"}]}
###
{\"appDisplayName\":\"App2\",\"appId\":\"aaaa-1111-111aeff-aad222221112\",\"appliedConditionalAccessPolicies\":[{\"displayName\":\"policy1\",\"enforcedGrantControls\":[\"Block1\"],\"enforcedSessionControls\":[\"SignInFrequency\",\"ContinuousAccessEvaluation\"],\"id\":\"f111113-111-400c-a251-2123bbe4233e1\",\"result\":\"notApplied\"},{\"displayName\":\"policy2\",\"enforcedGrantControls\":[\"Block2\"],\"enforcedSessionControls\":[],\"id\":\"sdsds-8c92-45ef-sdsds-c0b2e006d39b\",\"result\":\"failure\"}]}
"
| makemv data delim="###"
| mvexpand data
| spath input=data
| fields - data
| rename "appliedConditionalAccessPolicies{}.displayName" as displayName
| rename "appliedConditionalAccessPolicies{}.result" as result
| where mvfind(displayName,"policy1")=mvfind(result,"failure")
Here is a way using mvzip: (thanks to #warren)
You can join the multi-value fields together nad then just search for the string that contains both values. It looks like mvzip also hates field names like appliedConditionalAccessPolicies{}.displayName and appliedConditionalAccessPolicies{}.result so you need to rename them before you can use them with mvzip. This works for me:
| rename "appliedConditionalAccessPolicies{}.displayName" as displayName
| rename "appliedConditionalAccessPolicies{}.result" as result
| where mvzip(displayName,result)="policy1,failure"
Here is a full example that you can play with:
| makeresults
| eval data="
{\"appDisplayName\":\"App1\",\"appId\":\"aaaa-1111-111aeff-aad222221111\",\"appliedConditionalAccessPolicies\":[{\"displayName\":\"policy1\",\"enforcedGrantControls\":[\"Block1\"],\"enforcedSessionControls\":[\"SignInFrequency\",\"ContinuousAccessEvaluation\"],\"id\":\"f111113-111-400c-a251-2123bbe4233e1\",\"result\":\"failure\"},{\"displayName\":\"policy2\",\"enforcedGrantControls\":[\"Block2\"],\"enforcedSessionControls\":[],\"id\":\"sdsds-8c92-45ef-sdsds-c0b2e006d39b\",\"result\":\"notApplied\"}]}
###
{\"appDisplayName\":\"App2\",\"appId\":\"aaaa-1111-111aeff-aad222221112\",\"appliedConditionalAccessPolicies\":[{\"displayName\":\"policy1\",\"enforcedGrantControls\":[\"Block1\"],\"enforcedSessionControls\":[\"SignInFrequency\",\"ContinuousAccessEvaluation\"],\"id\":\"f111113-111-400c-a251-2123bbe4233e1\",\"result\":\"notApplied\"},{\"displayName\":\"policy2\",\"enforcedGrantControls\":[\"Block2\"],\"enforcedSessionControls\":[],\"id\":\"sdsds-8c92-45ef-sdsds-c0b2e006d39b\",\"result\":\"failure\"}]}
"
| makemv data delim="###"
| mvexpand data
| spath input=data
| fields - data
| rename "appliedConditionalAccessPolicies{}.displayName" as displayName
| rename "appliedConditionalAccessPolicies{}.result" as result
| where mvzip(displayName,result)="policy1,failure"
Related
I've a Splunk json event like this:
{
name: "my-name"
tasks: [
{
id: 1,
value: 1
},
{
id: 2,
value: 2
},
]
}
How to write a SPL command to return output with 2 records like:
{name: "my-name", id: 1, value: 1}
{name: "my-name", id: 2, value: 3}
Please help me, thank you guys !
Assuming you actually have ingested a valid JSON object, (as copied here your "json" event is not actually JSON, See the spec at json.org ) but it's possible you copied the prettified version instead of the raw event.
And assuming you're on a new enough version of Splunk to have JSON Functions with eval...
<your search>
| fields name
| spath tasks{} output=task
| mvexpand task
| eval _raw=json_set(task,"name",name)
I simulated an event with this:
| makeresults | eval _raw=json_object("name", "my-name", "tasks", json_array(json_object("id", 1, "value", 1), json_object("id", 2, "value", 2))) | spath
Admittedly you could also use | windbag | head 1 instead of | makeresults for simulation but that gets a bit into the obscure undocumented testing commands that happen to ship with the product.
In Splunk, I'm trying to extract the key value pairs inside that "tags" element of the JSON structure so each one of the become a separate column so I can search through them.
for example :
| spath data | rename data.tags.EmailAddress AS Email
This does not help though and Email field comes as empty.I'm trying to do this for all the tags. Any thoughts/pointers?
{
"timestamp": "2021-10-26T18:23:05.180707Z",
"data": {
"tags": [
{
"key": "Email",
"value": "john.doe#example.com"
},
{
"key": "ProjectCode",
"value": "ABCD"
},
{
"key": "Owner",
"value": "John Doe"
}
]
},
"field1": "random1",
"field2": "random2"
}
I think does what you want:
| spath data.tags{}
| mvexpand data.tags{}
| spath input=data.tags{}
| table key value
| transpose header_field=key
| fields - column
How it works:
| spath data.tags{} takes the json and creates a multi value field that contains each item in the tags array
| mvexpand data.tags{} splits the multi value field into individual events - each one contains one of the items in the tags array
| spath input=data.tags{} takes the json in each event and makes a field for each KVP in that item (key and value in this case)
| table key value limits further commands to these two fields
| transpose header_field=key makes a field for each value of the key field (including one for the field named column)`
| fields - column removes the column field from the output
Here is a fully runnable example:
| makeresults
| eval _raw="
{
\"timestamp\": \"2021-10-26T18:23:05.180707Z\",
\"data\": {
\"tags\": [
{\"key\": \"Email\", \"value\": \"john.doe#example.com\"},
{\"key\": \"ProjectCode\", \"value\": \"ABCD\"},
{\"key\": \"Owner\", \"value\": \"John Doe\"}
]
},
\"field1\": \"random1\",
\"field2\": \"random2\"
}
"
| spath data.tags{}
| mvexpand data.tags{}
| spath input=data.tags{}
| table key value
| transpose header_field=key
It creates this output:
+----------------------+-------------+----------+
| Email | ProjectCode | Owner |
+----------------------+-------------+----------+
| john.doe#example.com | ABCD | John Doe |
+----------------------+-------------+----------+
Trying to pull out text value out of column with json using varchar but get an invalid argument error on snowflake while running on mode. This json has a bit of different structure that what I'm used to seeing.
Have tried these to pull out the text:
changes:comment:new_value::varchar
changes:new_value::varchar
changes:comment::varchar
JSON looks like this:
{
"comment":
{
"new_value": "Hello there. Welcome to our facility.",
"old_value": ""
}
}
Wish to pull out the data in this column so the output reads:
Hello there. Welcome to our facility.
You can't extract fields from VARCHAR. If your string is JSON, you have to convert it to the VARIANT type, e.g. through PARSE_JSON function.
Example below:
create or replace table x(v varchar) as select * from values('{
"comment":
{
"new_value": "Hello there. Welcome to our facility.",
"old_value": ""
}
}');
select v, parse_json(v):comment.new_value::varchar from x;
--------------------------------------------------------------+------------------------------------------+
V | PARSE_JSON(V):COMMENT.NEW_VALUE::VARCHAR |
--------------------------------------------------------------+------------------------------------------+
{ | Hello there. Welcome to our facility. |
"comment": | |
{ | |
"new_value": "Hello there. Welcome to our facility.", | |
"old_value": "" | |
} | |
} | |
--------------------------------------------------------------+------------------------------------------+
I find using * match each response.xyz super powerful for merging object structure testing with content testing. Is there a way to use it with Examples tables and <placeholder>?
I've got something like this, that I want to use an Examples table on:
* match each response.Services ==
"""
{
"ServiceId" : #present,
"Name" : <Name>,
"Description" : #present,
"InActive" : <Inactive>,
}
"""
Examples:
| ClientId | Name | Status | ErrorCode | Inactive |
| 400152 | "Foxtrot" | 200 | 0 | false |
| 400152 | "Waltz" | 200 | 0 | false |
I get
"Services": [
{
"ServiceId": 3,
"Name": "Waltz",
"Description": "Waltzing like Matilda",
"InActive": false,
},
{
"ServiceId": 4,
"Name": "Foxtrot",
"Description": "",
"InActive": false,
},
back as a response.
Obviously, when I'm using multiple lines in Examples:, it results in several tests.
What I'm looking for is to test each object in the array against predefined values, but without knowing what order they'll show up in. And use the ordered approach like tables produce.
Instead of each, try this:
* match response.Services contains
"""
{
"ServiceId" : #present,
"Name" : <Name>,
"Description" : #present,
"InActive" : <Inactive>,
}
"""
EDIT: okay, an alternate option. By the way there are at least 5 different ways I can think of :P
Scenario:
* table data
| ClientId | Name | Status | ErrorCode | Inactive |
| 400152 | "Foxtrot" | 200 | 0 | false |
| 400152 | "Waltz" | 200 | 0 | false |
* def expected = karate.map(data, function(x){ return { ServiceId: '#present', Name: x.Name, Description: '#present', InActive: x.Inactive} })
* match response.Services contains expected
EDIT2: if you can control the whole table:
Scenario:
* table expected
| Name | InActive | ServiceId | Description |
| "Foxtrot" | false | '#present' | '#present' |
| "Waltz" | false | '#present' | '#present' |
* match response.Services contains expected
I receive JSON from API in the following format:
[
{
"scId": "000DD2",
"sensorId": 2,
"metrics": [
{
"s": 5414,
"dateTime": "2018-02-02T13:03:30+01:00"
},
{
"s": 5526,
"dateTime": "2018-02-02T13:04:56+01:00"
},
{
"s": 5631,
"dateTime": "2018-02-02T13:06:22+01:00"
}
}, .... ]
Currently trying to display these metrics on the linear chart with dateTime for the X-axis and "s" for Y.
I use the following search query:
index="main" source="rest://test3" | spath input=metrics{}.s| mvexpand metrics{}.s
| mvexpand metrics{}.dateTime | rename metrics{}.s as s
| rename metrics{}.dateTime as dateTime| table s,dateTime
And I receive the data in the following format which is not applicable for linear chart. The point is - how to correctly parse the JSON to apply date-time from dateTime field in JSON to _time in Splunk.
Query results
#Max Zhylochkin,
Can you please try following search?
index="main" source="rest://test3"
| spath input=metrics{}.s
| mvexpand metrics{}.s
| mvexpand metrics{}.dateTime
| rename metrics{}.s as s
| rename metrics{}.dateTime as dateTime
| table s,dateTime
| eval _time = strptime(dateTime,"%Y-%m-%dT%H:%M:%S.%3N")
Thanks