List comprehension equivalent in Kusto - kql

I recently started to use the Azure Resource Graph Explorer to obtain resource information. KQL is a new thing I'm figuring out along the way, and one of the problems I need help with is a means to manipulate an array of dictionaries into just an array of string values.
As an example:
Consider the following data
{
"customerId": "201",
"orders": [
{
"dept": "/packaging/fruits"
},
{
"dept": "/packaging/vegetables"
}
]
}
With the following query:
Customers
| where customerId == 201
| project customerId, orders
The result would be as follows:
My question is, how can I modify the query to produce the following result:
Tried to go through the KQL documentation, but can't seem to find the correct method to achieve the above. Any help would be much appreciated!

in Kusto, you could use mv-apply:
datatable(customerId:int, orders:dynamic)
[
201, dynamic([
{
"dept": "/packaging/fruits"
},
{
"dept": "/packaging/vegetables"
}
]),
201,
dynamic([
{
"dept": "/packaging2/fruits2"
},
{
"dept": "/packaging2/vegetables2"
}
])
]
| where customerId == 201
| mv-apply orders on (
summarize orders = make_list(orders.dept)
)
customerId
orders
201
[ "/packaging/fruits", "/packaging/vegetables"]
201
[ "/packaging2/fruits2", "/packaging2/vegetables2"]
In ARG, mv-apply isn't supported, so you can use mv-expand:
datatable(customerId:int, orders:dynamic)
[
201, dynamic([
{
"dept": "/packaging/fruits"
},
{
"dept": "/packaging/vegetables"
}
]),
201,
dynamic([
{
"dept": "/packaging2/fruits2"
},
{
"dept": "/packaging2/vegetables2"
}
])
]
| where customerId == 201
| extend rn = rand()
| mv-expand orders
| summarize orders = make_list(orders.dept) by rn, customerId
| project-away rn
customerId
orders
201
[ "/packaging/fruits", "/packaging/vegetables"]
201
[ "/packaging2/fruits2", "/packaging2/vegetables2"]

Related

How to parse this JSON file in Snowflake?

So I have a column in a Snowflake table that stores JSON data but the column is of a varchar data type.
The JSON looks like this:
{
"FLAGS": [],
"BANNERS": {},
"TOOLS": {
"game.appConfig": {
"type": [
"small",
"normal",
"huge"
],
"flow": [
"control",
"noncontrol"
]
}
},
"PLATFORM": {}
}
I want to filter only the data inside TOOLS and want to get the following result:
TOOLS_ID
TOOLS
game.appConfig
type
game.appConfig
flow
How can I achieve this?
I assumed that the TOOLs can have more than one tool ID, so I wrote this query:
with mydata as ( select
'{
"FLAGS": [],
"BANNERS": {},
"TOOLS": {
"game.appConfig": {
"type": [
"small",
"normal",
"huge"
],
"flow": [
"control",
"noncontrol"
]
}
},
"PLATFORM": {}
}' as v1 )
select main.KEY TOOLS_ID, sub.KEY TOOLS
from mydata,
lateral flatten ( parse_json(v1):"TOOLS" ) main,
lateral flatten ( main.VALUE ) sub;
+----------------+-------+
| TOOLS_ID | TOOLS |
+----------------+-------+
| game.appConfig | flow |
| game.appConfig | type |
+----------------+-------+
Assuming the column name is C1 and table name T1:
select a.t:"TOOLS":"game.appConfig"::string from (select
parse_json(to_variant(C1))t from T1) a

How to add Multiple JSON array Element depends on Example Data Set

During my Karate scenario automation got the below problem statement. Could anyone please help me on this?
My JSON file will contain,
{
"schema_id": 25,
"records": [
{
"value": {
"NUM_CHARGE_ID": "#(numChargeID)",
"DTE_START": "2017-05-26",
"NUM_DAY": "#(numDay)"
}
}
]
}
From my Karate feature file I want to pass data from Example and depends on the data set it should create multiple array element in the json. Like,
Examples:
| numChargeID | numDay|
|C10,C21,C15,C18|1,2,3,4|
{
"schema_id": 25,
"records": [
{
"value": {
"NUM_CHARGE_ID": "C10",
"DTE_START": "2017-05-26",
"NUM_DAY": "1"
},
{
"NUM_CHARGE_ID": "C21",
"DTE_START": "2017-05-26",
"NUM_DAY": "2"
},
{
"NUM_CHARGE_ID": "C15",
"DTE_START": "2017-05-26",
"NUM_DAY": "3"
},
{
"NUM_CHARGE_ID": "C18",
"DTE_START": "2017-05-26",
"NUM_DAY": "4"
}
}
]
}
Is there any way in Karate that i can handle this type of situation?
There are plenty of ways to solve this and maybe you shouldn't try to force everything into Examples:. For instance:
* def chargeIds = ['C1', 'C2']
* def dayNums = [1, 2]
* def fun = function(x, i){ return { NUM_CHARGE_ID: x, NUM_DAY: dayNums[i] } }
* def records = karate.map(chargeIds, fun)
* print records
Which gives you:
[
{
"NUM_CHARGE_ID": "C1",
"NUM_DAY": 1
},
{
"NUM_CHARGE_ID": "C2",
"NUM_DAY": 2
}
]
What I would recommend is something like this:
* table records
| NUM_CHARGE_ID | NUM_DAY |
| 'C1' | 1 |
| 'C2' | 2 |
* print records
Which gives you the exact same thing.
Also look at the possible use of set: https://github.com/intuit/karate#set-multiple

How to unpack Array to Rows in Snowflake?

I have a table that looks like the following in Snowflake:
ID | CODES
2 | [ { "list": [ { "item": "CODE1" }, { "item": "CODE2" } ] } ]
And I want to make it into:
ID | CODES
2 | 'CODE1'
2 | 'CODE2'
So far I've tried
SELECT ID,CODES[0]:list
FROM MY_TABLE
But that only gets me as far as:
ID | CODES
2 | [ { "item": "CODE1" }, { "item": "CODE2" } ]
How can I break out every 'item' element from every index of this list into its own row with each CODE as a string?
Update: Here is the answer I got working at the same time as the answer below, looks like we both used FLATTEN:
SELECT ID,f.value:item
FROM MY_TABLE,
lateral flatten(input => MY_TABLE.CODES[0]:list) f
So as you note you have hard coded your access into the codes, via codes[0] which gives you the first item from that array, if you use FLATTEN you can access all of the objects of the first array.
WITH my_table(id,codes) AS (
SELECT 2, parse_json('[ { "list": [ { "item": "CODE1" }, { "item": "CODE2" } ] } ]')
)
SELECT ID, c.*
FROM my_table,
table(flatten(codes)) c;
gives:
2 1 [0] 0 { "list": [ { "item": "CODE1" }, { "item": "CODE2" }]} [ { "list": [{"item": "CODE1"}, { "item": "CODE2" }]}]
so now you want to loop across the items in list, so we use another FLATTEN on that:
WITH my_table(id,codes) AS (
SELECT 2, parse_json('[ { "list": [ { "item": "CODE1" }, { "item": "CODE2" } ] } ]')
)
SELECT ID, c.value, l.value
FROM my_table,
table(flatten(codes)) c,
table(flatten(c.value:list)) l;
gives:
2 {"list":[{"item": "CODE1"},{"item":"CODE2"}]} {"item":"CODE1"}
2 {"list":[{"item": "CODE1"},{"item":"CODE2"}]} {"item":"CODE2"}
so you can pull apart that l.value how you need to access the parts you need.

Querying the Cosmos db collection of array data

I am new to cosmos i have a collection i am trying to query the sample is below.
as the data is nested due to the nature of it. we must be able to query the sub sections of the documents also.
I am getting stuck in trying to retrieve the data of the itemCode in the below collection.
Any info would be a great help. thanks in advance.
SELECT * FROM c
where c.customerSites.pricingGroup.itemCode ="2400953"
This query returns no results.
Data collection sample
{
"customerSites": [
{
"customerCode": "196729",
"businessUnitName": "XXXXX SOUTHERN LTD",
"siteCode": "96271",
"addressCode": "_MAINXXXX",
"pricingGroup": [
{
"itemCode": 2400953,
"branches": [
8999,
3001,
3002,
3003,
3004
],
"rates": [
{
"branchCodes": [
8999,
3001,
3002,
3003
],
"discountPercentage": null,
"derivedRateId": 77735584,
"derivedRateClassification": "customrate",
"branchLevel": 109,
"derivedRateType": "P",
"durationRates": [
{
"durationType": 1,
"rate": 125
}
]
}
]
}
]
}
],
"id": "196729",
"dataType": "AccountCustomer",
"_ts": 1547161022
}
Simply do:
SELECT * FROM c where c.customerSites[0].pricingGroup[0].itemCode = 2400953

RavenDb facet search in string array field with wildcard

Is it possible to have a RavenDb faceted search, in a string[] field, where I would want to show facets (counts) for only values starting with a particular string, rather a range?
I'll try to explain myself better to with a simple example, imagine having an index with the below entries
ID | Values
-------------------------
1 | CatPersian, CatNormal, DogLabrador
2 | CatPersian, Camel, DogPoodle
3 | CatNormal, CatBengali, DogNormal
4 | DogNormal
I would perform a query on the above documents, and the Facet search would include a range of 'Cat*', on the 'Values' field. Is this possible? Then, I would get a result based on just the different values for cats, like:
CatPersian [2]
CatNormal [2]
CatBengali [1]
Yes, you can do that. Index the array, and then just use facets normally.
Let's see the full example. You have the following documents:
{
"Name": "John",
"FavoriteAnimals": [
"Cats",
"Dogs",
"Snails"
],
"#metadata": {
"#collection": "Kids"
}
}
{
"Name": "Jane",
"FavoriteAnimals": [
"Cats",
"Rabits"
],
"#metadata": {
"#collection": "Kids"
}
}
Now, you create the following index:
from k in docs.Kids
from animal in k.FavoriteAnimals
select new { Animal = animal }
And run this query:
from index 'YourIndex'
where startsWith(Animal , 'ca')
select facet('Animal')
And the result will be:
{
"Name": "Animal",
"Values": [
{
"Count": 2,
"Range": "cats"
}
]
}
Alternatively, you can use this index:
from k in docs.Kids
select new { k.FavoriteAnimals }
And run this query:
from index 'YourIndex'
where startsWith(FavoriteAnimals , 'ca')
select facet('FavoriteAnimals')
The difference here is that you'll get all matches for the documents that have a match.
So in this case
{
"Name": "Animal",
"Values": [
{
"Count": 2,
"Range": "cats"
},
{
"Count": 1,
"Range": "dogs"// also, snails, rabbits
}
]
}