I have a few documents of the below format.
Document1:
{
"_index": "myIndex",
"preferenceCount": [
{
"name": "apple",
"count": 1
},
{
"name": "mango",
"count": 1
},
{
"name": "apple",
"count": 1
}
]
}
Document2:
{
"_index": "myIndex",
"preferenceCount": [
{
"name": "mango",
"count": 1
},
{
"name": "mango",
"count": 1
},
{
"name": "orange",
"count": 1
}
]
}
I want to visualise this data aggregated in such a way that I get the below graph (sorry for not uploading picture)
apple: 2 (sum of count for name = apple across documents in time range)
mango: 3
orange: 1
I tried
sum(preferenceCount.count) groupBy (preferenceCount.name.keyword)
But that sums count across documents and displays below graph
apple: 3
mango: 6
orange: 3
Please let me know how might I achieve this.
Thanks!
I don't know kibana but in Vega-Lite, you can extract data from a property:
{
"data": {
"property": "preferenceCount",
"type": "json",
...
}
}
Related
I have a CSV payload TV Programs & Episodes that I want to Transform (Nest & Condense) to a JSON, with the following conditions:
Merge consecutive Program Lines (that are not followed by an Episode Line), so that it becomes 1 Program with the Start Date of the 1st Instance and the Summation of the Duration.
Episode Lines after a Program Line are Nested under the Program
INPUT
Channel|Name|Start|Duration|Type
ACME|Broke Girls|2018-02-01T00:00:00|600|Program
ACME|Broke Girls|2018-02-01T00:10:00|3000|Program
ACME|S03_8|2018-02-01T00:13:05|120|Episode
ACME|S03_9|2018-02-01T00:29:10|120|Episode
ACME|S04_1|2018-02-01T00:44:12|120|Episode
ACME|Lost In Translation|2018-02-01T02:01:00|1800|Program
ACME|Lost In Translation|2018-02-01T02:30:00|1800|Program
ACME|The Demolition Man|2018-02-01T03:00:00|1800|Program
ACME|The Demolition Man|2018-02-01T03:30:00|1800|Program
ACME|The Demolition Man|2018-02-01T04:00:00|1800|Program
ACME|The Demolition Man|2018-02-01T04:30:00|1800|Program
ACME|Photon|2018-02-01T05:00:00|1800|Program
ACME|Photon|2018-02-01T05:30:00|1800|Program
ACME|Miles & Smiles|2018-02-01T06:00:00|3600|Program
ACME|S015_1|2018-02-01T06:13:53|120|Episode
ACME|S015_2|2018-02-01T06:29:22|120|Episode
ACME|S015_3|2018-02-01T06:46:28|120|Episode
ACME|Ice Age|2018-02-01T07:00:00|300|Program
ACME|Ice Age|2018-02-01T07:05:00|600|Program
ACME|Ice Age|2018-02-01T07:15:00|2700|Program
ACME|S01_4|2018-02-01T07:17:17|120|Episode
ACME|S01_5|2018-02-01T07:32:11|120|Episode
ACME|S01_6|2018-02-01T07:47:20|120|Episode
ACME|My Girl Friday|2018-02-01T08:00:00|3600|Program
ACME|S05_7|2018-02-01T08:17:28|120|Episode
ACME|S05_8|2018-02-01T08:31:59|120|Episode
ACME|S05_9|2018-02-01T08:44:42|120|Episode
ACME|Pirate Bay|2018-02-01T09:00:00|3600|Program
ACME|S01_1|2018-02-01T09:33:12|120|Episode
ACME|S01_2|2018-02-01T09:46:19|120|Episode
ACME|Broke Girls|2018-02-01T10:00:00|1200|Program
ACME|S05_3|2018-02-01T10:13:05|120|Episode
ACME|S05_4|2018-02-01T10:29:10|120|Episode
OUTPUT
{
"programs": [
{
"StartTime": "2018-02-01T00:00:00",
"Duration": 3600,
"Name": "Broke Girls",
"episode": [
{
"name": "S03_8",
"startDateTime": "2018-02-01T00:13:05",
"duration": 120
},
{
"name": "S03_9",
"startDateTime": "2018-02-01T00:29:10",
"duration": 120
},
{
"name": "S04_1",
"startDateTime": "2018-02-01T00:44:12",
"duration": 120
}
]
},
{
"StartTime": "2018-02-01T06:00:00",
"Duration": 3600,
"Name": "Miles & Smiles",
"episode": [
{
"name": "S015_1",
"startDateTime": "2018-02-01T06:13:53",
"duration": 120
},
{
"name": "S015_2",
"startDateTime": "2018-02-01T06:29:22",
"duration": 120
},
{
"name": "S015_3",
"startDateTime": "2018-02-01T06:46:28",
"duration": 120
}
]
},
{
"StartTime": "2018-02-01T07:00:00",
"Duration": 3600,
"Name": "Ice Age",
"episode": [
{
"name": "S01_4",
"startDateTime": "2018-02-01T07:17:17",
"duration": 120
},
{
"name": "S01_5",
"startDateTime": "2018-02-01T07:32:11",
"duration": 120
},
{
"name": "S01_6",
"startDateTime": "2018-02-01T07:47:20",
"duration": 120
}
]
},
{
"StartTime": "2018-02-01T08:00:00",
"Duration": 3600,
"Name": "My Girl Friday",
"episode": [
{
"name": "S05_7",
"startDateTime": "2018-02-01T08:17:28",
"duration": 120
},
{
"name": "S05_8",
"startDateTime": "2018-02-01T08:31:59",
"duration": 120
},
{
"name": "S05_9",
"startDateTime": "2018-02-01T08:44:42",
"duration": 120
}
]
},
{
"StartTime": "2018-02-01T09:00:00",
"Duration": 3600,
"Name": "Pirate Bay",
"episode": [
{
"name": "S01_1",
"startDateTime": "2018-02-01T09:33:12",
"duration": 120
},
{
"name": "S01_2",
"startDateTime": "2018-02-01T09:46:19",
"duration": 120
}
]
},
{
"StartTime": "2018-02-01T10:00:00",
"Duration": 1200,
"Name": "Broke Girls",
"episode": [
{
"name": "S05_3",
"startDateTime": "2018-02-01T10:13:05",
"duration": 120
},
{
"name": "S05_4",
"startDateTime": "2018-02-01T10:29:10",
"duration": 120
}
]
}
]
}
Give this a try, comments are embedded:
%dw 2.0
output application/dw
var data = readUrl("classpath://data.csv","application/csv",{separator:"|"})
var firstProgram = data[0].Name
---
// Identify the programs by adding a field
(data reduce (e,acc={l: firstProgram, c:0, d: []}) -> do {
var next = acc.l != e.Name and e.Type == "Program"
var counter = if (next) acc.c+1 else acc.c
---
{
l: if (next) e.Name else acc.l,
c: counter,
d: acc.d + {(e), pc: counter}
}
}).d
// group by the identifier of individual programs
groupBy $.pc
// Get just the programs throw away the program identifiers
pluck $
// Throw away the programs with no episodes
filter ($.*Type contains "Episode")
// Iterate over the programs
map do {
// sum the program duration
var d = $ dw::core::Arrays::sumBy (e) -> if (e.Type == "Program") e.Duration else 0
// Get the episodes and do a little cleanup
var es = $ map $-"pc" filter ($.Type == "Episode")
---
// Form the desired structure
{
($[0] - "pc" - "Duration"),
Duration: d,
Episode: es
}
}
NOTE1: I stored the contents in a file and read it using readUrl, you need to adjust to accommodate from where you get your data from.
NOTE2: Maybe you need to rethink your inputs, organize them better, if possible.
NOTE3: Studio will show errors (at least Studio 7.5.1 does). They are false positives, the code runs
NOTE4: Lots of steps because of the non-trivial input. Potentialy the code could be optimized but I did spend enough time on it--I 'll let you deal with the optimization or somebody else from the community can help.
I have a cosmosDB collection with below Data in it.
I have to find out the data only for EVENT named ABC and its value using SQL query.
[
{
"ID": "01XXXXX",
"EVENTS": [
{
"Name": "ABC",
"Value": 0
},
{
"Name": "XYZ",
"Value": 4
},
{
"Name": "PQR",
"Value": 5
}
]
},
{
"ID": "02XXXXX",
"EVENTS": [
{
"Name": "ABC",
"Value": 1
},
{
"Name": "XYZ",
"Value": 2
},
{
"Name": "PQR",
"Value": 3
}
]
}
]
I have tried the below code but it is not working since EVENT is an array.
SELECT * FROM c where c.EVENTS.Name = 'ABC'
Is there any way to find filter out the data only with Event Name as ABC using SQL?
Try using join
SELECT c FROM c
join l in c.EVENTS
where l.Name = 'ABC'
I am using Cosmos DB and have a document with the following simplified structure:
{
"id1":"123",
"stuff": [
{
"id2": "stuff",
"a": {
"b": {
"c": {
"d": [
{
"e": [
{
"id3": "things",
"name": "animals",
"classes": [
{
"name": "ostrich",
"meta": 1
},
{
"name": "big ostrich",
"meta": 1
}
]
},
{
"id3": "default",
"name": "other",
"classes": [
{
"name": "green trees",
"meta": 1
},
{
"name": "trees",
"score": 1
}
]
}
]
}
]
}
}
}
}
]
}
My issue is - I have an array of these documents and need to search name to see if it matches my search word. For example I want both big trees and trees to return if a user types in trees.
So currently I push every document into an array and do the following:
For each document
for each stuff
for each a.b.c.d[0].e
for each classes
var splice = name.split(' ')
if (splice.includes(searchWord))
return id1, id2 and id3.
Using cosmosDB I am using SQL with the following code:
client.queryDocuments(
collection,
`SELECT * FROM root r`
).toArray((err, results) => {stuff});
This effectively brings every document in my collection into an array to perform the search manually above as mentioned.
This is going to cause issues when I have 1000s or 1,000,000s of documents in the array and I believe I should be leveraging the search mechanics available within Cosmos itself. Is anyone able to help me to work out what SQL query would be able to perform this type of function?
Having searched everything is it also possible to search the 5 latest documents?
Thanks for any insight in advance!
1.Is anyone able to help me to work out what SQL query would be able to
perform this type of function?
According to your sample and description, I suggest you using ARRAY_CONTAINS in cosmos db sql. Please refer to my sample:
sample documents:
[
{
"id1": "123",
"stuff": [
{
"id2": "stuff",
"a": {
"b": {
"c": {
"d": [
{
"e": [
{
"id3": "things",
"name": "animals",
"classes": [
{
"name": "ostrich",
"meta": 1
},
{
"name": "big ostrich",
"meta": 1
}
]
},
{
"id3": "default",
"name": "other",
"classes": [
{
"name": "green trees",
"meta": 1
},
{
"name": "trees",
"score": 1
}
]
}
]
}
]
}
}
}
}
]
},
{
"id1": "456",
"stuff": [
{
"id2": "stuff2",
"a": {
"b": {
"c": {
"d": [
{
"e": [
{
"id3": "things2",
"name": "animals",
"classes": [
{
"name": "ostrich",
"meta": 1
},
{
"name": "trees",
"meta": 1
}
]
},
{
"id3": "default2",
"name": "other",
"classes": [
{
"name": "green trees",
"meta": 1
},
{
"name": "trees",
"score": 1
}
]
}
]
}
]
}
}
}
}
]
},
{
"id1": "789",
"stuff": [
{
"id2": "stuff3",
"a": {
"b": {
"c": {
"d": [
{
"e": [
{
"id3": "things3",
"name": "animals",
"classes": [
{
"name": "ostrich",
"meta": 1
},
{
"name": "big",
"meta": 1
}
]
},
{
"id3": "default3",
"name": "other",
"classes": [
{
"name": "big trees",
"meta": 1
}
]
}
]
}
]
}
}
}
}
]
}
]
query :
SELECT distinct c.id1,stuff.id2,e.id3 FROM c
join stuff in c.stuff
join d in stuff.a.b.c.d
join e in d.e
where ARRAY_CONTAINS(e.classes,{name:"trees"},true)
or ARRAY_CONTAINS(e.classes,{name:"big trees"},true)
output:
2.Having searched everything is it also possible to search the 5 latest
documents?
Per my research, features like LIMIT is not supported in cosmos so far. However , TOP is supported by cosmos db. So if you could add sort field(such as date or id), then you could use sql:
select top 5 from c order by c.sort desc
I have following documents in azure cosmos db collection.
// Document 1
{
"c": {
"firstName": "Robert"
}
"elements" : [
{
"a": "x2",
"b": {
"name": "yadda2",
"id": 1
}
}
]
}
// Document 2
{
"c": {
"firstName": "Steve"
}
"elements" : [
{
"a": "x5",
"b": {
"name": "yadda2",
"id": 4
}
},
{
"a": "x3",
"b": {
"name": "yadda8",
"id": 5
}
},
]
}
// Document 3
{
"c": {
"firstName": "Johnson"
}
"elements" : [
{
"a": "x4",
"b": {
"name": "yadda28",
"id": 25
}
},
{
"a": "x5",
"b": {
"name": "yadda30",
"id": 37
}
},
]
}
I need to write a query that returns all documents which have "b" object whose name is "yadda2" (i.e. /elements/*/b/name=yadda2). In other words, this query should return document 1 and 2 but NOT 3.
I tried following but it did not work:
SELECT * FROM x where ARRAY_CONTAINS(x.elements, {b: { name: "yadda2"}})
What am I doing wrong?
Just modify your sql to :
SELECT * FROM x where ARRAY_CONTAINS(x.elements, {b: { name: "yadda2"}},true)
Result:
Based on the official doc , the boolean expression could specify if the match is full or partial.
Hope it helps you.
I have json data similar to this:
{
"Sections": [
{
"Categories": [
{
"Name": "Book",
"Id": 1,
"Options": [
{
"Name": "AAAA",
"OptionId": 111
},
"Selected": 0
},
{
"Name": "Car",
"Id": 2,
"Options": [
{
"Name": "BBB",
"OptionId": 222
},
"Selected": 0
},
],
"SectionName": "Main"
},
... more sections like the one above
]
}
Given this data, I want to find a category inside a section based on its (Category) Id, and set its selected option, I tried this, but couldn't get it to work....Note Category Id will be unique in the whole data set.
_.find(model.Sections, { Categories: [ { Id: catId } ]});
According to your data model, it looks like you're trying to find an element that is inside a matrix: Sections can have multiple Categories and a Category can have multiple types (car, book...).
I'm afraid there isn't a function in lodash that allows a deep find, you'll have to implement it the 'traditional' way (a couple of fors).
I provide this solution that is a bit more 'functional flavoured' than the traditional nested fors. It also takes advantage of the fact that when you explicitly return false inside a forEach, the loop finishes. Thus, once an element with the provided id is found, the loop is ended and the element returned (if it's not found, undefined is returned instead).
Hope it helps.
const findCategoryById = (sections, id) => {
var category;
_.forEach(sections, (section) => {
category = _.find(section.Categories, ['Id', id]);
return _.isUndefined(category);
});
return category;
};
const ex = {
"Sections": [{
"Categories": [{
"Name": "Book",
"Id": 1,
"Options": [{
"Name": "AAAA",
"OptionId": 111
}],
"Selected": 0
},
{
"Name": "Car",
"Id": 2,
"Options": [{
"Name": "BBB",
"OptionId": 222
}],
"Selected": 0
}
],
"SectionName": "Main"
}]
};
console.log(findCategoryById(ex.Sections, 2));
<script src="https://cdn.jsdelivr.net/npm/lodash#4.17.5/lodash.min.js"></script>