Hi I have analytics events data moved from firebase to BigQuery and need to create visualization in PowerBI using that BigQuery dataset. I'm able to access the dataset in PowerBI but some fields are in array type I generally use UNNEST while querying in console but how to run the query inside PowerBI. Is there any other option available? Thanks.
Table In BigQuery
What we did until the driver fully supports arrays is to flatten in a view: create a view in bigquery with UNNEST() and query that in PBI instead.
You might need to Transform(parse Json into columns/rows) your specific column in your case event_params
So I have below Json as example for you.
{
"quiz": {
"sport": {
"q1": {
"question": "Which one is correct team name in NBA?",
"options": [
"New York Bulls",
"Los Angeles Kings",
"Golden State Warriros",
"Huston Rocket"
],
"answer": "Huston Rocket"
}
},
"maths": {
"q1": {
"question": "5 + 7 = ?",
"options": [
"10",
"11",
"12",
"13"
],
"answer": "12"
},
"q2": {
"question": "12 - 8 = ?",
"options": [
"1",
"2",
"3",
"4"
],
"answer": "4"
}
}
}
}
I had this json added to my table. currently it has only 1 column
Now I go to Edit queries and go on Transform Tab, there you find Parse, In my case I have Json
When you parse as Json you will have expandable column
Now click on expanding it and sometimes it asks for expand to new row.
Finally you will have such a Table
Related
What am I trying to achieve:
I would like to have a time series chart showing the total number of members in my club at any time. This member count should be calculated by using the field "Eintrittsdatum" (joining-date) and "Austrittsdatum" (leaving-date). I’m thinking of it as a running sum - every filled field with a joining-date means +1 on the member count, every leaving-date entry is a -1.
Data structure
I’m calling the API of webling.ch with a secret key. This is my data structure with sample data per member:
[
{
"type": "member",
"meta": {
"created": "2020-03-02 11:33:00",
"createuser": {
"label": "Joana Doe",
"type": "user"
},
"lastmodified": "2022-12-06 16:32:56",
"lastmodifieduser": {
"label": "Joana Doe",
"type": "user"
}
},
"readonly": true,
"properties": {
"Mitglieder ID": 99,
"Anrede": "Dear",
"Vorname": "Jon",
"Name": "Doe",
"Strasse": "Doeington Street",
"Adresszusatz": null,
"PLZ": "9999",
"Ort": "Doetown",
"E-Mail": "jon.doe#doenet.net",
"Telefon Privat": null,
"Telefon Geschäft": null,
"Mobile": "099 877 54 54",
"Geschlecht": "m",
"Geburtstag": "1966-03-10",
"Mitgliedschaftstyp": "Aktivmitgliedschaft",
"Eintrittsdatum": "2020-03-01",
"Austrittsdatum": null,
"Passfoto": null,
"Wordpress Benutzername": null,
"Wohnhaft im Glarnerland": false,
"Lat": "43.1563379",
"Long": "6.0474622"
},
"parents": [
240
],
"children": {
},
"links": {
"debitor": [
2124,
3056,
3897
],
"attendee": [
2576
]
},
"id": 1815
}
]
Grafana data source
I am using the “JSON API” by Marcus Olsson: GitHub - grafana/grafana-json-datasource: A data source plugin for loading JSON APIs into Grafana.
Grafana v9.3.1 (89b365f8b1) on Linux
My current approach
Queries:
Query C - uses a filter on the source-API to only show entries with "Eintrittsdatum" IS NOT EMPTY
Field 1 (alias "datum") has a JSONata-Query of:
properties.Eintrittsdatum
Field 2 (alias "names") should return the full name and has a query of:
$map($.properties, function($v) {(
($v.Vorname&" "&$v.Name);
)})
Field 3 (alias "value") should return "1" for every entry and has a query of:
$map($.properties, function($v) {(
(1);
)})
Query D - uses a filter on the source-API to only show entries with "Austrittsdatum" IS NOT EMPTY
Field 1 (alias "datum") has a JSONata-Query of:
properties.Austrittsdatum
Field 2 (alias "names") should return the full name and has a query of:
$map($.properties, function($v) {(
($v.Vorname&" "&$v.Name);
)})
Field 3 (alias "value") should return "1" for every entry and has a query of:
$map($.properties, function($v) {(
(1);
)})
Here's a screenshot to clarify things
(https://zigerschlitzmakers.ch/wp-content/uploads/2023/01/ScreenshotGrafana-1.png)
Transformations:
My applied transformations
(https://zigerschlitzmakers.ch/wp-content/uploads/2023/01/ScreenshotGrafana-2.png)
What's working
I can correctly gather the number of members added/subtracted per day.
What's not working
I can't get the graph to display the way i want: I'd like to have a running sum of these numbers instead of the following two graphs.
Time series graph with merged queries
(https://zigerschlitzmakers.ch/wp-content/uploads/2023/01/ScreenshotGrafana-3.png)
Time series graph with unmerged queries
(https://zigerschlitzmakers.ch/wp-content/uploads/2023/01/ScreenshotGrafana-4.png)
I can't get the names to display within the tooltip of the data points (really not THAT necessary).
I have a table with column called tableJson which contains information of the following type:
[
{
"type": "TABLE",
"content": {
"rows":
[{"Date": "2021-09-28","Monthly return": "1.44%"},
{"Date": "2021-11-24", "Yearly return": "0.62%"},
{"Date": "2021-12-03", "Monthly return": "8.57%"},
{},
]
}
}
]
I want to change "Monthly Return" to "Weekly Return" everywhere in the table column where it exists.
Thank you in advance!
I tried different approaches to Parse, read, OPENJSON, CROSS APPLY but could not make it.
I am working on a chatbot project where I have to create a dynamic list option. There is a list of 15 Items (first step). When user selects one, selected item should be removed from the list. After that It will return to first step (list) and will display only 14 items. Where is the issue in step 3?
Step 1: I have created a "item_list" in the memory field of one skill.
[{"item": "product_1", "value": 5},{"item": "product_2", "value": 6},{"item": "product_3", "value": 4}]
Step 2: Choose custom list and enter the following script in the same skill.
{
"type": "list",
"delay": "",
"content": {
"elements": [
{{#eachJoin memory.item_list}}
{
"title": "{{this.item_list.item}}",
"subtitle": "{{this.item_list.value}}",
"imageUrl": "",
"status": "",
"statusState": "<''/none/information/error/success/warning>",
"description": "",
"buttons": [
{
"title": "{{this.item_list.item}}",
"value": "{{this.item_list.item}}",
"type": "postback"
}
]
}{{/eachJoin}}
]
}
}
Step 3: In another skill, set memory field where the user input is stored and set another memory "item_list" to match the user input within the array value to remove it from the item_list. For this, I have entered the following script. But there is some problem with that script. How can I remove the item from the array?
{{#inArray memory.item_list.title memory.user_input.raw}} "{{remove memory.item_list memory.user_input.raw}}" {{else}} "{{memory.lists}}"{{/inArray}}
I have an entity like this:
{
"id": "xxxx",
"attributes": [{
"name": "name-01",
"value": "value-01"
}, {
"name": "name-02",
"value": "value-02"
}
]
}
Our "questions" to data usually: Give me entities with attribute or attribute with particular value;
in SQL it was written like as:
select *
from c
where
and array_contains(c.attributes, { "name": "name-01", "value": "value-01" }, true)
and array_contains(c.attributes, { "name": "name-02", "value": "value-02" }, true)
but I would like to extend a model to allow have suggestion of values in each attribute by transform an entity to:
{
"id": "xxxx",
"attributes": [{
"name": "name_01",
"value": "value-01",
"suggestions": ["a", "b", "c"]
}, {
"name": "name_02",
"value": "value-02",
"suggestions": ["a", "d", "e"]
}
]
}
With that structure I would like to ask: Give me all entities that has specified attribute and value equals to "XYZ" or suggestions array contains "XYZ";
In general scenario if always add value into array of suggestions the ask would be "Give me all entities that has specified attribute and suggestions contains XYZ"
N.B. Also I would like to make queries : Give me all entities that has more ALL specified attributes with constraints per each by suggestions?
Please suggest how to write such queries or rebuild a structure of entities in Cosmos DB;
P.S. We can technically switch from SQL to other protocol to better make such queries;
This should be doable using ARRAY_CONTAINS along with iterating the attributes array.
Give me items with value "value-01" or suggestion "f":
SELECT DISTINCT VALUE(c)
FROM c JOIN attr IN c.attributes
WHERE attr["value"] = "value-01" OR ARRAY_CONTAINS(attr.suggestions, "f")
Give me items with value "value-01" or both suggestions "a" and "f":
SELECT DISTINCT VALUE(c)
FROM c JOIN attr IN c.attributes
WHERE attr["value"] = "value-01" OR
ARRAY_CONTAINS(attr.suggestions, "a") AND ARRAY_CONTAINS(attr.suggestions, "f")
It is possible to store SQL query in JSON array of objects?? Because when i have something like this:
[{
"id": "1",
"query": "SELECT ID FROM table"
},
{
"id": "2",
"query": "SELECT ID FROM table"
},
{
"id": "3",
"query": "SELECT USER FROM table"
}
]
JSON file in VSCode is ok no error it is getting nasty when i want to store complex queries with joins etc.
for example this query even if i format it correctly it will generate error in JSON file about formatting
(just example i not it is not valid)
SELECT user, id, , count(price) as numrev
FROM price
where id = 1 and user = 0
group by user, id, price
that it can't be stored in string
It is bit easy to do, but requires on extra step.
Simply convert/encode you raw SQL queries in base64 text.
Decode the text before you execute the queries in you code.
If the JSON file is created automatically by a program/code
All most all programming languages proved base64 encode / decode functions as part of the core if not download compatible package / library to achieve this automation
var queries = [{
"id": "1",
"query": "U0VMRUNUIElEIEZST00gdGFibGU="
},
{
"id": "2",
"query": "U0VMRUNUIElEIEZST00gdGFibGU="
},
{
"id": "3",
"query": "U0VMRUNUIFVTRVIgRlJPTSB0YWJsZQ=="
},
{
"id": "4",
"query": "U0VMRUNUIHVzZXIsIGlkLCAsIGNvdW50KHByaWNlKSBhcyBudW1yZXYKICBGUk9NIHByaWNlCiAgd2hlcmUgaWQgPSAxIGFuZCB1c2VyID0gMCAKICBncm91cCBieSB1c2VyLCBpZCwgcHJpY2U="
}
];
for (i = 0; i < queries.length; i++) {
console.log("id = " + queries[i].id + ", query = " + atob(queries[i].query));
}
when you parse you JSON array make sure to decode before you execute the SQL queries.
let me know this one helped you.. ☺☻☺
FYI , refer http://www.utilities-online.info/base64/
enter image description here