I'm trying to draw custom graph with Google DataStudio community visualization and BigQuery source.
But even if data is exist and valid(check with other basic chart), input data of my drawViz function is empty.
See below Javascript code:
function drawViz(data) {
let rowData = data.tables.DEFAULT;
var metricName = data.fields['barMetric'][0].name;
var dimensionName = data.fields['barDimension'][0].name;
title = metricName + ' by ' + dimensionName + '2';
console.log(rowData , title )
}
Console output:
> {DEFAULT : Array(0)} "my metrics by my dimension"
Is there any restriction using community visualization functionality with bigquery?
Or need any additional setting except in codelab (https://codelabs.developers.google.com/codelabs/community-visualization/#0) ?
** update
manifest.json : https://storage.googleapis.com/vd-qoe-bucket/test/manifest.json
myViz.json : https://storage.googleapis.com/vd-qoe-bucket/test/myViz.json
From your links:
The "data" part of your config appears to be invalid:
"data": [
{
"id": "concepts",
"label": "Concepts",
"elements": [ // setting metric and dimension counts
{
"id": "barDimension",
"label": "Dimension",
"type": "DIMENSION",
"options": {
"min": 1,
"max": 2
}
},
{
"id": "barMetric",
"label": "Metric",
"type": "METRIC",
"options": {
"min": 1,
"max": 2
}
}
]
}
]
Removing the comment // setting dimensions... should work.
Related
This is my api response. Want to extract the value of the Id based on the displayNumber. This display number is a given in the list of values in examples/csv file.
{
"Acc": [
{
"Id": "2b765368696b3441673633325",
"code": "SGD",
"val": 406030.83,
"displayNumber": "8957",
"curval": 406030.83
},
{
"Id": "4e676269685a73787472355776764b50717a4",
"code": "GBP",
"val": 22.68,
"displayNumber": "1881",
"curval": 22.68
},
{
"Id": "526e666d65366e67626244626e6266467",
"code": "SGD",
"val": 38404.44,
"displayNumber": "1004",
"curval": 38404.44
},
],
"combinations": [
{
"displayNumber": "3444",
"Code": "SGD",
"Ids": [
{
"Id": "2b765368696b34416736333254462"
},
{
"Id": "4e676269685a7378747235577"
},
{
"Id": "526e666d65366e6762624d"
}
],
"destId": "3678434b643530456962435272d",
"curval": 3.85
},
{
"displayNumber": "8957",
"code": "SGD",
"Ids": [
{
"Id": "3678434b6435304569624357"
},
{
"Id": "4e676269685a73787472355776764b50717a4"
},
{
"Id": "526e666d65366e67626244626e62664679"
}
],
"destId": "2b765368696b344167363332544",
"curval": 406030.83
},
{
"displayNumber": "1881",
"code": "GBP",
"Ids": [
{
"Id": "3678434b643530456962435275"
},
{
"Id": "2b765368696b3441673"
},
{
"Id": "526e666d65366e67626244626e626"
}
],
"destId": "4e676269685a7378747d",
"curval": 22.68
},
]
}
Examples
|displayNumber|
|8957|
|3498|
|4943|
Below expression works if i give the value
* def tempid = response
* def fromAccount = get[0] tempid.Acc[?(#.displayNumber==8957].Id
I'm not sure how to make this comparison value (i.e. 1881) as a variable which can be read from examples (scenario outline) or a csv file. Went through the documentation, which recommends, karate filters or maps. However, not able to follow how to implement.
You almost got it :-). This is the way you want to solve this
Scenario Outline: Testing SO question for Navneeth
* def tempid = response
* def fromAccount = get[0] tempid.Acc[?(#.displayNumber == <displayNumber>)]
* print fromAccount
Examples:
|displayNumber|
|8957|
|1881|
|3444|
You need to pass the placeholder in examples as -
'<displayNumber>'
I have a data source A and I'd like to create a new data source B containing just the last element of A. What is the best way to do this in Vega?
This is relatively straight forward to do. Although I am slightly confused by your use of "max" in the aggregation since this isn't the last value?
Either way here is my solution for obtaining the last value in a dataset using this series of transforms,
transform: [
{
type: window
ops: [
row_number
]
}
{
type: joinaggregate
fields: [
row_number
]
ops: [
max
]
as: [
max_row_number
]
}
{
type: filter
expr: datum.row_number==datum.max_row_number
}
]
I was able to get this working in the Vega Editor using the following:
{
"$schema": "https://vega.github.io/schema/vega/v5.json",
"data": [
{
"name": "source",
"url": "https://raw.githubusercontent.com/vega/vega/master/docs/data/cars.json",
"transform": [
{
"type": "filter",
"expr": "datum['Horsepower'] != null && datum['Miles_per_Gallon'] != null && datum['Acceleration'] != null"
}
]
},
{
"name": "avg",
"source":"source",
"transform":[
{
"type":"aggregate",
"groupby":["Horsepower"],
"ops": ["average"],
"fields":["Miles_per_Gallon"],
"as":["Avg_Miles_per_Gallon"]
}
]
},
{
"name":"last",
"source": "avg",
"transform": [
{
"type": "aggregate",
"ops": ["max"],
"fields": ["Horsepower"],
"as": ["maxHorsepower"]
},
{
"type": "lookup",
"from": "avg",
"key": "Horsepower",
"fields": ["maxHorsepower"],
"values": ["Horsepower","Avg_Miles_per_Gallon"]
}
]
}
]
}
maxHorsepower
Horsepower
Avg_Miles_per_Gallon
230
230
16
I'd be interested to know if there are better ways, but this worked for me.
I am trying to filter my API response using JSON Path filter using Karate framework to extract a specific value that meets one of the condition using a value from a variable but I am not able to map variable properly, so my filter not working properly. I looked at the documentation and tried multiple ways but couldn't resolve this issue.
Sample response JSON:
"slices": [
{
"id": 7591164138534052,
"duration": {
"value": 1675,
"unit": "MINUTE"
},
"segments": [
{
"id": 1,
"segmentRefId": 23783268357325705
},
{
"id": 2,
"segmentRefId": 7591164138531002
},
{
"id": 3,
"segmentRefId": 7591164138532394
}
]
}
],
"segments": [
{
"id": 23783268357325705,
"departDateTime": "2019-10-05T19:50:00",
"arrivalDateTime": "2019-10-06T14:25:00",
"originAirport": "LAX",
"destinationAirport": "LHR",
"duration": {
"value": 635,
"unit": "MINUTE"
},
"marketingAirline": "BA",
"operatingAirline": "AA",
"flightNumber": "1509",
"equipmentCode": "77W",
"subjectToGovtApproval": false,
"numOfStops": 0,
"stops": []
}
]
}```
I am using the below script where I am using variable 'originRefId':
* def originRefId = response.slices[0].segments[0].segmentRefId
* def origin = karate.jsonPath(response, "$.segments[?(#.id=='originRefId')]")
* print 'the value of Origin is:', origin
Expected results LAX but I am getting an empty array.
17:42:15.119 [main] INFO com.intuit.karate - [print] the value of Origin is: [
]
Here is your solution:
* def originRefId = response.slices[0].segments[0].segmentRefId
* def segment = karate.jsonPath(response, "$.segments[?(#.id==" + originRefId + ")]")[0]
* def origin = segment.originAirport
* match origin == 'LAX'
I'm trying to query the output of a Natural Language Processing (NLP) call in Big Query (BQ) but I'm struggling to get the output in the right format for BQ.
I understand that BQ takes json files (as newline delimited) - but just not sure that (a) the output of NLP is json newline delimited and (b) if my schema is correct.
Here's the json output I'm working with:
{
"entities": [
{
"name": "Rowling",
"type": "PERSON",
"metadata": {
"wikipedia_url": "http://en.wikipedia.org/wiki/J._K._Rowling"
},
"salience": 0.65751493,
"mentions": [
{
"text": {
"content": " J.",
"beginOffset": -1
}
},
{
"text": {
"content": "K. Rowl",
"beginOffset": -1
}
}
]
},
{
"name": "LONDON",
"type": "LOCATION",
"metadata": {
"wikipedia_url": "http://en.wikipedia.org/wiki/London"
},
"salience": 0.14284456,
"mentions": [
{
"text": {
"content": "\ufeffLON",
"beginOffset": -1
}
}
]
},
{
"name": "Harry Potter",
"type": "WORK_OF_ART",
"metadata": {
"wikipedia_url": "http://en.wikipedia.org/wiki/Harry_Potter"
},
"salience": 0.0726779,
"mentions": [
{
"text": {
"content": "th Harry Pot",
"beginOffset": -1
}
},
{
"text": {
"content": "‘Harry Pot",
"beginOffset": -1
}
}
]
},
{
"name": "Deathly Hallows",
"type": "WORK_OF_ART",
"metadata": {
"wikipedia_url": "http://en.wikipedia.org/wiki/Harry_Potter_and_the_Deathly_Hallows"
},
"salience": 0.022565609,
"mentions": [
{
"text": {
"content": "he Deathly Hall",
"beginOffset": -1
}
}
]
}
],
"language": "en"
}
Is there a way to send the output directly to big query via the command line in Google Cloud shell?
Any information would be greatly appreciated!
Thanks
Glad you found my Harry Potter blog post! I'd recommend storing the NL API's JSON response as a string in BigQuery and then using a user-defined function to query it. You should be able to run the following (the table is publicly viewable) to get a count of how often each entity appears in the JSON you posted:
SELECT
COUNT(*) as entity_count, entity
FROM
JS(
(SELECT entities FROM [sara-bigquery:samples.hp_udf]),
entities,
"[{ name: 'entity', type: 'string'}]",
"function(row, emit) {
try {
x = JSON.parse(row.entities);
entities = x['entities'];
entities.forEach(function(data) {
emit({ entity: data.name });
});
} catch (e) {}
}"
)
GROUP BY entity
ORDER BY entity_count DESC
send the output directly to big query via the command line in Google Cloud shell
Look at this page, and search for "bq load"
https://cloud.google.com/bigquery/bq-command-line-tool
Here they have some example about json schema.
Schema to load json data to google big query
I have a ElasticSearch cluster on which I have to perform a sort query based on the size of the object array field 'contents'.
So far I have tried,
{
"size": 10,
"from": 0,
"fields" : ['name'],
"query": {
"match_all": {}
},
"sort" : {
"script" : {
"script" : "doc['contents'].values.length",
"order": "desc"
}
}
}
The above query gives me SearchPhaseExecutionException. The ES query is made from client side using elasticsearch.angular.js.
Any kind of help will be appreciate.
The security has changed for scripts in versions 1.2.x. In ES_HOME/config/scripts create a file called script_score.mvel and add the script:
doc.containsKey('content') == false ? 0 : doc['content'].values.size()
Restart Elasticsearch and change your query to:
{
"size": 10,
"from": 0,
"query": {
"match_all": {}
},
"sort": {
"_script": {
"script": "script_score",
"order": "desc",
"type" : "string"
}
}
}
For more information take a look here:
http://www.elasticsearch.org/blog/scripting-security/