How to get selected json objects from a nested json in postgres sql - sql

I have the data stored in a table named 'userfields' the below format in one of the columns named 'params' in Postgres SQL. I want to select two json objects and the output should be in the same format
{
"default": {
"model": "core.location",
"pk": 119
},
"field1": {
"name": "Vista Community Clinic- The Gary Center, S. Harbour Blvd",
"full_address": "201 S. Harbor Boulevard, \nLa Habra, CA 90631"
}
"fields2": {
"name": "xyz- The Gary Center, S. Harbour Blvd",
"full_address": "abc, \nLa Habra, CA 90631"
}
}
I have tried and achieved for one json object which will return for example default
The query I have used is
select json_extract_path(params::json,'default') as selectedparams from userfields.
I am not able to select two json objects like default and feild1
The output I have got is
{
"model": "core.location",
"pk": 119
}
The output I want is I want to get default and field1 as the nested JSON by using PostgreSQL13 query like below
Desired Result:
{
"default": {
"model": "core.location",
"pk": 119
},
"field1": {
"name": "Vista Community Clinic- The Gary Center, S. Harbour Blvd",
"full_address": "201 S. Harbor Boulevard, \nLa Habra, CA 90631"
}
}

Related

Add computed field to Query in Grafana using JSON API als data source

What am I trying to achieve:
I would like to have a time series chart showing the total number of members in my club at any time. This member count should be calculated by using the field "Eintrittsdatum" (joining-date) and "Austrittsdatum" (leaving-date). I’m thinking of it as a running sum - every filled field with a joining-date means +1 on the member count, every leaving-date entry is a -1.
Data structure
I’m calling the API of webling.ch with a secret key. This is my data structure with sample data per member:
[
{
"type": "member",
"meta": {
"created": "2020-03-02 11:33:00",
"createuser": {
"label": "Joana Doe",
"type": "user"
},
"lastmodified": "2022-12-06 16:32:56",
"lastmodifieduser": {
"label": "Joana Doe",
"type": "user"
}
},
"readonly": true,
"properties": {
"Mitglieder ID": 99,
"Anrede": "Dear",
"Vorname": "Jon",
"Name": "Doe",
"Strasse": "Doeington Street",
"Adresszusatz": null,
"PLZ": "9999",
"Ort": "Doetown",
"E-Mail": "jon.doe#doenet.net",
"Telefon Privat": null,
"Telefon Geschäft": null,
"Mobile": "099 877 54 54",
"Geschlecht": "m",
"Geburtstag": "1966-03-10",
"Mitgliedschaftstyp": "Aktivmitgliedschaft",
"Eintrittsdatum": "2020-03-01",
"Austrittsdatum": null,
"Passfoto": null,
"Wordpress Benutzername": null,
"Wohnhaft im Glarnerland": false,
"Lat": "43.1563379",
"Long": "6.0474622"
},
"parents": [
240
],
"children": {
},
"links": {
"debitor": [
2124,
3056,
3897
],
"attendee": [
2576
]
},
"id": 1815
}
]
Grafana data source
I am using the “JSON API” by Marcus Olsson: GitHub - grafana/grafana-json-datasource: A data source plugin for loading JSON APIs into Grafana.
Grafana v9.3.1 (89b365f8b1) on Linux
My current approach
Queries:
Query C - uses a filter on the source-API to only show entries with "Eintrittsdatum" IS NOT EMPTY
Field 1 (alias "datum") has a JSONata-Query of:
properties.Eintrittsdatum
Field 2 (alias "names") should return the full name and has a query of:
$map($.properties, function($v) {(
($v.Vorname&" "&$v.Name);
)})
Field 3 (alias "value") should return "1" for every entry and has a query of:
$map($.properties, function($v) {(
(1);
)})
Query D - uses a filter on the source-API to only show entries with "Austrittsdatum" IS NOT EMPTY
Field 1 (alias "datum") has a JSONata-Query of:
properties.Austrittsdatum
Field 2 (alias "names") should return the full name and has a query of:
$map($.properties, function($v) {(
($v.Vorname&" "&$v.Name);
)})
Field 3 (alias "value") should return "1" for every entry and has a query of:
$map($.properties, function($v) {(
(1);
)})
Here's a screenshot to clarify things
(https://zigerschlitzmakers.ch/wp-content/uploads/2023/01/ScreenshotGrafana-1.png)
Transformations:
My applied transformations
(https://zigerschlitzmakers.ch/wp-content/uploads/2023/01/ScreenshotGrafana-2.png)
What's working
I can correctly gather the number of members added/subtracted per day.
What's not working
I can't get the graph to display the way i want: I'd like to have a running sum of these numbers instead of the following two graphs.
Time series graph with merged queries
(https://zigerschlitzmakers.ch/wp-content/uploads/2023/01/ScreenshotGrafana-3.png)
Time series graph with unmerged queries
(https://zigerschlitzmakers.ch/wp-content/uploads/2023/01/ScreenshotGrafana-4.png)
I can't get the names to display within the tooltip of the data points (really not THAT necessary).

Why does this AWS API Gateway Mapping Template not output the correct variable?

I am using AWS API Gateway. I have a response from my backend and I am trying to map the response into a different output using a Mapping Template.
I am trying to return each of the hits.hits._source.display elements. But the mapping template returns the entire object with the literal "_source.display" string added to the end.
The original unmodified server response is like -
{
"took":7,
"timed_out":false,
"_shards":{
"total":1,
"successful":1,
"skipped":0,
"failed":0
},
"hits":{
"total":{
"value":10000,
"relation":"gte"
},
"max_score":1.0,
"hits":[
{
"_index":"address",
"_type":"_doc",
"_id":"GAVIC411064535",
"_score":1.0,
"_source":{
"id":"GAVIC411064535",
"display":"1/28 George Street, Casterton VIC 3311",
"location":{
"lat":-37.59205672,
"lon":141.38825665
},
"component":{
"buildingName":"",
"number":"1/28",
"street":"George Street",
"locality":"Casterton",
"state":"VIC",
"postcode":"3311"
}
}
},
{
"_index":"address",
"_type":"_doc",
"_id":"GAVIC411066597",
"_score":1.0,
"_source":{
"id":"GAVIC411066597",
"display":"52 St Georges Road, Corio VIC 3214",
"location":{
"lat":-37.59205672,
"lon":141.38825665
},
"component":{
"buildingName":"",
"number":"52",
"street":"St Georges Road",
"locality":"Corio ",
"state":"VIC",
"postcode":"3214"
}
}
},
My Mapping Template so far looks like this -
#set($inputRoot = $input.path('$'))
{
#foreach($elem in $inputRoot.hits.hits)
{
"address": $elem._source.display,
}#if($foreach.hasNext),#end
#end
}
The resulting output looks like this. You can see that its returing the entire $elem rather than returning the properties I've asked for. That is ._source.display
It has also literally printed ._source.display to the end.
{
"address": {_index=address, _type=_doc, _id=GAVIC411064535, _score=1.0, _source={id=GAVIC411064535, display=1/28 George Street, Casterton VIC 3311, location={lat=-37.59205672, lon=141.38825665}, component={buildingName=, number=1/28, street=George Street, locality=Casterton, state=VIC, postcode=3311}}}._source.display,
},
{
"address": {_index=address, _type=_doc, _id=GAVIC411066597, _score=1.0, _source={id=GAVIC411066597, display=52 St Georges Road, Corio VIC 3214, location={lat=-37.59205672, lon=141.38825665}, component={buildingName=, number=52, street=Georges Road, locality=Corio , state=VIC, postcode=3214}}}._source.display,
},
I have truncated the server response and mapping template output for brevity.
The desired mapping template output is -
{
"address" : "28 George Street, Casterton VIC 3311",
"address" : "52 St Georges Road, Corio VIC 3214"
}
This is a bug in version 1.7 which has been fixed in 2.x versions (in July 2016... AWS really ought to upgrade its libraries sometimes).
You can work around this bug like this:
#foreach($elem in $inputRoot.hits.hits)
{
"address": "$elem.get('_source').display"
}#if($foreach.hasNext),#end
#end

Issue with creating a dynamic request body from external json file in karate

I am trying to create my request body dynamically from an external json file.
I want to update few values and keep the remaining ones same as received from the json.
The idea here is to keep one maintainable json file and manipulate it at run time to execute various scenarios.
Here's my feature file:
* def myJson = read('testFile.json')
* def requestBody = { "product": "#(myJson.product)", "properties": { "make": "#(brand)", "color": "#(myJson.color)" }
When request requestBody
And method post
Then status 200
Examples:
| brand |
| honda |
Contents of testFile.json are -
{
"product": "car",
"properties": {
"make": "brand",
"color": "red"
}
}
The problem is that whenever there is nested json object, those fields won't keep the value from json. If the value is passed from the feature file as an example, then it gets evaluated correctly. Here's how the request body gets passed in the service call-
{
"product": "car",
"properties": {
"make": "honda",
"color": null
}
}
I need the color key's value to be taken from myJson i.e. red but it get evaluated as null.
Shouldn't it be:
"color": "#(myJson.properties.color)"

How to extract the field from JSON object with QueryRecord

I have been struggling with this problem for a long time. I need to create a new JSON flowfile using QueryRecord by taking an array (field ref) from input JSON field refs and skip the object field as shown in example below:
Input JSON flowfile
{
"name": "name1",
"desc": "full1",
"refs": {
"ref": [
{
"source": "source1",
"url": "url1"
},
{
"source": "source2",
"url": "url2"
}
]
}
}
QueryRecord configuration
JSONTreeReader setup as Infer Schema and JSONRecordSetWriter
select name, description, (array[rpath(refs, '//ref[*]')]) as sources from flowfile
Output JSON (need)
{
"name": "name1",
"desc": "full1",
"references": [
{
"source": "source1",
"url": "url1"
},
{
"source": "source2",
"url": "url2"
}
]
}
But got error:
QueryRecord Failed to write MapRecord[{references=[Ljava.lang.Object;#27fd935f, description=full1, name=name1}] with schema ["name" : "STRING", "description" : "STRING", "references" : "ARRAY[STRING]"] as a JSON Object due to java.lang.ClassCastException: null
Try the following approach, in your case it shoud work:
1) Read your JSON field fully (I imitated it with GenerateFlowFile processor with your example)
2) Add EvaluateJsonPath processor which will put 2 header fileds (name, desc) into the attributes:
3) Add SplitJson processor which will split your JSON byt refs/ref/ groups (split by "$.refs.ref"):
4) Add ReplaceText processor which will add you header fields (name, desc) to the split lines (replace "[{]" value with "{"name":"${json.name}","desc":"${json.desc}","):
5) It`s done:
Full process in my demo case:
Hope this helps.
Solution!: use JoltTransformJSON to transform JSON by Jolt specification. About this specification.

pass stream with array to multiple rows in SQL by Stream Analytics

I have a stream from IoT Hub like:
{
"topic": "saveData",
"deviceId": "testDevice",
"data": [
{
"timestamp": "2018-04-06T11:46:11.842305",
"device": "baiTest",
"variable": "Status01_Test",
"name": "m_01_test",
"value": 365
},
{
"timestamp": "2018-04-06T11:46:11.842306",
"device": "hmuTest",
"variable": "Status02_Test",
"name": "m_02_test",
"value": 817
},
{
"timestamp": "2018-04-06T11:46:11.842307",
"device": "vwzTest",
"variable": "Status03_Test",
"name": "m_03_test",
"value": 247
}
]
}
I want to pass this stream in a SQL DB like this:
deviceId timestamp device variable name value
testDevice 2018-04-06T11:46:11.842305 baiTest Status01_Test m_01_test 365
testDevice 2018-04-06T11:46:11.842306 hmuTest Status02_Test m_02_test 817
testDevice 2018-04-06T11:46:11.842307 vwzTest Status03_Test m_03_test 247
My code so far is:
WITH itemList AS (
SELECT deviceId, GetArrayElement(data,0) as datas
FROM [iotHub] WHERE topic = 'saveData' )
SELECT deviceId, datas.timestamp, datas.device, datas.variable, datas.name, datas.value
INTO [sqlTable]
FROM itemList
But this only stores the first index [0] of the data.array into the SQL.
I think storing the array could be handled with the **GetArrayElements** function but I was not able to manage it.
You should use GetArrayElements to flatten complex json in Azure Stream Analytics. Please refer to the following query.
SELECT
iothubAlias.deviceId,
arrayElement.ArrayValue.timestamp,
arrayElement.ArrayValue.device,
arrayElement.ArrayValue.variable,
arrayElement.ArrayValue.name,
arrayElement.ArrayValue.value
FROM [iothub-input] as iothubAlias
CROSS APPLY GetArrayElements(iothubAlias.data) AS arrayElement
You will get the result what you want.