Given this dataset:
[
{
"dataChannelId": 8516,
"measures": [
{
"dateTime": "2019-01-01T12:00:00",
},
{
"dateTime": "2019-01-02T12:00:00",
}
}]
And this query:
WITH
temp AS
(
SELECT
dataChannelId,
arrayElement.ArrayValue as element
FROM GriegInputStream
CROSS APPLY GetArrayElements([mesurasdfes]) AS arrayElement
)
SELECT
temp.dataChannelId as sensorId, temp.element.dateTime, temp.element.value,temp.element.unit,temp.element.maxValue, temp.element.minValue
INTO
Sensoroutput
FROM
temp
I get invalid column name, does not exist on dataChannelId, but measures seem to work fine. How can I access this value without stream analytics complaining?
Your sample json data lack a square bracket ].
sample data:
[
{
"dataChannelId": 8516,
"measures": [
{
"dateTime": "2019-01-01T12:00:00",
},
{
"dateTime": "2019-01-02T12:00:00",
}
]
}
]
Query sql:
WITH
temp AS
(
SELECT
jsoninput.dataChannelId,
arrayElement.ArrayValue as element
FROM jsoninput
CROSS APPLY GetArrayElements(jsoninput.measures) AS arrayElement
)
SELECT
temp.dataChannelId as sensorId, temp.element.dateTime
INTO
Sensoroutput
FROM
temp
Output:
Related
My Postgres DB Table BookingDetails has columns - id(bigint), info(jsonb)
My Json structure:
"bookingType": “special”,
“travellers”: [
{
"id": 1,
"nationality": "SomeValue",
}
],
“someTag”: “abc”
}
The travellers here is an array (and in this example, there’s only one element)
I want to:
Select records where nationality=‘CertainValue’
Update records set nationality=‘AnotherValue’ where nationality=‘CertainValue’
I could retrieve the array using:
select CAST (info->'travellers' AS TEXT) from BookingDetails
[
{
"id": 1,
"nationality": "SomeValue",
}
]
I could retrieve the array first element using:
select CAST (info->'travellers'->>0 AS TEXT) from BookingDetails
{
"id": 1,
"nationality": "SomeValue"
}
But how to do a select based on nationality=‘CertainValue’?
try this :
SELECT jsonb_set(info, array['travellers', (a.id - 1) :: text, 'nationality'], to_jsonb('AnotherValue' :: text))
FROM BookingDetails
CROSS JOIN LATERAL jsonb_array_elements(info->'travellers') WITH ORDINALITY AS a(content, id)
WHERE a.content->>'nationality' = 'CertainValue'
see the test result in dbfiddle
I want to remove a JSONB object by their unique 'id' value from a JSONB array. I am no expert at writing SQL code, but I managed to write the concatenate function.
For an example: Remove this object from an array below.
{
"id": "ad26e2be-19fd-4862-8f84-f2f9c87b582e",
"title": "Wikipedia",
"links": [
"https://en.wikipedia.org/1",
"https://en.wikipedia.org/2"
]
},
Schema:
CREATE TABLE users (
url text not null,
user_id SERIAL PRIMARY KEY,
name VARCHAR,
list_of_links jsonb default '[]'
);
list_of_links format:
[
{
"id": "ad26e2be-19fd-4862-8f84-f2f9c87b582e",
"title": "Wikipedia",
"links": [
"https://en.wikipedia.org/1",
"https://en.wikipedia.org/2"
]
},
{
"id": "451ac172-b93e-4158-8e53-8e9031cfbe72",
"title": "Russian Wikipedia",
"links": [
"https://ru.wikipedia.org/wiki/",
"https://ru.wikipedia.org/wiki/"
]
},
{
"id": "818b99c8-479b-4846-ac15-4b2832ec63b5",
"title": "German Wikipedia",
"links": [
"https://de.wikipedia.org/any",
"https://de.wikipedia.org/any"
]
},
...
]
The concatenate function:
update users set list_of_links=(
list_of_links || (select *
from jsonb_array_elements(list_of_links)
where value->>'id'='ad26e2be-19fd-4862-8f84-f2f9c87b582e'
)
)
where url='test'
returning *
;
Your json data is structured so you have to unpack it, operate on the unpacked data, and then repack it again:
SELECT u.url, u.user_id, u.name,
jsonb_agg(
jsonb_build_object('id', l.id, 'title', l.title, 'links', l.links)
) as list_of_links
FROM users u
CROSS JOIN LATERAL jsonb_to_recordset(u.list_of_links) AS l(id uuid, title text, links jsonb)
WHERE l.id != 'ad26e2be-19fd-4862-8f84-f2f9c87b582e'::uuid
GROUP BY 1, 2, 3
The function jsonb_to_recordset is a set-returning function so you have to use it as a row source, joined to its originating table with the LATERAL clause so that the list_of_links column is available to the function to be unpacked. Then you can delete the records you are not interested in using the WHERE clause, and finally repack the structure by building the record fields into a jsonb structure and then aggregating the individual records back into an array.
I wrote this on JS but that does not matter to how it works. Essentially, its getting all the items from the array, then finding the matching id which returns an index. And using that index, I use "-" operator which takes the index and removes it from the array. Sorry if my grammar is bad.
//req.body is this JSON object
//{"url":"test", "id": "ad26e2be-19fd-4862-8f84-f2f9c87b582e"}
var { url, id } = req.body;
pgPool.query(
`
select list_of_links
from users
where url=$1;
`,
[url],
(error, result) => {
//block code executing further if error is true
if (error) {
res.json({ status: "failed" });
return;
}
if (result) {
// this function just returns the index of the array element where the id matches from request's id
// 0, 1, 2, 3, 4, 5
var index_of_the_item = result.rows.list_of_links
.map(({ id: db_id }, index) =>
db_id === id ? index : false
)
.filter((x) => x !== false)[0];
//remove the array element by it's index
pgPool.query(
`
update users
set list_of_links=(
list_of_links - $1::int
)
where url=$2
;
`,
[index_of_the_item, url], (e, r) => {...}
);
}
}
);
I have a table available in Athena which has one column with JSON structured as following:
{
"455a9410-29a8-48a3-ad22-345afa3cd295":
{
"legacy_id": 1599677886,
"w_ids":
[
"845254682",
"831189092"
]
},
"5e74c911-0b63-4b84-8ad4-77dd9bed7b53":
{
"legacy_id": 1599707069,
"w_ids":
[
"1032024432"
]
},
"7b988890-20ff-4279-94df-198369a58848":
{
"legacy_id": 1601097861,
"w_ids":
[
"1032024432"
]
}
}
I would like to convert this into an ARRAY in the following format:
[
{"new_id"="455a9410-29a8-48a3-ad22-345afa3cd295","legacy_id"=1599677886,"w_ids"=["845254682","831189092"]},
{"new_id"="5e74c911-0b63-4b84-8ad4-77dd9bed7b53","legacy_id"=1599707069,"w_ids"=["1032024432"]},
{"new_id"="7b988890-20ff-4279-94df-198369a58848","legacy_id"=1601097861,"w_ids"=["1032024432"]}
]
I already was able to extract legacy_idand w_ids with the following statement but I struggle to add the original key as value:
with example_data as
(
select * from (
VALUES('{ "455a9410-29a8-48a3-ad22-345afa3cd295": { "legacy_id": 1599677886, "w_ids": [ "845254682", "831189092" ] }, "5e74c911-0b63-4b84-8ad4-77dd9bed7b53": { "legacy_id": 1599707069, "w_ids": [ "1032024432" ] }, "7b988890-20ff-4279-94df-198369a58848": { "legacy_id": 1601097861, "w_ids": [ "1032024432" ] }}')
) as t(col)
)
select *
,transform(map_values(cast(json_parse(col) AS map(varchar, json))),entry -> MAP_FROM_ENTRIES(ARRAY[('legacy_id',json_extract(entry,'$.legacy_id')),('w_ids',json_extract(entry,'$.w_ids'))]))
from example_data;
One approach can be using map_values over transform_values instead of transform over map_values:
select map_values(
transform_values(
cast(json_parse(col) AS map(varchar, json)),
(key, entry)->MAP_FROM_ENTRIES(
ARRAY [('new_id', cast(key as json)),
('legacy_id', json_extract(entry, '$.legacy_id')),
('w_ids', json_extract(entry, '$.w_ids')) ]
)
)
)
from example_data;
Output:
_col0
[{new_id='455a9410-29a8-48a3-ad22-345afa3cd295', legacy_id=1599677886, w_ids=['845254682','831189092']}, {new_id='5e74c911-0b63-4b84-8ad4-77dd9bed7b53', legacy_id=1599707069, w_ids=['1032024432']}, {new_id='7b988890-20ff-4279-94df-198369a58848', legacy_id=1601097861, w_ids=['1032024432']}]
Each record in my database has a json list, from which I'm trying to return a value inside each element.
I'm trying to return
environmental services, energy & utilities
From
[
{
"key": "environmental services"
},
{
"key": "energy & utilities"
}
]
And.
construction
From
[
{
"key": "construction"
}
]
Where the length of each json list in each record is undetermined.
I can get the first value with:
select
column_name -> 0 -> 'key'
from table
But how to get all.
How can I do this in postgresql?
Use the function jsonb_array_elements() in a lateral join to get all elements of the arrays and the aggregate function string_agg():
with my_table(json_col) as (
values
('[
{
"key": "environmental services"
},
{
"key": "energy & utilities"
}
]'::jsonb),
('[
{
"key": "construction"
}
]')
)
select string_agg(value->>'key', ', ')
from my_table
cross join jsonb_array_elements(json_col)
group by json_col -- use PK here (if exists)
string_agg
--------------------------------------------
environmental services, energy & utilities
construction
(2 rows)
Each row in my table has a field that is an array, and I'd like to get a field from the first array entry.
For example, if my row is
[
{
"user_dim": {
"user_id": "123",
"user_properties": [
{
"key": "content_group",
"value": {
"value": {
"string_value": "my_group"
}
}
}
]
},
"event_dim": [
{
"name": "main_menu_item_selected",
"timestamp_micros": "1517584420597000"
},
{
"name": "screen_view",
"timestamp_micros": "1517584420679001"
}
]
}
]
I'd like to get
user_id: 123, content_group: my_group, timestamp_1517584420597000
As Elliott mentioned - BigQuery Standard SQL has way much better support for ARRAYs than legacy SQL. And in general, BigQuery team recommend using Standard SQL
So, below is for BigQuery Standard SQL (including handling wildcard stuff)
#standardSQL
SELECT
user_dim.user_id AS user_id,
(SELECT value.value.string_value
FROM UNNEST(user_dim.user_properties)
WHERE key = 'content_group' LIMIT 1
) content_group,
(SELECT event.timestamp_micros
FROM UNNEST(event_dim) event
WHERE name = 'main_menu_item_selected'
) ts
FROM `project.dataset.app_events_*`
WHERE _TABLE_SUFFIX BETWEEN '20180129' AND '20180202'
with result (for the dummy example from your question)
Row user_id content_group ts
1 123 my_group 1517584420597000