I am currently working through Step 2 of this tutorial from Snowflake but am using my own JSON, stored in a column I'll call my_column in a table called my_table:
https://docs.snowflake.com/en/user-guide/json-basics-tutorial-query.html
The JSON file I am using has a key that contains the '#' character.
Example:
"#characteristics": {
"XXX": "XXXX",
"YYY": "YYYY",
"ZZZ": "ZZZZ"
}
When I try to use a SELECT statement that includes the FLATTEN function, ie something like this
select
value:xxx::number
from
my_table
, lateral flatten( input => my_column:#characteristics);
When I try this, SnowFlake gives me the error 'SQL compilation error: syntax error line 3 at position 57 unexpected '#characteristics'.' I have tried to escape the '#' character in front of attributes but have not had any luck.
You need to use double quotes. Try this one:
create table my_table ( my_column variant ) as
select parse_json ( $$ { "#characteristics": {
"XXX": "XXXX",
"YYY": "YYYY",
"ZZZ": "ZZZZ"
}} $$ );
select
value
from my_table
,lateral flatten( input => my_column:"#characteristics");
Related
I need to remove a few records (that contain t) in order to parse/flatten the data column. The query in the CTE that creates 'tab', works independent but when inside the CTE i get the same error while trying to parse json, if I were not have tried to filter out the culprit.
with tab as (
select * from table
where data like '%t%')
select b.value::string, a.* from tab a,
lateral flatten( input => PARSE_JSON( a.data) ) b ;
;
error:
Error parsing JSON: unknown keyword "test123", pos 8
example data:
Date Data
1-12-12 {id: 13-43}
1-12-14 {id: 43-43}
1-11-14 {test12}
1-11-14 {test2}
1-02-14 {id: 44-43}
It is possible to replace PARSE_JSON(a.data) with TRY_PARSE_JSON(a.data) which will produce NULL instead of error for invalid input.
More at: TRY_PARSE_JSON
I've got some problems with extracting values from nested json values in column.
I've got a column of data with values that looks almost like nested json, but some of jsons got \ between values and I need to clean them.
JSON looks like this:
{"mopub_json":
"{\"currency\":\"USD\",
\"country\":\"US\",
\"publisher_revenue\":0.01824}
"}
I need to get currency and publisher revenue as different columns and try this:
SET json_serialization_enable TO true;
SET json_serialization_parse_nested_strings TO true;
SELECT
JSON_EXTRACT_PATH_TEXT(column_name, 'mopub_json', 'publisher_revenue') as revenue_mopub,
JSON_EXTRACT_PATH_TEXT(column_name, 'mopub_json', 'currency') as currency_mopub
FROM(
SELECT replace(column_name, "\t", '')
FROM table_name)
I receive the next error:
[Amazon](500310) Invalid operation: column "\t" does not exist in events
When I'm trying this:
SET json_serialization_parse_nested_strings TO true;
SELECT
JSON_EXTRACT_PATH_TEXT(column_name, 'mopub_json', 'publisher_revenue') as revenue_mopub,
JSON_EXTRACT_PATH_TEXT(column_name, 'mopub_json', 'currency') as currency_mopub
FROM(
SELECT replace(column_name, chr(92), '')
FROM table_name)
I receive
Invalid operation: JSON parsing error
When I'm trying to extract values without replacing , I'm receiving empty columns.
Thank you for your help!
So your json isn't valid. JSON doesn't allow multiline text strings but I expect that the issue. Based on your query I think you don't want a single key and string but the whole structure. The reason the that quotes are backslashed is because they are inside a string. The json should look like:
{
"mopub_json": {
"currency": "USD",
"country": "US",
"publisher_revenue": 0.01824
}
}
Then the SQL you have should work.
I got this error.
ERROR: column "name" does not exist LINE 6:
SELECT JS, location, location->>Name
^ SQL state: 42703 Character: 278
from this query
WITH JsonTable AS
(
SELECT
'[
{
"Id":1,
"Name":"A01",
"Address":"aaa",
"SearchVector":null,
"ParentLocationId":null
},
{
"Id":4,
"Name":"B-01",
"Address":"bbb",
"SearchVector":null,
"ParentLocationId":null
}
]'::jsonb AS JS
)
SELECT JS, location, location->>Name
FROM JsonTable, jsonb_array_elements(JS) x (location)
How can I select JSON value?
You are missing quotes around the name of the JSON attribute that you want to select. Just like object keys must always be quoted when the JSON object is declared, they need to be quoted when they are accessed.
See the Postgres documentation for the JSON datatype and JSON Functions and Operators.
You would need to change this:
SELECT JS, location, location->>Name
To:
SELECT JS, location, location->>'Name'
Demo on DB Fiddle.
Test data
DROP TABLE t;
CREATE TABLE t(_id serial PRIMARY KEY, data jsonb);
INSERT INTO t(data) VALUES
('{"a":1,"b":2, "c":3}')
, ('{"a":11,"b":12, "c":13}')
, ('{"a":21,"b":22, "c":23}')
Problem statement: I want to receive an arbitrary JSONB parameter which acts as a filter on column t.data, such as
{ "b":{ "from":0, "to":20 }, "c":13 }
and use this to select matching rows from my test table t.
In this example, I want rows where b is between 0 and 20 and c = 13.
No error is required if the filter specifies a "column" (or "tag") which does not exist in t.data - it just fails to find a match.
I've used numeric values for simplicity but would like an approach which generalises to text as well.
What I have tried so far. I looked at the containment approach, which works for equality conditions, but am stumped on a generic way of handling range conditions:
select * from t
where t.data#> '{"c":13}'::jsonb;
Background: This problem arose when building a generic table-preview page on a website (for Admin users).
The page displays a filter based on various columns in whichever table is selected for preview.
The filter is then passed to a function in Postgres DB which applies this dynamic filter condition to the table.
It returns a jsonb array of the rows matching the filter specified by the user.
This jsonb array is then used to populate the Preview resultset.
The columns which make up the filter may change.
My Postgres version is 9.6 - thanks.
if you want to parse { "b":{ "from":0, "to":20 }, "c":13 } you need a parser. It is out of scope of json functions, but you can write "generic" query using AND and OR to filter by such json, eg:
https://www.db-fiddle.com/f/jAPBQggG3p7CxqbKLMbPKw/0
with filt(f) as (values('{ "b":{ "from":0, "to":20 }, "c":13 }'::json))
select *
from t
join filt on
(f->'b'->>'from')::int < (data->>'b')::int
and
(f->'b'->>'to')::int > (data->>'b')::int
and
(data->>'c')::int = (f->>'c')::int
;
Thanks for the comments/suggestions.
I will definitely look at GraphQL when I have more time - I'm working under a tight deadline at the moment.
It seems the consensus is that a fully generic solution is not achievable without a parser.
However, I got a workable first draft - it's far from ideal but we can work with it. Any comments/improvements are welcome ...
Test data (expanded to include dates & text fields)
DROP TABLE t;
CREATE TABLE t(_id serial PRIMARY KEY, data jsonb);
INSERT INTO t(data) VALUES
('{"a":1,"b":2, "c":3, "d":"2018-03-10", "e":"2018-03-10", "f":"Blah blah" }')
, ('{"a":11,"b":12, "c":13, "d":"2018-03-14", "e":"2018-03-14", "f":"Howzat!"}')
, ('{"a":21,"b":22, "c":23, "d":"2018-03-14", "e":"2018-03-14", "f":"Blah blah"}')
First draft of code to apply a jsonb filter dynamically, but with restrictions on what syntax is supported.
Also, it just fails silently if the syntax supplied does not match what it expects.
Timestamp handling a bit kludgey, too.
-- Handle timestamp & text types as well as int
-- See is_timestamp(text) function at bottom
with cte as (
select t.data, f.filt, fk.key
from t
, ( values ('{ "a":11, "b":{ "from":0, "to":20 }, "c":13, "d":"2018-03-14", "e":{ "from":"2018-03-11", "to": "2018-03-14" }, "f":"Howzat!" }'::jsonb ) ) as f(filt) -- equiv to cross join
, lateral (select * from jsonb_each(f.filt)) as fk
)
select data, filt --, key, jsonb_typeof(filt->key), jsonb_typeof(filt->key->'from'), is_timestamp((filt->key)::text), is_timestamp((filt->key->'from')::text)
from cte
where
case when (filt->key->>'from') is null then
case jsonb_typeof(filt->key)
when 'number' then (data->>key)::numeric = (filt->>key)::numeric
when 'string' then
case is_timestamp( (filt->key)::text )
when true then (data->>key)::timestamp = (filt->>key)::timestamp
else (data->>key)::text = (filt->>key)::text
end
when 'boolean' then (data->>key)::boolean = (filt->>key)::boolean
else false
end
else
case jsonb_typeof(filt->key->'from')
when 'number' then (data->>key)::numeric between (filt->key->>'from')::numeric and (filt->key->>'to')::numeric
when 'string' then
case is_timestamp( (filt->key->'from')::text )
when true then (data->>key)::timestamp between (filt->key->>'from')::timestamp and (filt->key->>'to')::timestamp
else (data->>key)::text between (filt->key->>'from')::text and (filt->key->>'to')::text
end
when 'boolean' then false
else false
end
end
group by data, filt
having count(*) = ( select count(distinct key) from cte ) -- must match on all filter elements
;
create or replace function is_timestamp(s text) returns boolean as $$
begin
perform s::timestamp;
return true;
exception when others then
return false;
end;
$$ strict language plpgsql immutable;
I have postgresql with jsonb field that always contains array.
I need to append new values to that array or update already existing values by index.
Looks like jsonb_set function meet my requirements. And for append new element i just need to max array index and update element with it.
But i have a trouble doing this. Lets make it step by step.
We have table campaigns with jsonb field team_members.
select id, jsonb_set(team_members, '{0}', '{"name" : "123"}') from campaigns;
id | jsonb_set
-----+-------------------
102 | [{"name": "123"}]
Okay great, if set path '{0}' statically everything works.
Lets do that dynamically
SQL for getting array length (it is our index for append)
select '{' || jsonb_array_length(team_members) || '}'::text from campaigns;
?column?
----------
{0}
Getting all together
select jsonb_set(team_members, '{' || jsonb_array_length(team_members) || '}', '{"name" : "123"}') from campaigns;
ERROR: function jsonb_set(jsonb, text, unknown) does not exist
LINE 1: select jsonb_set(team_members, '{' ||
jsonb_array_length(tea...
^ HINT: No function matches the given name and argument types. You might
need to add explicit type casts.
My question is - how can i get rid of this error ? What i'm doing wrong ?
Thanks in advance.
something like this?..
t=# with jpath as (select concat('{',0,'}')::text[] path) select jsonb_set('[]'::jsonb,path,'{"name": "123"}'::jsonb) from jpath;
jsonb_set
-------------------
[{"name": "123"}]
(1 row)
In your case should be like:
select
jsonb_set(
team_members
, concat('{',jsonb_array_length(team_members),'}')::text[]
, '{"name" : "123"}'
)
from campaigns;