I have table with one column called cols. I want to separate cols to new column based on type: SKU and MARK using bigquery.
cols
"dsc":[{"amount":30000,"c_amount":0,"d_amount":0,"d_id":"","scope":"CART","title":"Promo","type":"SKU"},{"amount":7000,"c_amount":0,"d_amount":7000,"d_id":"x","scope":"CART_D","title":"","type":"MARK"}]
The result i want is like:
sku_amount sku_c_amount sku_d_amount sku_d_id sku_scope sku_title mark_amount mark_c_amount mark_d_amount mark_d_id mark_scope
30000 0 0 CART Promo 7000 0 0 x CART_D
Anyone know the script? thank you
Consider below approach
create temp table temp_table as
select id1,
max(if(key = 'type', value, null)) over (partition by id1, id2) || '_' || key as key,
value, offset
from (
select md5(cols) as id1, md5(json) id2, arr[offset(0)] as key, arr[offset(1)] as value, offset
from your_table, unnest(json_extract_array('{' || cols || '}', '$.dsc')) json with offset,
unnest(split(translate(json, '{}"', ''))) kv,
unnest([struct(split(kv, ':') as arr)])
);
execute immediate (select '''
select * except(id1) from (select * except(offset) from temp_table)
pivot (any_value(value) for key in ("''' || string_agg(key, '","' order by offset, key) || '''"))
'''
from (select distinct key, offset from temp_table where not ends_with(key, '_type'))
);
if applied to sample data in your question - output is
Related
Trying to transpose a table in GBQ. The solution needs to be able to work dynamically with varying columns and rows. I know there is a way to do this by listing out columns specifically, but I have varying tables with large numbers of rows. There is an unpivot function in GBQ now and my idea was to Unpivot, then Pivot it. Not sure if anyone has dealt with this yet?
Starting Data
Result
Consider below approach
select * from (
select * from data
unpivot (value for Fruit in (Apple, Bananas, Orange))
)
pivot (max(value) for `Group` in ('North', 'South', 'West'))
If applied to sample data in your question - output is
Below version builds all references needed for unpivot and pivot dynamically on fly
execute immediate (select '''select * from (
select * from `project.dataset.table`
unpivot (value for Fruit in (''' || (select array_to_string(regexp_extract_all(to_json_string((select as struct * except(`Group`) from unnest([t]))), r'"([^"]+)":'), ', ')
from `project.dataset.table` t limit 1) || '''))
)
pivot (max(value) for `Group` in (''' || (select '"' || string_agg(distinct `Group`, '", "' order by `Group`) || '"' from `project.dataset.table`) || '''))
'''
);
In order to pivot my big query table, I found this code
SELECT 'SELECT id, ' ||
STRING_AGG(
'MAX(IF(key = "' || key || '", value, NULL)) as `' || key || '`'
)
|| ' FROM `project.dataset.table` GROUP BY id ORDER BY id'
FROM (
SELECT key
FROM `project.dataset.table`
GROUP BY key
ORDER BY key
But even if I apply EXECUTE IMMEDIATE function, it returns a string of the code above.
What did I missed in that function ?
Thanks for your help
Use below
EXECUTE IMMEDIATE(
SELECT 'SELECT id, ' ||
STRING_AGG(
'MAX(IF(key = "' || key || '", value, NULL)) as `' || key || '`'
)
|| ' FROM `project.dataset.table` GROUP BY id ORDER BY id'
FROM (
SELECT key
FROM `project.dataset.table`
GROUP BY key
ORDER BY key
)
);
We are generating a dynamic query using the script below :
execute immediate (select
''' select event_id, value, ''' || string_agg('''
(select value from b.key_value where key = "''' || key_name || '''") as ''' || key_name , ''', ''')
|| '''
from (
select event_id, value,
array(
select as struct
json_extract_scalar(kv, '$.key') key,
json_extract_scalar(kv, '$.value') value
from a.kvs kv
) key_value
from `project.dataset.table`,
unnest([struct(json_extract_array(dimensions, '$.key_value') as kvs)]) a
) b
'''
from (
select distinct json_extract_scalar(kv, '$.key') key_name
from `project.dataset.table`,
unnest(json_extract_array(dimensions, '$.key_value')) as kv
)
)
We want to create a view out of this is there any way in BQ by which we can create a view from this, so as user we can directly query our view.
Should be as simple as just adding create or replace view project.dataset.myview as to original query
execute immediate (select
''' create or replace view `project.dataset.myview` as select event_id, value, ''' || string_agg('''
(select value from b.key_value where key = "''' || key_name || '''") as ''' || key_name , ''', ''')
|| '''
from (
select event_id, value,
array(
select as struct
json_extract_scalar(kv, '$.key') key,
json_extract_scalar(kv, '$.value') value
from a.kvs kv
) key_value
from `project.dataset.table`,
unnest([struct(json_extract_array(dimensions, '$.key_value') as kvs)]) a
) b
'''
from (
select distinct json_extract_scalar(kv, '$.key') key_name
from `project.dataset.table`,
unnest(json_extract_array(dimensions, '$.key_value')) as kv
)
)
If customer first_name-'Monika',
last_name='Awasthi'
Then I am using below query to return value in json format:
SELECT *
FROM
(
SELECT JSON_ARRAYAGG(JSON_OBJECT('CODE' IS '1','VALUE' IS 'Monika'||' '||'Awasthi'))
FROM DUAL
);
It is working fine & give below output:
[{"CODE":"1","VALUE":"Monika Awasthi"}]
But I want one more value which should be reversed means output should be:
[{"CODE":"1","VALUE":"Monika Awasthi"},{"CODE":"2","VALUE":"Awasthi Monika"}]
Kindly give me some suggestions. Thank You
Another approach is to use a CTE to generate the two codes and values; your original version could be written to get the name data from a table or CTE:
-- CTE for sample data
WITH cte (first_name, last_name) AS (
SELECT 'Monika', 'Awasthi' FROM DUAL
)
-- query against CTE or table
SELECT JSON_ARRAYAGG(JSON_OBJECT('CODE' IS '1','VALUE' IS last_name ||' '|| first_name))
FROM cte;
And you could then extend that with a CTE that generates the value with the names in both orders:
WITH cte1 (first_name, last_name) AS (
SELECT 'Monika', 'Awasthi' FROM DUAL
),
cte2 (code, value) AS (
SELECT 1 AS code, first_name || ' ' || last_name FROM cte1
UNION ALL
SELECT 2 AS code, last_name || ' ' || first_name FROM cte1
)
SELECT JSON_ARRAYAGG(JSON_OBJECT('CODE' IS code,'VALUE' IS value))
FROM cte2;
which gives:
JSON_ARRAYAGG(JSON_OBJECT('CODE'ISCODE,'VALUE'ISVALUE))
-------------------------------------------------------------------------
[{"CODE":1,"VALUE":"Monika Awasthi"},{"CODE":2,"VALUE":"Awasthi Monika"}]
db<>fiddle
A simple logic through use of SQL(without using PL/SQL) in order to generate code values as only be usable for two columns as in this case might be
SELECT JSON_ARRAYAGG(
JSON_OBJECT('CODE' IS tt.column_id,
'VALUE' IS CASE WHEN column_id=1
THEN name||' '||surname
ELSE surname||' '||name
END)
) AS result
FROM t
CROSS JOIN (SELECT column_id FROM user_tab_cols WHERE table_name = 'T') tt
where t is a table which hold name and surname columns
Demo
More resilient solution might be provided through use of PL/SQL, even more columns exist within the data source such as
DECLARE
v_jso VARCHAR2(4000);
v_arr OWA.VC_ARR;
v_arr_t JSON_ARRAY_T := JSON_ARRAY_T();
BEGIN
FOR c IN ( SELECT column_id FROM user_tab_cols WHERE table_name = 'T' )
LOOP
SELECT 'JSON_OBJECT( ''CODE'' IS '||MAX(c.column_id)||',
''VALUE'' IS '||LISTAGG(column_name,'||'' ''||')
WITHIN GROUP (ORDER BY ABS(column_id-c.column_id))
||' )'
INTO v_arr(c.column_id)
FROM ( SELECT * FROM user_tab_cols WHERE table_name = 'T' );
EXECUTE IMMEDIATE 'SELECT '||v_arr(c.column_id)||' FROM t' INTO v_jso;
v_arr_t.APPEND(JSON_OBJECT_T(v_jso));
END LOOP;
DBMS_OUTPUT.PUT_LINE(v_arr_t.STRINGIFY);
END;
/
Demo
As I explained in a comment under your question, I am not clear on how you define the CODE values for your JSON string (assuming you have more than one customer).
Other than that, if you need to create a JSON array of objects from individual strings (as in your attempt), you probably need to use JSON_ARRAY rather than JSON_ARRAYAGG. Something like I show below. Incidentally, I also don't know why you needed to SELECT * FROM (subquery) - the outer SELECT seems entirely unnecessary.
So, if you don't actually aggregate over a table, but just need to build a JSON array from individual pieces:
select json_array
(
json_object('CODE' is '1', 'VALUE' is first_name || ' ' || last_name ),
json_object('CODE' is '2', 'VALUE' is last_name || ' ' || first_name)
) as result
from ( select 'Monika' as first_name, 'Awasthi' as last_name from dual )
;
RESULT
------------------------------------------------------------------------------
[{"CODE":"1","VALUE":"Monika Awasthi"},{"CODE":"2","VALUE":"Awasthi Monika"}]
Using PostgreSQL 9.3, I'm creating a Jasper reports template to make a pdf report. I want to create reports of different tables, with multiple columns, all with the same template. A solution could be to get values of register as pairs of column name and value per id.
By example, if I had a table like:
id | Column1 | Column2 | Column3
-------------------------------------------------
1 | Register1C1 | Register1C2 | Register1C3
I would like to get the register as:
Id | ColumnName | Value
-----------------------------
1 | Column1 | Register1C1
1 | Column2 | Register1C2
1 | Column3 | Register1C3
The data type of value columns can vary!
Is it possible? How can I do this?
If all your columns share the same data type and order of rows does not have to be enforced:
SELECT t.id, v.*
FROM tbl t, LATERAL (
VALUES
('col1', col1)
, ('col2', col2)
, ('col3', col3)
-- etc.
) v(col, val);
About LATERAL (requires Postgres 9.3 or later):
What is the difference between LATERAL and a subquery in PostgreSQL?
Combining it with a VALUES expression:
Crosstab transpose query request
SELECT DISTINCT on multiple columns
For varying data types, the common denominator would be text, since every type can be cast to text. Plus, order enforced:
SELECT t.id, v.col, v.val
FROM tbl t, LATERAL (
VALUES
(1, 'col1', col1::text)
, (2, 'col2', col2::text)
, (3, 'col3', col3::text)
-- etc.
) v(rank, col, val)
ORDER BY t.id, v.rank;
In Postgres 9.4 or later use the new unnest() for multiple arrays:
SELECT t.id, v.*
FROM tbl t, unnest('{col1,col2,col3}'::text[]
, ARRAY[col1,col2,col3]) v(col, val);
-- , ARRAY[col1::text,col2::text,col3::text]) v(col, val);
The commented alternative for varying data types.
Full automation for Postgres 9.4:
The query above is convenient to automate for a dynamic set of columns:
CREATE OR REPLACE FUNCTION f_transpose (_tbl regclass, VARIADIC _cols text[])
RETURNS TABLE (id int, col text, val text) AS
$func$
BEGIN
RETURN QUERY EXECUTE format(
'SELECT t.id, v.* FROM %s t, unnest($1, ARRAY[%s]) v'
, _tbl, array_to_string(_cols, '::text,') || '::text'))
-- , _tbl, array_to_string(_cols, ','))) -- simple alternative for only text
USING _cols;
END
$func$ LANGUAGE plpgsql;
Call - with table name and any number of column names, any data types:
SELECT * FROM f_transpose('table_name', 'column1', 'column2', 'column3');
Weakness: the list of column names is not safe against SQL injection. You could gather column names from pg_attribute instead. Example:
How to perform the same aggregation on every column, without listing the columns?
SELECT id
,unnest(string_to_array('col1,col2,col3', ',')) col_name
,unnest(string_to_array(col1 || ',' || col2 || ',' || col3, ',')) val
FROM t
Try following method:
My sample table name is t,to get the n columns name you can use this query
select string_agg(column_name,',') cols from information_schema.columns where
table_name='t' and column_name<>'id'
this query will selects all columns in your table except id column.If you want to specify schema name then use table_schema='your_schema_name' in where clause
To create select query dynamically
SELECT 'select id,unnest(string_to_array(''' || cols || ''','','')) col_name,unnest(string_to_array(' || cols1 || ','','')) val from t'
FROM (
SELECT string_agg(column_name, ',') cols -- here we'll get all the columns in table t
,string_agg(column_name, '||'',''||') cols1
FROM information_schema.columns
WHERE table_name = 't'
AND column_name <> 'id'
) tb;
And using following plpgsql function dynamically creates SELECT id,unnest(string_to_array('....')) col_name,unnest(string_to_array(.., ',')) val FROM t and execute.
CREATE OR replace FUNCTION fn ()
RETURNS TABLE (
id INT
,columname TEXT
,columnvalues TEXT
) AS $$
DECLARE qry TEXT;
BEGIN
SELECT 'select id,unnest(string_to_array(''' || cols || ''','','')) col_name,unnest(string_to_array(' || cols1 || ','','')) val from t'
INTO qry
FROM (
SELECT string_agg(column_name, ',') cols
,string_agg(column_name, '||'',''||') cols1
FROM information_schema.columns
WHERE table_name = 't'
AND column_name <> 'id'
) tb;
RETURN QUERY
EXECUTE format(qry);
END;$$
LANGUAGE plpgsql
Call this function like select * from fn()