I’m trying to write a query in BigQuery that produces the count of the unique transactions and the combination of column names populated.
I have a table:
TRAN CODE
Full Name
Given Name
Surname
DOB
Phone
The result set I’m after is:
TRAN CODE
UNIQUE TRANSACTIONS
NAME OF POPULATED COLUMNS
A
3
Full Name
A
4
Full Name,Phone
B
5
Given Name,Surname
B
10
Given Name,Surname,DOB,Phone
The result set shows that for TRAN CODE A
3 distinct customers provided Full Name
4 distinct customers provided Full Name and Phone #
For TRAN CODE B
5 distinct customers provided Given Name and Surname
10 distinct customers provided Given Name, Surname, DOB, Phone #
Currently to produce my results I’m doing it manually.
I tried using ARRAY_AGG but couldn’t get it working.
Any advice work be appreciated.
Thank you.
I think you want something like this:
select tran_code,
array_to_string(array[case when full_name is not null then 'full_name' end,
case when given_name is not null then 'given_name' end,
case when surname is not null then 'surname' end,
case when dob is not null then 'dob' end,
case when phone is not null then 'phone' end
], ','),
count(*)
from t
group by 1, 2
Consider below approach - no any dependency on column names rather than TRAN_CODE - quite generic!
select TRAN_CODE,
count(distinct POPULATED_VALUES) as UNIQUE_TRANSACTIONS,
POPULATED_COLUMNS
from (
select TRAN_CODE,
( select as struct
string_agg(col, ', ' order by offset) POPULATED_COLUMNS,
string_agg(val order by offset) POPULATED_VALUES,
string_agg(cast(offset as string) order by offset) pos
from unnest(regexp_extract_all(to_json_string(t), r'"([^"]+?)":')) col with offset
join unnest(regexp_extract_all(to_json_string(t), r'"[^"]+?":("[^"]+?"|null)')) val with offset
using(offset)
where val != 'null'
and col != 'TRAN_CODE'
).*
from `project.dataset.table` t
)
group by TRAN_CODE, POPULATED_COLUMNS
order by TRAN_CODE, any_value(pos)
below is output example
#Gordon_Linoff's solution is the best, but an alternative would be to do the following:
SELECT
TRAN_CODE,
COUNT(TRAN_ROW) AS unique_transactions,
populated_columns
FROM (
SELECT
TRAN_CODE,
TRAN_ROW,
# COUNT(value) AS unique_transactions,
STRING_AGG(field, ",") AS populated_columns
FROM (
SELECT
* EXCEPT(DOB),
CAST(DOB AS STRING ) AS DOB,
ROW_NUMBER() OVER () AS TRAN_ROW
FROM
sample) UNPIVOT(value FOR field IN (Full_name,
Given_name,
Surname,
DOB,
Phone))
GROUP BY
TRAN_CODE,
TRAN_ROW )
GROUP BY
TRAN_CODE,
populated_columns
But this should be more expensive...
Related
I am trying to create a column with a case statement, then concatenate the column. Here is an example code.
WITH base AS (
SELECT ID, Date, Action, case when (Date is null then Action || '**' else Action End) Action_with_no_date
FROM <Table_Name>
)
SELECT ID, "array_join"("array_agg"(DISTINCT Action_with_no_date), ', ') Action_with_no_date
FROM base
GROUP BY ID;
Basically, the Action_with_no_date will display the concatenation of values in Action with '**' string added to the values where Date is null for each ID
After I did this, I found an edge case.
If there is the same Action (i.e. play) taken for one ID, and if one action has date and the other one doesn't, then the output will have one play and one play** for the ID
However, I want this to display just one play with **.
Below is the example data for ID = 1
ID Date Action
1 1/2/22 read
1 1/3/22 play
1 NULL play
and expected result for the ID
ID Action_with_no_date
1 read, play**
How should I handle this?
You can calculate ** suffix if there is any row with null per id and action using analytic max() with case expression. Then concatenate suffix with action.
Demo:
with mytable as (
SELECT * FROM (
VALUES
(1, '1/2/22', 'read'),
(1, '1/3/22', 'play'),
(1, NULL, 'play')
) AS t (id, date, action)
)
select id, array_join(array_agg(DISTINCT action||suffix), ', ')
from
(
select id, date, action,
max(case when date is null then '**' else '' end) over(partition by id, action) as suffix
from mytable
)s
group by id
Result:
1 play**, read
example:
there is a json array column(type:string) from a hive table like:
"[{"filed":"name", "value":"alice"}, {"filed":"age", "value":"14"}......]"
how to convert it into :
name age
alice 14
by hive sql?
I've tried lateral view explode but it's not working.
thanks a lot!
This is working example of how it can be parsed in Hive. Customize it yourself and debug on real data, see comments in the code:
with your_table as (
select stack(1,
1,
'[{"field":"name", "value":"alice"}, {"field":"age", "value":"14"}, {"field":"something_else", "value":"somevalue"}]'
) as (id,str) --one row table with id and string with json. Use your table instead of this example
)
select id,
max(case when field_map['field'] = 'name' then field_map['value'] end) as name,
max(case when field_map['field'] = 'age' then field_map['value'] end) as age --do the same for all fields
from
(
select t.id,
t.str as original_string,
str_to_map(regexp_replace(regexp_replace(trim(a.field),', +',','),'\\{|\\}|"','')) field_map --remove extra characters and convert to map
from your_table t
lateral view outer explode(split(regexp_replace(regexp_replace(str,'\\[|\\]',''),'\\},','}|'),'\\|')) a as field --remove [], replace "}," with '}|" and explode
) s
group by id --aggregate in single row
;
Result:
OK
id name age
1 alice 14
One more approach using get_json_object:
with your_table as (
select stack(1,
1,
'[{"field":"name", "value":"alice"}, {"field":"age", "value":"14"}, {"field":"something_else", "value":"somevalue"}]'
) as (id,str) --one row table with id and string with json. Use your table instead of this example
)
select id,
max(case when field = 'name' then value end) as name,
max(case when field = 'age' then value end) as age --do the same for all fields
from
(
select t.id,
get_json_object(trim(a.field),'$.field') field,
get_json_object(trim(a.field),'$.value') value
from your_table t
lateral view outer explode(split(regexp_replace(regexp_replace(str,'\\[|\\]',''),'\\},','}|'),'\\|')) a as field --remove [], replace "}," with '}|" and explode
) s
group by id --aggregate in single row
;
Result:
OK
id name age
1 alice 14
I have it
-- -- -- --
01 A1 B1 99
01 A1 B1 98
02 A2 B2 97
02 A2 B2 96
I need this
-- -- -- --
01 A1 B1 99
98
02 A2 B2 97
96
------------
I can not repeat the data that I will present in a excel,
My result needs to be just so.
In my actual table, the last column are responses of forms and the first columns (those that can not repeat) are customer data as (phone, name ...).
The end result of this "query" will populate a "DataTable" and will be presented in a file "xlsx".
Thanks for sharing knowledge ^^
If you have SQL2012+
SELECT
ISNULL(NULLIF(Column1,LAG(Column1) OVER(ORDER BY Column1)),'')
,ISNULL(NULLIF(Column2,LAG(Column2) OVER(ORDER BY Column1,Column2)),'')
,ISNULL(NULLIF(Column3,LAG(Column3) OVER(ORDER BY Column1,Column2,Column3)),'')
,Column4
FROM #mytable
ORDER BY Column1,Column2,Column3,Column4 DESC
It's a little messy, but you can do it in the database. You basically make a subquery that gets the smallest value, and then join that to the regular table and blank out values that don't match. I created your sample set like this:
CREATE TABLE mytable (N1 VARCHAR(2), A VARCHAR(2), B VARCHAR(2), N2 VARCHAR(2))
INSERT INTO mytable VALUES
('01', 'A1', 'B1', '99'),
('01', 'A1', 'B1', '98'),
('02', 'A2', 'B2', '97'),
('02', 'A2', 'B2', '96')
And then was able to get the result like this:
SELECT
CASE WHEN O.N2 = I.N2 THEN O.N1 ELSE '' END,
CASE WHEN O.N2 = I.N2 THEN O.A ELSE '' END,
CASE WHEN O.N2 = I.N2 THEN O.B ELSE '' END,
O.N2
FROM
(SELECT MAX(N2) AS N2, N1, A, B FROM mytable GROUP BY N1, A, B) I
INNER JOIN mytable O
ON O.A = I.A AND O.B = I.B AND O.N1 = I.N1
ORDER BY O.N1 ASC
we can use ROW_NUMBER to get the sequence and substitute '' for all rows where sequence is greater than 1
with CTE
AS
( SELECT ID, ColumnA, ColumnB, value,ROW_NUMBER() over ( PARTITION by id order by id) as seq
FROM tableA
)
, CTE1
AS
(
select id, ColumnA, ColumnB, value, seq from CTE where seq =1
UNION
SELECT id ,'','', value , seq from CTE where seq >1
)
SELECT case when seq >1 THEN NULL ELSE id END as id, columnA, columnB, value from CTE1
You can achieve what you want using a query.
You haven't provided DDL so I am going to asume your columns are called a, b, c and d respectively
; WITH cte AS (
SELECT a
, b
, c
, d
, Row_Number() OVER (PARTITION BY a, b, c ORDER BY d) As sequence
FROM your_table
)
SELECT CASE WHEN sequence = 1 THEN a ELSE '' END As a
, CASE WHEN sequence = 1 THEN b ELSE '' END As b
, CASE WHEN sequence = 1 THEN c ELSE '' END As c
, d
FROM cte
ORDER
BY a
, b
, c
, d
The idea is to assign an incremental counter to each row, that restarts after each change of a + b + c.
We then use a conditional statement to show a value or not (basically only show on the first instance of each group)
The analytic ROW_NUMBER() function is good for this. I've made up column names because you didn't supply any. To assign a row number by customer, use something like this:
SELECT
Name,
Phone,
Address,
Response,
ROW_NUMBER() OVER (PARTITION BY Name, Phone, Address ORDER BY Response) AS CustRow
FROM myTable
That will assign row number within each customer. Try it yourself and I think it will make sense.
You can put it into a subquery or CTE from there and only show customer ID information like name, phone, and address when you're on the first row for each customer:
SELECT
CASE WHEN CustRow = 1 THEN Name ELSE '' END AS Name,
CASE WHEN CustRow = 1 THEN Phone ELSE '' END AS Phone,
CASE WHEN CustRow = 1 THEN Address ELSE '' END AS Address,
Response
FROM (
SELECT
Name,
Phone,
Address,
Response,
ROW_NUMBER() OVER (PARTITION BY Name, Phone, Address ORDER BY Response) AS CustRow
FROM myTable) custSubquery
ORDER BY Name, Phone, Address
The custSubquery on the second-to-last line is because SQL Server requires all subqueries to be aliased, even if the alias isn't used.
The most important thing is to determine how your last column will be ordered for display and to make sure that it's consistent in the ROW_NUMBER() function as well as the final ORDER BY.
If you need more help, please supply table and column names, and specify how results are ordered within each customer.
I have a table with two columns: col_order (int), and name (text). I would like to retrieve ordered rows such that, when col_order is not null, it determines the order, but when its null, then name determines the order. I thought of an order by clause such as
order by coalesce( col_order, name )
However, this won't work because the two columns have different type. I am considering converting both into bytea, but: 1) to convert the integer is there a better method than just looping moding by 256, and stacking up individual bytes in a function, and 2) how do I convert "name" to insure some sort of sane collation order (assuming name has order ... well citext would be nice but I haven't bothered to rebuild to get that ... UTF8 for the moment).
Even if all this is possible (suggestions on details welcome) it seems like a lot of work. Is there a better way?
EDIT
I got an excellent answer by Gordon, but it shows I didn't phrase the question correctly. I want a sort order by name where col_order represents places where this order is overridden. This isn't a well posed problem, but here is one acceptable solution:
col_order| name
----------------
null | a
1 | foo
null | foo1
2 | bar
Ie -- here if col_order is null name should be inserted after name closest in alphabetical order but less that it. Otherwise, this could be gotten by:
order by col_order nulls last, name
EDIT 2
Ok ... to get your creative juices flowing, this seems to be going in the right direction:
with x as ( select *,
case when col_order is null then null else row_number() over (order by col_order) end as ord
from temp )
select col_order, name, coalesce( ord, lag(ord,1) over (order by name) + .5) as ord from x;
It gets the order from the previous row, sorted by name, when there is no col_order. It isn't right in general... I guess I'd have to go back to the first row with non-null col_order ... it would seem that sql standard has "ignore nulls" for window functions which might do this, but isn't implemented in postgres. Any suggestions?
EDIT 3
The following would seem close -- but doesn't work. Perhaps window evaluation is a bit strange with recursive queries.
with recursive x(col_order, name, n) as (
select col_order, name, case when col_order is null then null
else row_number() over (order by col_order) * t end from temp, tot
union all
select col_order, name,
case when row_number() over (order by name) = 1 then 0
else lag(n,1) over (order by name) + 1 end from x
where x.n is null ),
tot as ( select count(*) as t from temp )
select * from x;
Just use multiple clauses:
order by (case when col_order is not null then 1 else 2 end),
col_order,
name
When col_order is not null, then 1 is assigned for the first sort key. When it is null, then 2 is assigned. Hence, the not-nulls will be first.
Ok .. the following seems to work -- I'll leave the question "unanswered" though pending criticism or better suggestions:
Using the last_agg aggregate from here:
with
tot as ( select count(*) as t from temp ),
x as (
select col_order, name,
case when col_order is null then null
else (row_number() over (order by col_order)) * t end as n,
row_number() over (order by name) - 1 as i
from temp, tot )
select x.col_order, x.name, coalesce(x.n,last_agg(y.n order by y.i)+x.i, 0 ) as n
from x
left join x as y on y.name < x.name
group by x.col_order, x.n, x.name, x.i
order by n;
I'm getting a frustrating error in one of my SQL Server 2008 queries. It parses fine, but crashes when I try to execute. The error I get is the following:
Msg 8120, Level 16, State 1, Line 4
Column
'customertraffic_return.company' is
invalid in the select list because it
is not contained in either an
aggregate function or the GROUP BY
clause.
SELECT *
FROM (SELECT ctr.sp_id AS spid,
Substring(ctr.company, 1, 20) AS company,
cci.email_address AS tech_email,
CASE
WHEN rating IS NULL THEN 'unknown'
ELSE rating
END AS rating
FROM customer_contactinfo cci
INNER JOIN customertraffic_return ctr
ON ctr.sp_id = cci.sp_id
WHERE cci.email_address <> ''
AND cci.email_address NOT LIKE '%hotmail%'
AND cci.email_address IS NOT NULL
AND ( region LIKE 'Europe%'
OR region LIKE 'Asia%' )
AND SERVICE IN ( '1', '2' )
AND ( rating IN ( 'Premiere', 'Standard', 'unknown' )
OR rating IS NULL )
AND msgcount >= 5000
GROUP BY ctr.sp_id,
cci.email_address) AS a
WHERE spid NOT IN (SELECT spid
FROM customer_exclude)
GROUP BY spid,
tech_email
Well, the error is pretty clear, no??
You're selecting those columns in your inner SELECT:
spid
company
tech_email
rating
and your grouping only by two of those (GROUP BY ctr.sp_id, cci.email_address).
Either you need group by all four of them (GROUP BY ctr.sp_id, cci.email_address, company, rating), or you need to apply an aggregate function (SUM, AVG, MIN, MAX) to the other two columns (company and rating).
Or maybe using a GROUP BY here is totally the wrong way to do - what is it you're really trying to do here??
The inner query:
SELECT ctr.sp_id AS spid,
Substring(ctr.company, 1, 20) AS company,
cci.email_address AS tech_email,
CASE
WHEN rating IS NULL THEN 'unknown'
ELSE rating
END AS rating
FROM customer_contactinfo cci
INNER JOIN customertraffic_return ctr
ON ctr.sp_id = cci.sp_id
WHERE cci.email_address <> ''
AND cci.email_address NOT LIKE '%hotmail%'
AND cci.email_address IS NOT NULL
AND ( region LIKE 'Europe%'
OR region LIKE 'Asia%' )
AND SERVICE IN ( '1', '2' )
AND ( rating IN ( 'Premiere', 'Standard', 'unknown' )
OR rating IS NULL )
AND msgcount >= 5000
GROUP BY ctr.sp_id,
cci.email_address
has 4 non-aggregate things in the select (sp_id, company, email_address, rating) and you only group on two of them, so it is throwing an error on the first one it sees
So you either need to not group by any of them or group by all of them
i suggest replacing the * with a fully specified column list.
you can either group by all selected columns or use the other columns (not in group by clause) in a aggregate function (like sum)
you cannot: select a,b,c from bla group by a,b
but you can: select a,b,sum(c) from bla groupy by a,b