Count total and count having in the same query - sql

Question

is there a way to get a total count of rows per {id, date} and the count > 1 per {id, date, columnX} in the same query?
For example, having such a table:
id date columnX
1 2017-04-20 a
1 2017-04-20 a
1 2017-04-18 b
1 2017-04-17 c
2 2017-04-20 a
2 2017-04-20 a
2 2017-04-20 c
2 2017-04-19 b
2 2017-04-19 b
2 2017-04-19 b
2 2017-04-19 b
2 2017-04-19 c
As the result, I wanna get the following table:
id date columnX count>1 count_total
1 2017-04-20 a 2 2
2 2017-04-20 a 2 3
2 2017-04-19 b 4 5
I tried to do it with partition by but receive weird results. I've heard Rollup function might be used but it seems like it's applicable only in legacy SQL, which is not the option for me.

Answer 1

If I understand correctly, you can use window functions:
select id, date, columnx, cnt,
(case when cnt > 1 then cnt else 0 end) as cnt_gt_1,
total_cnt
from (select id, date, columnx, count(*) as cnt
sum(count(*)) over (partition by id, date) as total_cnt
from t
group by id, date, columnx
) x
where cnt > 1;

Answer 2

Another possibility:
SELECT
id,
date,
data.columnX columnX,
data.count_ count_bigger_1,
count_total
FROM(
SELECT
id,
date,
ARRAY_AGG(columnX) data,
COUNT(1) count_total
FROM
`your_table_name`
GROUP BY
id, date
),
UNNEST(ARRAY(SELECT AS STRUCT columnX, count(1) count_ FROM UNNEST(data) columnX GROUP BY columnX HAVING count(1) > 1)) data
You can test it with simulated data:
WITH data AS(
SELECT 1 AS id, '2017-04-20' AS date, 'a' AS columnX UNION ALL
SELECT 1 AS id, '2017-04-20' AS date, 'a' AS columnX UNION ALL
SELECT 1 AS id, '2017-04-18' AS date, 'b' AS columnX UNION ALL
SELECT 1 AS id, '2017-04-17' AS date, 'c' AS columnX UNION ALL
SELECT 2 AS id, '2017-04-20' AS date, 'a' AS columnX UNION ALL
SELECT 2 AS id, '2017-04-20' AS date, 'a' AS columnX UNION ALL
SELECT 2 AS id, '2017-04-20' AS date, 'c' AS columnX UNION ALL
SELECT 2 AS id, '2017-04-19' AS date, 'b' AS columnX UNION ALL
SELECT 2 AS id, '2017-04-19' AS date, 'b' AS columnX UNION ALL
SELECT 2 AS id, '2017-04-19' AS date, 'b' AS columnX UNION ALL
SELECT 2 AS id, '2017-04-19' AS date, 'b' AS columnX UNION ALL
SELECT 2 AS id, '2017-04-19' AS date, 'c' AS columnX
)
SELECT
id,
date,
data.columnX columnX,
data.count_ count_bigger_1,
count_total
FROM(
SELECT
id,
date,
ARRAY_AGG(columnX) data,
COUNT(1) count_total
FROM
data
GROUP BY
id, date
),
UNNEST(ARRAY(SELECT AS STRUCT columnX, count(1) count_ FROM UNNEST(data) columnX GROUP BY columnX HAVING count(1) > 1)) data
This solution avoids the analytical function (which can be quite expensive depending on the input) and scales well to large volumes of data.

Answer 3

I recommend you to add into your example two more below rows
1 2017-04-20 x
1 2017-04-20 x
and check what solutions in two previous answers will give you:
It will be something like below:
id date columnX count>1 count_total
1 2017-04-20 a 2 4
1 2017-04-20 x 2 4
2 2017-04-20 a 2 3
2 2017-04-19 b 4 5
Notice two rows for id=1 and date=2017-04-20 and both having count_total=4
I am not sure if this is what you want - even though you might not even considered this scenario in your question
Anyway, I feel that to support more generic case like above your expectation of output should of be like below
Row id date x.columnX x.countX count_total
1 1 2017-04-20 x 2 4
a 2
2 2 2017-04-20 a 2 3
3 2 2017-04-19 b 4 5
where x is repeated field and each value represents respective columnX with its count
Below query does exactly this
#standardSQL
SELECT id, date,
ARRAY(SELECT x FROM UNNEST(x) AS x WHERE countX > 1) AS x,
count_total
FROM (
SELECT id, date, SUM(countX) AS count_total,
ARRAY_AGG(STRUCT<columnX STRING, countX INT64>(columnX, countX) ORDER BY countX DESC) AS X
FROM (
SELECT id, date,
columnX, COUNT(1) countX
FROM `yourTable`
GROUP BY id, date, columnX
)
GROUP BY id, date
HAVING count_total > 1
)
you can play/test it with dummy data from your question
#standardSQL
WITH `yourTable` AS(
SELECT 1 AS id, '2017-04-20' AS date, 'a' AS columnX UNION ALL
SELECT 1, '2017-04-20', 'a' UNION ALL
SELECT 1, '2017-04-20', 'x' UNION ALL
SELECT 1, '2017-04-20', 'x' UNION ALL
SELECT 1, '2017-04-18', 'b' UNION ALL
SELECT 1, '2017-04-17', 'c' UNION ALL
SELECT 2, '2017-04-20', 'a' UNION ALL
SELECT 2, '2017-04-20', 'a' UNION ALL
SELECT 2, '2017-04-20', 'c' UNION ALL
SELECT 2, '2017-04-19', 'b' UNION ALL
SELECT 2, '2017-04-19', 'b' UNION ALL
SELECT 2, '2017-04-19', 'b' UNION ALL
SELECT 2, '2017-04-19', 'b' UNION ALL
SELECT 2, '2017-04-19', 'c'
)
SELECT id, date,
ARRAY(SELECT x FROM UNNEST(x) AS x WHERE countX > 1) AS x,
count_total
FROM (
SELECT id, date, SUM(countX) AS count_total,
ARRAY_AGG(STRUCT<columnX STRING, countX INT64>(columnX, countX) ORDER BY countX DESC) AS X
FROM (
SELECT id, date,
columnX, COUNT(1) countX
FROM `yourTable`
GROUP BY id, date, columnX
)
GROUP BY id, date
HAVING count_total > 1
)