In BigQuery, can I add rows of missing data? [duplicate] - sql

Question

This question already has answers here:
How to add records for each user based on another existing row in BigQuery?
(3 answers)
Closed 2 years ago.
I have a table where each row represents the number of transactions a user has per day. If they had no transaction that day then they don't have a row for that date. How can I add these 'missing rows' and set the number of transactions to 0
My table:
Date | User | numTransactions
2020-01-01 | anna | 2
2020-01-01 | john | 3
2020-01-02 | anna | 1
2020-01-04 | anna | 1
2020-01-05 | john | 2
Anna had transactions on Jan 1,2, and 4 but not Jan 3, and 5
John had transactions on Jan 1, and 5 but not Jan 2, 3, and 4
I want to add rows which shows the dates there are 0 transactions
Date | User | numTransactions
2020-01-01 | anna | 2
2020-01-01 | john | 3
2020-01-02 | anna | 1
2020-01-04 | anna | 1
2020-01-05 | john | 2
2020-01-02 | john | 0
2020-01-03 | anna | 0
2020-01-03 | john | 0
2020-01-04 | john | 0
2020-01-05 | anna | 0

Answer 1

You can join with GENERATE_DATE_ARRAY:
WITH test_table AS (
SELECT DATE '2020-01-01' AS Date, 'anna' AS User, 2 AS numTransactions UNION ALL
SELECT '2020-01-01', 'john', 3 UNION ALL
SELECT '2020-01-02', 'anna', 1 UNION ALL
SELECT '2020-01-04', 'anna', 1 UNION ALL
SELECT '2020-01-05', 'john', 2
),
clients_list AS (
SELECT DISTINCT User FROM test_table
)
SELECT
Date,
User,
IFNULL(numTransactions, 0) AS numTransactions
FROM UNNEST(GENERATE_DATE_ARRAY('2020-01-01', '2020-01-05')) AS Date
CROSS JOIN clients_list
LEFT JOIN test_table USING(Date, User)

Answer 2

I recommend writing the code in this fashion:
WITH t AS (
SELECT DATE '2020-01-01' AS Date, 'anna' AS User, 2 AS numTransactions UNION ALL
SELECT '2020-01-01', 'john', 3 UNION ALL
SELECT '2020-01-02', 'anna', 1 UNION ALL
SELECT '2020-01-04', 'anna', 1 UNION ALL
SELECT '2020-01-05', 'john', 2
)
SELECT u.user, COALESCE(dte, u.date) as date,
(CASE WHEN dte = u.date THEN u.numTransactions ELSE 0 END) as numTransactions
FROM (SELECT user, date, numTransactions,
COALESCE(DATE_ADD(LEAD(DATE) OVER (PARTITION BY user ORDER BY date), INTERVAL -1 DAY), DATE '2020-01-05') as end_date
FROM t
) u LEFT JOIN
UNNEST(GENERATE_DATE_ARRAY(date, end_date, INTERVAL 1 DAY)) dte
ON 1=1
ORDER BY user, date;
This is slightly simpler than generating all the dates up-front (not requiring getting the unique names and then re-joining to the same table).
Much more important are the performance characteristics, which have proven very important in my experience in making this scalable. Basically, the CROSS JOIN for generating all user/date combinations uses a lot of resources. This version keeps all the operations "local" to a given user (well, there is some data movement to get all the users co-located on the same node).
Specifically, I have seen queries that run out of resources or literally take hours to complete using the CROSS JOIN method finish within a minute using this method.