How do I coalesce NULLs across multiple rows in BigQuery? - sql

Question

I have the following table:
Date |event_number| customer_id1 | customer_age | customer_gender
10/01/2020 | 1 | abc | NULL | NULL
10/01/2020 | 2 | abc | NULL | male
10/01/2020 | 3 | abc | 45 | NULL
10/01/2020 | 1 | def | 30 | NULL
I want to run a SQL query each day to look for new combinations of custom_id1, customer_age, customer_gender.
Output should look like this:
query_run_time | customer_id1 | customer_age | customer gender
11/01/2020 | abc | 45 | male
11/01/2020 | def | 30 | NULL
Query run time is the date the query was run. If the combination (customer_id, custmer_age, customer_gender) is already in the table I don't want to insert the row.
Thanks

Answer 1

You can use window functions to assign internal row numbers for merge multiple queries, e.g. like this:
SELECT COALESCE(a.customer_id, b.customer_id) as customer_id
, customer_age
, customer_gender
FROM (
SELECT customer_id, customer_age
, ROW_NUMBER() OVER ( PARTITION BY customer_id ORDER BY customer_age ) AS row_no
FROM customer_event
WHERE customer_age IS NOT NULL
) a
FULL JOIN (
SELECT customer_id, customer_gender
, ROW_NUMBER() OVER ( PARTITION BY customer_id ORDER BY customer_gender ) AS row_no
FROM customer_event
WHERE customer_gender IS NOT NULL
) b ON b.customer_id = a.customer_id
AND b.row_no = a.row_no
ORDER BY COALESCE(a.customer_id, b.customer_id)
, COALESCE(a.row_no, b.row_no)
Schema and Test Data
CREATE TABLE customer_event (
event_number INT NOT NULL,
customer_id VARCHAR(10) NOT NULL,
customer_age INT,
customer_gender VARCHAR(10)
);
INSERT INTO customer_event VALUES
( 1, 'abc', NULL, NULL ),
( 2, 'abc', NULL, 'male' ),
( 3, 'abc', 45 , NULL ),
( 4, 'abc', 50 , 'female' ),
( 5, 'abc', 27 , NULL ),
( 1, 'def', 30 , NULL );
Output
customer_id customer_age customer_gender
abc 27 female
abc 45 male
abc 50 (null)
def 30 (null)
The above is from testing with PostgreSQL 9.6 on SQL Fiddle.

Answer 2

Use Window function
SELECT query_run_time, customer_id, MAX(customer_age) customer_age,
MAX(customer_gender)customer_gender
FROM tbl
GROUP BY query_run_time, customer_id
FIDDLE DEMO
Output
query_run_time | customer_id1 | customer_age | customer gender
11/01/2010 | abc | 45 | male
11/01/2020 | def | 30 | NULL

Answer 3

I suspect that what you really want is the most recent value for each column. Here is one method:
select date, customerid1,
array_agg(customer_age ignore nulls order by event_number desc limit 1)[safe_ordinal(1) as age,
array_agg(customer_gender ignore nulls order by event_number desc limit 1)[safe_ordinal(1) as gender
from t
group by date, customerid1;