How do you merge duplicate rows in a table in BigQuery - replacing missing values with most recent records - sql

Question

For example, I have a table of leads from a marketing database. There are multiple records with duplicate email values. I'd like to merge all of the duplicate records to roll up into the latest updated record and if the latest updated record is missing values for certain fields then update those fields from other records most recently updated.
Table:
First
last
Email
Phone
Job Title
State
Last Updated
John
Doe
john.doe#example.com
MD
1/1/2019
John
low
john.doe#example.com
1234567891
Coach
VA
1/1/2018
John
Doe
john.doe#example.com
3214569875
Teacher
CA
1/1/2017
Andy
Yes
john.doe#example.com
DC
1/1/2021
Roby
Doe
john.doe#example.com
8628423578
Scientist
VA
1/1/2025
Output - One record:
First
last
Email
Phone
Job Title
State
Last Updated
Andy
Yes
john.doe#example.com
1234567891
Coach
DC
1/1/2021
In this example, since the 2021 record is missing a phone number and job title, those values are pulled from the most recent updated records (2018).
I've thought about using Distinct or Unique functions but not sure how to execute on the merge using the last updated record and then filling in the blank values with the other most recent records. Any help would be greatly appreciated!!
Thank you in advance.
Best,
Dawit

Answer 1

Consider below approach - I think it is most generic - you need just make sure you have correct list of fields in unpivot and pivot lines. Though there is an assumption that following fields (First, Last, Phone, Job_Title, State) are all of string data type
select First, Last, Email, Phone, Job_Title, State, max_Last_Updated as Last_Updated
from (
select * except(Last_Updated),
max(Last_Updated) over(partition by Email) as max_Last_Updated
from data
unpivot (value for col in (First, Last, Phone, Job_Title, State))
where true
qualify row_number() over(partition by Email, col order by Last_Updated desc) = 1
)
pivot (max(value) for col in ('First', 'Last', 'Phone', 'Job_Title', 'State', 'Last_Updated'))
If applied to sample data in your question (excluding 2025 row) - output is

Answer 2

You need a method to know that these are all the same record. You can use last_value(ignore nulls) for this purpose:
select t.*,
last_value(first ignore nulls) over (partition by email order by last_updated) as imputed_first,
last_value(last ignore nulls) over (partition by email order by last_updated) as imputed_first,
. . . -- and so on for the other columns
from t;