From Change Log Table to Status on a Given Day - sql

Question

I am trying to convert a change log table into a historical status table using BigQuery's Standard SQL.
The part giving me a hang up is how to select the most recent change log that is before the date to join on.
I had not encountered window functions or indexing during my college years, so I would appreciate guidance on how to apply those functions if they're part of the ideal solution.
Change_Logs table
Update Key Tostring
1 2019-01-30 17:57:51.910 PS-5864 To Do
2 2019-02-11 20:59:08.582 PS-5864 In Progress
3 2019-02-12 19:52:18.733 PS-5864 Done
4 2019-01-31 16:52:12.832 PS-4672 To Do
5 2019-02-11 14:11:13.442 PS-4672 In Progress
6 2019-02-12 04:22:33.111 PS-4672 Done
Dates table
Date
1 2019-02-10
2 2019-02-11
3 2019-02-12
4 2019-02-13
Desired Result:
Date Key Status
1 2019-02-10 00:00:00.000 PS-5864 To Do
2 2019-02-10 00:00:00.000 PS-4672 To Do
3 2019-02-11 00:00:00.000 PS-5864 To Do
4 2019-02-11 00:00:00.000 PS-4672 To Do
5 2019-02-12 00:00:00.000 PS-5864 In Progress
6 2019-02-12 00:00:00.000 PS-4672 In Progress
7 2019-02-13 00:00:00.000 PS-5864 Done
8 2019-02-13 00:00:00.000 PS-4672 Done

Answer 1

Below is for BigQuery Standard SQL
#standardSQL
SELECT d.date, key,
ARRAY_AGG(status ORDER BY l.update DESC LIMIT 1)[OFFSET(0)] status
FROM `project.dataset.dates` d
JOIN `project.dataset.change_logs` l
ON DATE_DIFF(d.date, DATE(l.update), DAY) > 0
GROUP BY d.date, key
You can test, play with above using sample data from your question as in example below
#standardSQL
WITH `project.dataset.change_logs` AS (
SELECT DATETIME '2019-01-30 17:57:51.910' `update`, 'PS-5864' key, 'To Do' status UNION ALL
SELECT '2019-02-11 20:59:08.582', 'PS-5864', 'In Progress' UNION ALL
SELECT '2019-02-12 19:52:18.733', 'PS-5864', 'Done' UNION ALL
SELECT '2019-01-31 16:52:12.832', 'PS-4672', 'To Do' UNION ALL
SELECT '2019-02-11 14:11:13.442', 'PS-4672', 'In Progress' UNION ALL
SELECT '2019-02-12 04:22:33.111', 'PS-4672', 'Done'
), `project.dataset.dates` AS (
SELECT DATE '2019-02-10' `date` UNION ALL
SELECT '2019-02-11' UNION ALL
SELECT '2019-02-12' UNION ALL
SELECT '2019-02-13'
)
SELECT d.date, key,
ARRAY_AGG(status ORDER BY l.update DESC LIMIT 1)[OFFSET(0)] status
FROM `project.dataset.dates` d
JOIN `project.dataset.change_logs` l
ON DATE_DIFF(d.date, DATE(l.update), DAY) > 0
GROUP BY d.date, key
-- ORDER BY d.date, key
with result
Row date key status
1 2019-02-10 PS-4672 To Do
2 2019-02-10 PS-5864 To Do
3 2019-02-11 PS-4672 To Do
4 2019-02-11 PS-5864 To Do
5 2019-02-12 PS-4672 In Progress
6 2019-02-12 PS-5864 In Progress
7 2019-02-13 PS-4672 Done
8 2019-02-13 PS-5864 Done

Answer 2

The key idea is to generate the rows with a cross join. Then what you really want is lag(. . . ignore nulls) -- but not supported in BigQuery.
Instead, you can do some array manipulation:
select d.date, cl.key,
array_agg(cl.status ignore nulls order by d.date desc limit 2)[ordinal(2)]
from dates d cross join
(select distinct key from change_logs cl) k left join
change_logs cl
on date(cl.update) = d.date and cl.key = k.key;
EDIT:
The above is not quite correct, because we are missing dates that occur before the specified period. I think the simplest method is to add them and then remove them:
select *
from (select d.date, cl.key,
array_agg(cl.status ignore nulls order by d.date desc limit 2)[ordinal(2)]
from (select d.date
from dates d
union
select distinct date(cl.update)
from change_logs
) d cross join
(select distinct key from change_logs cl) k left join
change_logs cl
on date(cl.update) = d.date and cl.key = k.key
)
where date in (select d.date from dates);