Query to find and subtract two timestamps associated with the same identifier - sql

Question

I'm very new to BigQuery and not terribly familiar with SQL. I have a table of data that looks like this, where MyDate is a Timestamp object:
Row
MyDate
StateTransition
MyIdentifier
1
2022-09-23 00:08:00 UTC
Start
6371
2
2022-10-10 01:17:14 UTC
Finished
6371
3
2022-09-26 04:51:40 UTC
Start
7768
4
2022-10-05 03:44:32 UTC
Finished
7768
etc.
My query looks something like
SELECT *
FROM <my-data-source>
WHERE (StateTransition="Start" OR StateTransition="Finished")
ORDER BY MyIdentifier, MyDate
What I'm trying to do is calculate the elapsed time (in days) between the Start and Finished timestamps associated with each MyIdentifier, and to have that displayed in another column. It could look like:
Row
MyDate
StateTransition
MyIdentifier
ElapsedTime
1
2022-09-23 00:08:00 UTC
Start
6371
2
2022-10-10 01:17:14 UTC
Finished
6371
0.33
3
2022-09-26 04:51:40 UTC
Start
7768
4
2022-10-05 03:44:32 UTC
Finished
7768
0.04
Alternatively, it could even be flattened a little to something like:
Row
StartTransition
FinishedTransition
MyIdentifier
ElapsedTime
1
2022-09-23 00:08:00 UTC
2022-10-10 01:17:14 UTC
6371
0.33
2
2022-09-26 04:51:40 UTC
2022-10-05 03:44:32 UTC
7768
0.04
I've tried looking through the BigQuery docs and Stack Overflow but haven't found anything that addresses this use case of selecting items from multiple rows with a common identifier and then performing an operation on them. It seems like subtracting the two timestamps would be done with the TIMESTAMP_DIFF function.
Any assistance is appreciated!

Answer 1

WITH sample_table AS (
SELECT TIMESTAMP '2022-09-23 00:08:00 UTC' MyDate, 'Start' StateTransition, 6371 MyIdentifier, 'aaa' AS author UNION ALL
SELECT '2022-10-10 01:17:14 UTC' MyDate, 'Finished' StateTransition, 6371 MyIdentifier, 'bbb' AS author UNION ALL
SELECT '2022-09-26 04:51:40 UTC' MyDate, 'Start' StateTransition, 7768 MyIdentifier, 'ccc' AS author UNION ALL
SELECT '2022-10-05 03:44:32 UTC' MyDate, 'Finished' StateTransition, 7768 MyIdentifier, 'ccc' AS author
)
SELECT MyIdentifier,
ARRAY_AGG(author ORDER BY MyDate LIMIT 1)[SAFE_OFFSET(0)] AS author,
MIN(MyDate) AS StartTransition,
MAX(MyDate) AS FinishedTransition,
TIMESTAMP_DIFF(MAX(MyDate), MIN(MyDate), DAY) AS ElapsedTime,
FROM sample_table
WHERE (StateTransition="Start" OR StateTransition="Finished")
GROUP BY 1;
Query results
If Start and Finished has different author name and you want the name of Finished, you can use below instead.
ARRAY_AGG(author ORDER BY MyDate DESC LIMIT 1)[SAFE_OFFSET(0)] AS author,

Answer 2

For flattened result, you might consider below using an aggregation.
WITH sample_table AS (
SELECT TIMESTAMP '2022-09-23 00:08:00 UTC' MyDate, 'Start' StateTransition, 6371 MyIdentifier UNION ALL
SELECT '2022-10-10 01:17:14 UTC' MyDate, 'Finished' StateTransition, 6371 MyIdentifier UNION ALL
SELECT '2022-09-26 04:51:40 UTC' MyDate, 'Start' StateTransition, 7768 MyIdentifier UNION ALL
SELECT '2022-10-05 03:44:32 UTC' MyDate, 'Finished' StateTransition, 7768 MyIdentifier
)
SELECT MyIdentifier,
MIN(MyDate) AS StartTransition,
MAX(MyDate) AS FinishedTransition,
TIMESTAMP_DIFF(MAX(MyDate), MIN(MyDate), DAY) AS ElapsedTime,
FROM sample_table
WHERE (StateTransition="Start" OR StateTransition="Finished")
GROUP BY 1;
Query results
But for the intermediate result, we need a window function.
SELECT *,
IF(
StateTransition = 'Finished',
TIMESTAMP_DIFF(MyDate, FIRST_VALUE(IF(StateTransition = 'Start', MyDate, NULL) IGNORE NULLS) OVER w, DAY),
NULL
) AS ElapsedTime
FROM sample_table
WINDOW w AS (PARTITION BY MyIdentifier ORDER BY MyDate);
and if you want flattend result from the above result (using a window function), the query will looks like below which shows same result as the first query using an aggregation.
SELECT MyIdentifier,
FIRST_VALUE(IF(StateTransition = 'Start', MyDate, NULL) IGNORE NULLS) OVER w AS StartTransition,
MyDate AS FinishedTransition,
IF(
StateTransition = 'Finished',
TIMESTAMP_DIFF(MyDate, FIRST_VALUE(IF(StateTransition = 'Start', MyDate, NULL) IGNORE NULLS) OVER w, DAY),
NULL
) AS ElapsedTime
FROM sample_table
QUALIFY StateTransition = 'Finished'
WINDOW w AS (PARTITION BY MyIdentifier ORDER BY MyDate);

Answer 3

Consider below approach
select *, timestamp_diff(Transition_Finished, Transition_Start, day) as ElapsedTime
from your_table
pivot (max(MyDate) Transition for StateTransition in ('Start', 'Finished'))
if applied to sample data in your question - output is
Use below to test
WITH your_table AS (
SELECT TIMESTAMP '2022-09-23 00:08:00 UTC' MyDate, 'Start' StateTransition, 6371 MyIdentifier UNION ALL
SELECT '2022-10-10 01:17:14 UTC' MyDate, 'Finished' StateTransition, 6371 MyIdentifier UNION ALL
SELECT '2022-09-26 04:51:40 UTC' MyDate, 'Start' StateTransition, 7768 MyIdentifier UNION ALL
SELECT '2022-10-05 03:44:32 UTC' MyDate, 'Finished' StateTransition, 7768 MyIdentifier
)
select *, timestamp_diff(Transition_Finished, Transition_Start, day) as ElapsedTime
from your_table
pivot (max(MyDate) Transition for StateTransition in ('Start', 'Finished'))

Answer 4

I think that for each MyIdentifier you should have only one start and one finish, so you can simply split and join:
;WITH
ts AS ( SELECT * FROM <my-data-source> WHERE StateTransition = 'Start'),
tf AS ( SELECT * FROM <my-data-source> WHERE StateTransition = 'Finished')
SELECT
ts.MyIdentifier,
ts.MyDate StartTransition,
tf.MyDate FinishedTransition,
TIMESTAMP_DIFF(ts.MyDate, tf.MyDate, DAY) ElapsedTime
FROM ts
LEFT JOIN tf on ts.MyIdentifier = tf.MyIdentifier
UPDATE
If you have more start and finish for each identifier you need to choice which one to keep, I will assume you want to keep the 1st start and the last finish:
;WITH
ts AS (
SELECT *,
ROW_NUMBER() OVER (PARTITION BY MyIdentifier ORDER BY MyDate) n
FROM <my-data-source> WHERE StateTransition = 'Start'),
tf AS (
SELECT *,
ROW_NUMBER() over (partition by MyIdentifier ORDER BY MyDate DESC) n
FROM <my-data-source> WHERE StateTransition = 'Finished')
SELECT
ts.MyIdentifier,
ts.MyDate StartTransition,
tf.MyDate FinishedTransition,
TIMESTAMP_DIFF(ts.MyDate, tf.MyDate, DAY) ElapsedTime
FROM ts
LEFT JOIN tf on ts.MyIdentifier = tf.MyIdentifier and tf.n=1
WHERE ts.n=1