Retrieve unknown values based on non-schema specific sequence of column values - sql

Question

I want to return and operate on time values based on their related event values, but only if a specific sequence of events occurs. A simplified example table below:
+--------+------------+-------+-------------+-------+-------------+-------+-------------+-------+-------------+-------+
| id | event1 | time1 | event2 | time2 | event3 | time3 | event4 | time4 | event5 | time5 |
+--------+------------+-------+-------------+-------+-------------+-------+-------------+-------+-------------+-------+
| abc123 | firstevent | 10:00 | secondevent | 10:01 | thirdevent | 10:02 | fourthevent | 10:03 | fifthevent | 10:04 |
| abc123 | thirdevent | 10:10 | secondevent | 10:11 | thirdevent | 10:12 | firstevent | 10:13 | secondevent | 10:14 |
| def456 | thirdevent | 10:20 | firstevent | 10:21 | secondevent | 10:22 | thirdevent | 10:24 | fifthevent | 10:25 |
+--------+------------+-------+-------------+-------+-------------+-------+-------------+-------+-------------+-------+
For this table we want to retrieve the times whenever this particular sequence of events occurs: firstevent, secondevent, thirdevent, and a final event of any non-zero value. Meaning the relevant entries returned would be the following:
+--------+------------+-------+-------------+-------+-------------+-------+-------------+-------+------------+-------+
| id | event1 | time1 | event2 | time2 | event3 | time3 | event4 | time4 | event5 | time5 |
+--------+------------+-------+-------------+-------+-------------+-------+-------------+-------+------------+-------+
| abc123 | firstevent | 10:00 | secondevent | 10:01 | thirdevent | 10:02 | fourthevent | 10:03 | null | null |
| null | null | null | null | null | null | null | null | null | null | null |
| def456 | null | null | firstevent | 10:21 | secondevent | 10:22 | thirdevent | 10:24 | fifthevent | 10:26 |
+--------+------------+-------+-------------+-------+-------------+-------+-------------+-------+------------+-------+
As shown above the columns are irrelevant to the occurrence of the sequence, with two results being returned starting in both the event1 and event2 columns, thus the solution should be independent and support n number of columns. These values can then be aggregated by the final non-zero event that occurs in the sequence after the 3 fixed variables to give something like the following:
+-------------+-------------------------------+
| FinalEvent | AverageTimeBetweenFinalEvents |
+-------------+-------------------------------+
| fourthevent | 1:00 |
| fifthevent | 2:00 |
+-------------+-------------------------------+

Answer 1

Below is for BigQuery Standard SQL
#standardSQL
WITH search_events AS (
SELECT ['firstevent', 'secondevent', 'thirdevent'] search
), temp AS (
SELECT *, REGEXP_EXTRACT(events, CONCAT(search, r',(\w*)')) FinalEvent
FROM (
SELECT id, [time1, time2, time3, time4, time5] times,
(SELECT STRING_AGG(event) FROM UNNEST([event1, event2, event3, event4, event5]) event) events,
(SELECT STRING_AGG(search) FROM UNNEST(search) search) search
FROM `project.dataset.table`, search_events
)
)
SELECT FinalEvent,
times[SAFE_OFFSET(ARRAY_LENGTH(REGEXP_EXTRACT_ALL(REGEXP_EXTRACT(events, CONCAT(r'(.*?)', search, ',', FinalEvent )), ',')) + 3)] time
FROM temp
WHERE IFNULL(FinalEvent, '') != ''
If to apply to sample data from your question - result is
Row FinalEvent time
1 fourthevent 10:03
2 fifthevent 10:25
So, as you can see - all final events are extracted along with their respective times
Now, you can do here whatever analytics you need - I was not sure about logic behind AverageTimeBetweenFinalEvents, so I am leaving this to you - especially that I think that main focus of the question was extraction of those final events
would you be able to provide the logic behind this statement please?
times[SAFE_OFFSET(ARRAY_LENGTH(REGEXP_EXTRACT_ALL(REGEXP_EXTRACT(events, CONCAT(r'(.*?)', search, ',', FinalEvent )), ',')) + 3)] time
Sure, hope below helps to get a logic behind that expression
assemble regular expression to extract list of events happened before matched events
extract those events
extract all commas into array
calculate position of final event by taking number of commas in above array + 3 (three is to reflect number of positions in search sequence)
extract respective time as an element of times array