Full outer join on a table itself and run some window functions - sql

Question

Background
I have some ETL job processing real-time log files hourly. Whenever the system generates a new event, it will take a snapshot of all historical event summary (if exists) and record it together with the current event. Then the data is loaded into Redshift.
Example
The table looks like something below:
+------------+--------------+---------+-----------+-------+-------+
| current_id | current_time | past_id | past_time | freq1 | freq2 |
+------------+--------------+---------+-----------+-------+-------+
| 2 | time2 | 1 | time1 | 13 | 5 |
| 3 | time3 | 1 | time1 | 13 | 5 |
| 3 | time3 | 2 | time2 | 2 | 1 |
| 4 | time4 | 1 | time1 | 13 | 5 |
| 4 | time4 | 2 | time2 | 2 | 1 |
| 4 | time4 | 3 | time3 | 1 | 1 |
+------------+--------------+---------+-----------+-------+-------+
This is what happened for the above table:
time1: event 1 happened. System took a snapshot, but nothing is recorded.
time2: event 2 happened. System took a snapshot and record event 1.
time3: event 3 happened. System took a snapshot and record event 1 & 2.
time4: event 4 happened. System took a snapshot and record event 1, 2 & 3.
Desired Outcome
I will need to transform the data into the following format in order to do some analysis:
+----+------------+-------+-------+
| id | event_time | freq1 | freq2 |
+----+------------+-------+-------+
| 1 | time1 | 0 | 0 |
| 2 | time2 | 13 | 5 | -- 13 | 5
| 3 | time3 | 15 | 6 | -- 13 + 2 | 5 + 1
| 4 | time4 | 16 | 7 | -- 15 + 1 | 6 + 1
+----+------------+-------+-------+
Basically, the new freq1 and freq2 are cumulative sum of lagged freq1 and freq2.
My Idea
I am thinking of a self full outer join on current_id and past_id and achieve the following result first:
+----+------------+-------+-------+
| id | event_time | freq1 | freq2 |
+----+------------+-------+-------+
| 1 | time1 | 13 | 5 |
| 2 | time2 | 2 | 1 |
| 3 | time3 | 1 | 1 |
| 4 | time4 | null | null |
+----+------------+-------+-------+
Then I can do a window function of lag over() and then sum over().
Question
Is this the correct approach? Is there a more efficient way to do this? This is just a small sample of the actual data, so performance could be a concern.
My query is always returning a lot of duplicated values, so I am not sure what went wrong.
Solution
Answer from #GordonLinoff is correct for the above use case. I am adding some minor updates in order to get it working on my actual table. The only difference is that my event_id are some 36-character Java UUID and the event_time are timestamp.
select distinct past_id, past_time, 0 as freq1, 0 as freq2
from (
select past_id, past_time,
row_number() over (partition by current_id order by current_time desc) as seqnum
from t
) a
where a.seqnum = 1
union all
select current_id, current_time,
sum(freq1) over (order by current_time rows unbounded preceding) as freq1,
sum(freq2) over (order by current_time rows unbounded preceding) as freq2
from (
select current_id, current_time, freq1, freq2,
row_number() over (partition by current_id order by past_id desc) as seqnum
from t
) b
where b.seqnum = 1;

Answer 1

I'm thinking you want union all along with window functions. Here is an example:
select min(past_id) as id, min(past_time) as event_time, 0 as freq1, 0 as freq2
from t
union all
(select current_id, current_time,
sum(freq1) over (order by current_time),
sum(freq2) over (order by current_time)
from (select current_id, current_time, freq1, freq2,
row_number() over (partition by current_id order by past_id desc) as seqnum
from t
) t
where seqnum = 1
);

Answer 2

The way your data is in your snapshot table, I think the following SQL should give you what you are looking for in the desired outcome that you posted
SELECT 1 AS id
,"time1" AS event_time
,0 AS freq1
,0 AS freq2
UNION
SELECT T.id
,T.current_time AS event_time
,SUM(T.freq1) AS freq1
,SUM(T.freq2) AS freq2
FROM snapshot AS T
GROUP
BY T.id
,T.current_name
The first SELECT in the above UNION is so that you can get the first record for time1 since it does not really have an entry in your base table which holds all the snapshots.. It does not have a FROM in it since we are only selecting variables, if Redshift does not support it you might need to look for something equivalent to the DUAL table in Oracle.
Hope this helps..